Teaching Vision-Language-Action Models What to See and Where to Look

Vision-Language-Action (VLA) models have emerged as apromising paradigm for end-to-end autonomous driving. However, exist-ing VLAs’ traffic relies heavily on text-centric visual question answeringand chain-of-thought reasoning data, which emphasizes linguistic rea-soning rather than action-grounded planning. As a result, the learnedrepresentations capture semantic knowledge but lack spatial dependenciescrucial for reliable trajectory prediction. We propose DriveTeach-VLA,a framework that explicitly teaches VLAs what to see and where tolook. Driving-aware Vision Distillation (DVD) injects driving-specificperceptual priors into the vision encoder, while 2D Trajectory-GuidedPrompts (2D-TGP) provide spatial conditioning aligned with feasibledriving trajectories. Together they form a vision-guided learning pipeline:what to see (DVD pretraining) → where to look (TGP-guided SFT) →how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at:https://github.com/ShivaTeam/DriveTeach-VLA.