PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving

Vision-Language-Action Models (VLAs), which leverage theadvanced reasoning capabilities of Vision-Language Models (VLMs), showpromising generalization in complex autonomous driving scenarios. Ex-isting VLAs typically predict and optimize 3D trajectories from 2D im-ages. While intuitive, this 2D-to-3D prediction is inherently entangledwith camera parameters, leading to limited data scalability across het-erogeneous driving datasets. Moreover, directly optimizing in 3D spaceinduces severe convergence to trivial solutions, where VLAs rely on ego-status rather than visual scene understanding. To address these issues,we propose PixelPilot, a novel VLA featuring a decoupled planning andlifting paradigm. In the planning phase, PixelPilot reformulates scene un-derstanding and trajectory prediction as sensor-agnostic 2D-to-2D tasksin the image plane, thereby facilitating scalable training across diversedatasets. The planned 2D trajectories are then deterministically lifted to3D only during inference, ensuring the full exploitation of visual cues andgeneralization across different vehicles. To realize this paradigm, we pro-pose a knowledge-instilled policy learning strategy that applies dense, in-termediate rewards via Group Relative Policy Optimization (GRPO) toenforce a rigorous causal chain from visual perception to spatial planning.Extensive experiments demonstrate that PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop settings, validatingits superior scalability and visual reasoning capabilities.