Doe-2: 3D Representation World Model for Unified Driving Scene Forecasting

Forecasting future scene evolutions is crucial for autonomousdriving safety. Most existing driving world models focus on appearancegeneration and fail to model geometric and semantic evolutions. In thispaper, we propose a native 3D world model, Doe-2, to forecast the com-prehensive scene evolutions in a unified 3D latent representation space.This representation contains both appearance, geometry, and semanticinformation and can be decoded into multi-view RGB, depth, semantics,and 3D occupancy. For efficiency, we adopt an autoregressive transformerarchitecture to generate all latents for the next scene in one pass and thenuse a lightweight diffusion head to refine them for better fidelity. Operat-ing on the compact 3D latent space, our model supports efficient futurerollouts conditioned on different actions. Extensive experiments on thenuScenes dataset show the ability of Doe-2 to efficiently and comprehen-sively forecast future driving scenes. Despite being trained from scratchwith only 700 scenes, Doe-2 already shows high-quality scene generationand good generalization with action control ability.