ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation

Controllable traffic simulation is critical for testing autonomousdriving systems, yet existing approaches often require retraining largegenerative models with extensive annotated data. We introduce a lightweightcontrol adaptation framework that enables multi-modal controllability(sketch, latent behavior codes, and text) for pretrained state-of-the-artdiffusion and autoregressive traffic models. By modulating intermediatefeatures through identity-initialized FiLM layers, our method efficientlyadds new control modalities while preserving the base model’s generativeprior. Evaluated on Waymo Open Sim Agents Challenge, our approachdemonstrates strong controllability with less than 1% of the paired con-trol data. Through context-aware condition transfer, our framework en-ables counterfactual scenario generation and long-tail synthesis whilemaintaining stable closed-loop driving realism and safety. Our framework* Equal contribution.unlocks new possibilities for controllable traffic simulation, enabling tar-geted scenario generation through lightweight adaptation of pretrainedgenerative models. Project page: https://ecosim-web.github.io/