Beyond Inpainting: Unleash 3D Understanding for Stable Camera-Controlled Video Re-rendering

Camera control has been extensively studied in conditional video generation; however, stable altering camera trajectories while faithfully preserving video content remains a challenging task. The mainstream approach to achieving camera control is warping a 3D representation according to the target trajectory. However, such methods fail to fully leverage the 3D priors of video diffusion models (VDMs) and often fall into the Inpainting Trap, resulting in subject inconsistency and degraded generation quality. To address this problem, we propose DepthDirector, a video re-rendering framework with stable camera controllability. Our key insight is to decompose the conditional injection of video content and camera trajectories by leveraging the depth video from explicit 3D representation as camera-control guidance. Specifically, we design a View-Content Dual-Stream Condition mechanism, enabling VDMs to comprehend camera movements and leverage their 3D understanding capabilities. Additionally, we construct a large-scale multi-camera synchronized dataset named MultiCamWarp Dataset using Unreal Engine 5. Extensive experiments show that DepthDirector outperforms existing methods. Our code and dataset will be publicly available on our project website: https://eleanor6725.github.io/DepthDirector/.