DINOv3D: 2D-3D Joint Optimization for Unified Spatial Understanding

Adapting 2D foundation models for 3D spatial understanding faces a critical dilemma. Fine-tuning 2D models to learn 3D geometry causes the catastrophic forgetting of native 2D knowledge. Conversely, freezing the 2D model restricts 3D spatial perception. To resolve this issue, we propose DINOv3D, a joint optimization framework built upon DINOv3 that employs a homologous teacher-student architecture to establish a regularized integration between the 2D and 3D understanding. To preserve original 2D priors, DINOv3D distills 2D knowledge into a 3D Gaussian regularization field, which aligns the rendered features with the reference visual features from a frozen teacher model. Meanwhile, a student model processes long-context multi-view inputs through parameter-efficient fine-tuning. This step injects 3D spatial consistency priors to 2D model while using the regularization field to mitigate the forgetting of 2D knowledge. To enrich the hierarchical representation of 3D spatial understanding, the student model predicts additional semantic and instance Gaussian features. We then apply a ray-depth-semantic alignment mechanism, which uses 3D depth-ray priors to enforce multiview consistency across 2D semantic rendering. Extensive experiments demonstrate that, despite updating only 10% of the parameters of a 1B-parameter foundation model VGGT, DINOv3D achieves state-of-theart results in comprehensive 3D scene understanding across challenging novel view synthesis, depth estimation, and open-vocabulary semantic segmentation. Furthermore, DINOv3D consistently enhances the generalizability of the DINOv3 backbone on 2D linear probing benchmarks.