ORACLE-3D: Open-world Region-aligned Cross-modal Learning for Label-efficient 3D Scene Understanding

The recent surge in vision-language models (VLMs) has revolutionized open-world perception, yet transferring this success to 3D scene understanding remains impeded by the scarcity of large-scale point-text pairs and the computational overhead of existing frozen-VLM approaches. To bridge this gap, we present ORACLE-3D (Open-world Region-Aligned Cross-modal LEarning for Label-efficient 3D Scene Understanding), a unified framework that effectively co-embeds point clouds, images, and text into a shared latent space without relying on labor-intensive 3Dtext pair construction. ORACLE-3D leverages the image modality as a semantic bridge, employing a novel cross-modal distillation strategy that comprises logit distillation and feature distillation. These components transfer the rich, open-vocabulary capabilities of pre-aligned 2D VLMs directly to the 3D domain. Furthermore, to mitigate the semantic noise arising from inevitable calibration errors and motion artifacts during 3D-2D projection, we propose a vision-point matching module that dynamically rectifies point-pixel misalignments. To ensure robust convergence amidst the gradient bias inherent in heterogeneous multimodal learning, we introduce a two-stage optimization strategy coupled with four task-specific weighted losses. Extensive evaluations on the nuScenes, Waymo, SemanticKITTI, and ScanNet datasets demonstrate the superiority of our paradigm. Specifically, ORACLE-3D outperforms state-of-the-art methods by an average of 11.6 and 14.8 in mIoU on base-annotated and annotation-free tasks, respectively. Project Page: https://ocean-luna.github.io/ORACLE.github.io/.