Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

Despite advances in 3D scene understanding, existing 3DLarge Multimodal Models operate in offline settings, requiring completescene observations or predefined video clips. In this paper, we presentan online 3D vision-language model that enables real-time spatial un-derstanding from streaming video. Our approach adopts an autoregres-sive streaming control modeling based on the LLM’s next-token predic-tion objective to learn when to respond, and employs a lightweight Vi-sual–Spatial Feature Integration (VSFI) module to incrementally injecttemporally aligned geometry priors into the visual stream. To alleviatelong-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual tokencompression. To address the scarcity of streaming 3D–language data,we further develop a scalable data generation pipeline that curates over1M online spatio-temporal 3D QA pairs and establishes a comprehensivebenchmark spanning 29 tasks. Extensive experiments show that our ap-proach significantly outperforms both proprietary and open-source mod-els across online and offline 3D spatial understanding, reasoning, andgrounding tasks.