Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs

Open-vocabulary 3D scene graph methods typically operatein two stages: first reconstruct, then enrich with vision-language mod-els, leaving the graph unqueryable during exploration. We argue thatthis sequential coupling is unnecessary and propose an asynchronous ar-chitecture in which lightweight online mapping runs concurrently withheavyweight semantic refinement. A probabilistic voxel-based backbonemaintains stable object identities incrementally, while background VLMagents progressively enrich the graph. This framework resolves duplicateobject tracks through semantic loop closure, attaches fine-grained visualattributes and derives spatial relations between objects. A multi-targetframe scheduler amortizes VLM cost by selecting a small set of infor-mative frames that jointly cover multiple targets. The resulting scenegraph is queryable during exploration and grows in semantic richnessover time. Our method matches or outperforms existing open-vocabulary3D scene graph methods on semantic segmentation (ScanNet, Replica)and surpasses the prior state-of-the-art across three visual groundingbenchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25. Projectpage: https://denizbickici.github.io/thinkgraphs/