Online 3D Instance Segmentation at task-oriented granularity with Unposed Monocular Video

We present a real-time, task-oriented 3D instance segmenta-tion framework for unposed monocular video, enabling embodied agentsto task-adaptively perceive and interact with objects in open-world scenes.Unlike most previous bottom-up segmentation paradigms that segmentbefore recognition, we adopt a task-oriented segmentation approach.Specifically, objects are decoupled within each frame using an open-vocabulary detector combined with a prompt-based 2D segmentationmodel, while the 3D underlying geometry of the scene is simultane-ously being reconstructed using a modern dense SLAM system, whosecontinuously re-optimized camera poses and depth are unsuitable forincremental mask association. Guided by the SLAM-derived pose graph,we selectively associate multi-view masks and reuse the dense correspon-dences provided by the SLAM system, incrementally converting theminto geometric association scores with minimal additional computation.By incorporating semantic similarity and mutual exclusivity metrics, wedesign a priority-ordered mask clustering algorithm for efficient onlinemulti-view mask matching and merging. Evaluations on open-vocabulary3D instance segmentation benchmarks show that our method effectivelymitigates the performance degradation of existing approaches when usingdense SLAM reconstructions instead of depth-sensor point clouds. Onthe Replica dataset, using only unposed images, it even achieves resultscomparable to methods leveraging ground-truth depth and poses. Moreimportantly, qualitative results show that our method achieves more reli-able task-oriented 3D object disentanglement than previous bottom-upsegmentation paradigms.