Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers

We present Point2Pose, a model-free method for causal 6Dpose tracking of multiple rigid objects from monocular RGB-D video.Initialized only from sparse image points on the objects, our approachtracks multiple unseen objects without requiring object CAD modelsor category priors. Point2Pose leverages a 2D point tracker to obtainlong-range correspondences, enabling instant recovery after complete oc-clusion. Simultaneously, the system incrementally reconstructs an on-line Truncated Signed Distance Function (TSDF) representation of thetracked targets. Alongside the method, we introduce a new multi-objecttracking dataset comprising both simulation and real-world sequences,with motion-capture ground truth for evaluation. Experiments show thatPoint2Pose trades some single-object pose accuracy for broader model-free tracking capabilities, including multi-object tracking and recoveryfrom complete occlusion. Project page: https://point2pose.github.io/.