Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D

Detecting and localizing objects in space is a fundamentalcomputer vision problem. While much progress has been made to solve2D object detection, 3D object localization is much less explored and farfrom solved, especially for open-world categories. To address this researchchallenge, we propose Boxer, an algorithm to estimate static 3D boundingboxes (3DBBs) from 2D open-vocabulary object detections, posed images,and optional depth represented either as a sparse point cloud or densedepth. At its core is BoxerNet, a transformer-based network that lifts2D bounding box (2DBB) proposals into 3D, followed by multi-viewfusion and geometric filtering to produce globally consistent de-duplicated3DBBs in metric world space. Boxer leverages the power of existing 2DBBdetection algorithms (e.g., DETIC [52], OWLv2 [29], SAM3 [8]) to localizeobjects in 2D. This allows the main BoxerNet model to focus on liftingto 3D rather than detecting, ultimately reducing the demand for costlyannotated 3DBB training data. Extending the CuTR [20] formulation, weincorporate an aleatoric uncertainty for robust regression, a mean depthpatch encoding to support sparse depth inputs, and large-scale trainingwith over 1.2 million unique 3DBBs. BoxerNet outperforms state-of-the-art baselines in open-world 3DBB lifting, including CuTR in egocentricsettings without dense depth (0.532 vs. 0.010 mAP) and on CA-1M withdense depth available (0.412 vs. 0.250 mAP). Project page and code areavailable at https://facebookresearch.github.io/boxer.