MessyKitchens: Contact-rich object-level 3D scene reconstruction

Monocular 3D scene reconstruction has recently seen signif-icant progress. Powered by the modern neural architectures and large-scale data, recent methods achieve high performance in depth estimationfrom a single image. Meanwhile, reconstructing and decomposing com-mon scenes into individual 3D objects remains a hard challenge due tothe large variety of objects, frequent occlusions and complex object rela-tions. Notably, beyond shape and pose estimation of individual objects,applications in robotics and animation require physically-plausible scenereconstruction where objects obey physical principles of non-penetrationand realistic contacts. In this work we advance object-level scene re-construction along two directions. First, we introduce MessyKitchens,a new dataset with real-world scenes featuring cluttered environmentsand providing high-fidelity object-level ground truth in terms of 3D ob-ject shapes, poses and accurate object contacts. Second, we build onthe recent SAM 3D approach for single-object reconstruction and ex-tend it with Multi-Object Decoder (MOD) for joint object-level scenereconstruction. To validate our contributions, we demonstrate MessyK-itchens to significantly improve previous datasets in registration accuracyand inter-object penetration. We also compare our multi-object recon-struction approach on three datasets and demonstrate consistent andsignificant improvements of MOD over the state of the art. Our newbenchmark, code and pre-trained models are publicly available on ourproject website: https://messykitchens.github.io/.