Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

Estimation of Human-Object Interactions in 3D (3D HOI)is a fundamental problem in 3D computer vision with applications inAR/VR, robotics, and embodied AI. However, reconstructing these in-teractions in 3D remains challenging due to depth ambiguities, occlu-sions, and object shape variability. Existing approaches are primarilyconcerned with reprojection and contact constraints, fitting parametrichuman models and object templates to 2D images. In this paper, weexplore a different avenue. We present MILO, a framework that lever-ages the visual capabilities of Large Reconstruction Models (LRMs) torecover detailed 3D human-object interactions from a single image. Ourkey observation is that LRMs provide a powerful geometric scaffold thatpreserves relative human-object arrangement and proximity cues. Thissignificantly simplifies the reconstruction procedure, reframing the prob-lem as interpreting the LRM mesh: we segment it into human and objectcomponents, fit a parametric body model to the human part, and op-tionally align an object template to the object part (if such a templateis available). MILO achieves strong reconstruction accuracy and out-performs existing baselines across multiple benchmarks and interactionscenarios. Our code is available at https://ac5113.github.io/MILO.