EgoPHI: Estimating 3D Hand-Object Contact and Force from Egocentric Vision

Understanding hand-object interaction from egocentric vi-sion is essential for modeling how people physically engage with thesurrounding world. Yet reasoning about physically grounded interactionrequires estimating the forces acting on hands and objects, beyond lo-calizing contact. We present EgoPHI, the first method that jointly esti-mates dense contact maps and 3D force distributions on hand and objectmeshes from a single egocentric RGB image and object geometry. To ad-dress the lack of scalable ground-truth force annotations, we introducea physics-based simulation pipeline that augments existing hand-objectdatasets with dense per-vertex force supervision. EgoPHI then learnsdense 3D contact and force on interacting hand and articulated objectmeshes, extending vision-based force estimation beyond image-space orplanar settings. Our evaluation on in-distribution and out-of-distributionbenchmarks shows that EgoPHI improves force estimation over existingapproaches while generalizing to unseen datasets. To evaluate sim-to-realtransfer, we constructed two physical objects that capture dense objectcontact and force magnitude and used them to record a dataset of in-teractions from eight participants across diverse touch and grasp types.Our results demonstrate that EgoPHI recovers meaningful 3D contactand force distributions in simulated, out-of-distribution, and real-worldsettings, advancing egocentric hand-object understanding from contactlocalization toward physically grounded interaction reasoning.Code: https://github.com/eth-siplab/EgoPHI