Towards Real-World Wearable Motion Reconstruction

The modern-day surge in popularity of wearable devices posesa fundamentally unique motion capture problem: reconstructing full-body movement from any set of sensing hardware worn at a given mo-ment. Yet, most research efforts assume fixed sensor configurations (e.g.,IMU suits or HMD-centric rigs) and cannot generalize across them.In contrast, we argue that motion capture should prioritize unobtru-sive and lightweight devices such as smartphones, smartwatches, smartglasses, and smart insoles, and study the interplay between them. To thisend, we make three contributions. First, we present a large-scale multi-modal dataset synchronizing these consumer-grade sensors with ground-truth 3D motion, spanning 50 diverse activities including everyday tasks,sports, and social interactions. Second, we propose WHIP, a baseline gen-erative model that reconstructs motion from arbitrary subsets of avail-able sensors, robustly handling missing modalities and producing physi-cally plausible motions. Third, we conduct a systematic study of sensorcomplementarity, quantifying how different modalities complement oneanother. Code and dataset are available at this URL.