Ego-Human Motion Prediction with 3D-Aware LLM

Anticipating human motion from an egocentric perspectiveis fundamental for proactive assistance in AR/VR, human-robot collab-oration, and embodied AI. While recent works incorporate language asa semantic prior to reduce the ill-posed nature of egocentric forecasting,they largely neglect the 3D spatial and semantic context that governshow motion unfolds, and treat pose and language prediction as separateinference streams. We introduce Ego3DLM, built on two core principles:accurate motion forecasting requires explicit spatial and semantic un-derstanding of the 3D environment, and pose and language must be pre-dicted holistically in a single pass, since motion is inherently tied to thesemantic interpretation of actions being performed. Given three-pointtracking, 3D scene features, and egocentric video, Ego3DLM simultane-ously decodes past pose, future pose, past narration, and future narrationin a single autoregressive pass, grounding predicted poses and descrip-tions in one another to enforce cross-modal and temporal consistency. Weadopt a three-stage training scheme: (1) spatial-semantic scene aware-ness pretraining; (2) holistic instruction tuning over all four outputs ina single pass; and (3) GRPO-based reinforcement finetuning with intra-and inter-modal rewards that directly optimize pose-language fidelity.Experiments on the Nymeria benchmark demonstrate that Ego3DLMachieves state-of-the-art performance across future motion prediction,past motion tracking, and motion description, showing that 3D scenegrounding and holistic cross-modal prediction yield physically plausibleand semantically coherent motion forecasts. The project page is availableat https://jaewoo97.github.io/Ego3DLM/.