Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

Vision Language Models (VLMs) have achieved strong per-formance across diverse video understanding tasks. However, their view-point invariant training limits their ability to understand egocentricproperties (e.g., human-object interactions) from exocentric video ob-servations. This limitation is critical for applications such as Activi-ties of Daily Living (ADL) monitoring, where understanding egocen-tric properties is essential yet egocentric cameras are impractical to de-ploy, making it impossible to simply collect egocentric data at test time.To address this challenge, we propose Ego2ExoVLM, a VLM frame-work that learns to infer egocentric properties from exocentric videos.Our key insight is that time-synchronized ego-exo video pairs can beleveraged during training, where the egocentric viewpoint provides priv-ileged supervision (rich egocentric signal available only at training time).Ego2ExoVLM achieves this through two components: Ego2Exo Se-quence Distillation, which transfers egocentric reasoning through alanguage-level sequence distillation objective, and Ego Adaptive Vi-sual Tokens, which encourages the model to surface relevant interac-tion cues within exocentric video representations. To measure this ca-pability, we introduce Ego-in-Exo Perception, a benchmark designedto evaluate the understanding of egocentric properties from exocentricvideos. Ego2ExoVLM is evaluated on 10 tasks across Ego-in-Exo Per-ception and existing ADL benchmarks, achieving state-of-the-art resultson the ADL-X benchmark suite and outperforming strong baselines onour proposed benchmark. Code, models, and data will be released athttps://github.com/dominickrei/EgoExo4ADL.