EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding

We introduce EgoSAT, the first comprehensive benchmarkfor egocentric video reasoning in streaming settings, designed to evaluatethe capabilities of modern vision–language models (VLMs). The bench-mark targets streaming interaction understanding, where video framesarrive sequentially and models must continuously interpret evolving vi-sual context. EgoSAT unifies several previously distinct tasks within asingle streaming framework. In this formulation, queries about completedevents correspond to retrospective reasoning, queries about ongoing ac-tivities require online understanding, and queries about future actionsinvolve prospective anticipation. This unified setting requires models toreason about the past, present, and future while operating under theconstraint that only previously observed frames are available. EgoSATcontains 1,997 unique videos spanning 165 hours of egocentric footageand around 4,800 high-quality question–answer pairs, carefully designedto probe reasoning across varying temporal contexts. Using this bench-mark, we evaluate a diverse set of both open-weight and closed-weightVLMs, providing a systematic assessment of their ability for streaminginteraction understanding. By distinguishing answerability and conduct-ing diagnostics on confidence of models, we find existing models not onlystruggle with prospective and retrospective modeling, but also exhibit se-vere mis-calibration: confidence often fails to track inherent answerabil-ity, leading to dangerous “confidently wrong” behaviors. Project page:https://leiyj23.github.io/EgoSAT/