Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention

We introduce a novel learning problem: decoding gaze intonatural language descriptions of human goals across diverse visual tasks.Unlike prior work, which frames gaze decoding as a discriminative taskover predefined categories, we formulate it as a generative learning prob-lem: training a model to produce free-form descriptions that capture therich nuances and open-ended nature of human intentions beyond fixedlabels. To this end, we introduce Gazette, the first gaze-to-text decod-ing framework. Based on multimodal large language models (MLLMs),Gazette learns to decode gaze scanpaths into natural language for goalsthat may extend beyond categorical labels and require articulation innatural language. To help Gazette filter out individual differences in gazebehavior and learn the goal-specific spatiotemporal dynamics crucial forgenerating accurate natural language goal descriptions, we propose anovel strategy that leverages the encyclopedic knowledge and reason-ing abilities of a large language model to synthesize natural languageexplanations of goal-directed attentional behavior called think-aloud tran-scripts. Instruction tuning on these synthetic narratives allows Gazetteto achieve state-of-the-art performance in gaze decoding across mul-tiple tasks, demonstrating its generalizability and versatility, therebyenabling gaze to serve as a powerful, non-intrusive cue for inferringhuman goals and intentions in diverse scenarios. Code is available athttps://github.com/cvlab-stonybrook/Gazette.