The Language of Visual Attention: Modeling Scanpaths via Autoregressive Token Prediction

Understanding human visual attention on a scene over timehas applications in domains such as autonomous driving, interface de-sign, inferring cognitive states. Modeling human visual scanpaths has his-torically relied on specialized architectures with hand-crafted geometricpriors. While these architectures successfully model x001Cxation sequences,their rigid structural biases restrict easy extendability and x001Dexible con-ditioning. For instance, integrating task-specix001Cc instructions or adaptingto distinct viewer identities traditionally requires custom, disjoint ar-chitectural additions. We address this limitation by framing scanpathprediction purely as a discrete sequence modeling task. By mapping spa-tial coordinates into an autoregressive vocabulary, we leverage the pre-trained representations of Large Vision-Language Models (LVLMs). Thisformulation naturally absorbs diverse factors of variation: simple prompt-ing modix001Ccations allow for global conditioning, such as providing vieweridentities to capture personalized biases, or task-specix001Cc objectives (likefree-viewing versus visual search). The framework can also easily inte-grate the modeling of per-x001Cxation attributes, such as individual x001Cxa-tion durations alongside spatial locations. Crucially, this autoregressivealignment enables the scalable, exact computation of per-x001Cxation log-likelihoods, directly equivalent to the commonly used Information Gain(IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003,a 46% improvement over matched-backbone DeepGaze III. This advan-tage persists even when baselines use identical high-capacity vision en-coders. Beyond predictive performance, our generative framework servesas a powerful computational tool for direct behavioral interventions, al-lowing for controlled in-silico simulations that would be experimentallydix001Ecult or impossible to conduct in vivo. We demonstrate this abilityby performing controlled interventions on the durations of pre-saccadicx001Cxations, recovering known oculomotor phenomena purely from data.