Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

Recently, self-evolving large multimodal models (LMMs) havereceived attention for improving visual reasoning in a purely unsuper-vised setting. However, multi-role self-play and self-consistency rewardschemes in existing self-evolving LMMs optimize answer agreement with-out ensuring the decoder attends to visual content, relying instead onstatistical language priors to produce self-consistent outputs. This leadsto a persistent failure mode we term visual under-conditioning, wherethe decoder relies on language priors rather than the image during gen-eration, manifesting as insufficient attention to visual tokens. As a result,current self-evolving LMMs struggle on vision–language understandingtasks such as image captioning and visual question answering. To addressthis, we propose VISE (Visual Invariance Self-Evolution), a purely un-supervised self-evolving framework that directly regularizes the model’svisual conditioning policy through two complementary invariance-basedrewards: a geometric invariance reward that enforces spatial consistencyunder known transformations, and a semantic invariance reward thatpenalizes evidence-agnostic generation by requiring the model to recog-nize the absence of evidence when predicted regions are perturbed. VISEoperates within a single model without specialist roles, external rewardmodels, or annotations, and is trained on raw unlabeled images. Ex-periments on 18 benchmarks demonstrate the efficacy of our approach.Using Qwen3-VL-2B as the base model, VISE achieves gains of +16.85CIDEr on COCO and +19.66 CIDEr on TextCaps, reduces object hal-lucination by 5.0 Chair-I points, and generalizes across four model fam-ilies and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE/.