Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning

Vision-Language Models (VLMs) have demonstrated remark-able success across diverse visual tasks, yet their performance degradesin complex visual environments. Existing enhancement approaches re-quire additional training, rely on external segmentation tools, or operateat coarse-grained levels, overlooking VLMs’ innate attention capabilities.To bridge this gap, we investigate VLMs’ attention patterns and discoverthat: (1) visual complexity strongly correlates with attention entropy,negatively impacting reasoning performance; (2) attention progressivelyrefines from global scanning in shallow layers to focused convergencein deeper layers, with the degree of convergence determined by visualcomplexity; (3) theoretically, under a multiplicative decomposition as-sumption, we show that contrasting attention maps between general andtask-specific queries approximately separates visual signal into seman-tic and visual noise components. Building on these insights, we proposeContrastive Attention Refinement for Visual Enhancement (CARVE),a training-free method that extracts task-relevant visual signals throughattention contrasting at the pixel level. Experiments on seven bench-marks show that CARVE improves performance on visual perceptiontasks, with notable gains on both recent and earlier-generation models.Our analysis reveals how visual complexity affects attention mechanismsand demonstrates an effective strategy for improving visual reasoningthrough attention contrasting.