ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision–Language Models

Spatial reasoning remains a persistent challenge for manyvision–language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysisreveals that rescaling activations in selected transformer layers—withoutmodifying pretrained weights—can significantly influence downstreamperformance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coeffi-cients to modulate last-token attention and MLP activations in a fullyfrozen backbone. We evaluate our method on the synthetic spatial rea-soning benchmark SpatialEval and on real-world VQA datasets (CO-COQA and VGQA) across multiple model families. Our method, ScAle,achieves up to 134.1% relative accuracy gains using only 1K train-able parameters without requiring millions of trainable parameters as instandard LoRA. Despite its extreme compactness, our approach recov-ers a substantial fraction of standard PEFT performance while preserv-ing strong non-spatial VQA accuracy. These results demonstrate thatbounded activation reweighting provides a simple, architecture-agnostic,and highly parameter-efficient alternative for adapting pretrained VLMs.Code is available at https://github.com/rchowdhubnor/ScAle.git.