Rethinking Attention Reallocation for Multimodal Emotion Recognition

The emergence of Multimodal Large Language Models hasenabled a new generative paradigm for multimodal emotion recognition(MER), where emotional interpretations are produced through multi-modal token-level reasoning. In view of multimodal inputs, existing stud-ies commonly observe that reallocating attention from generated tokensto modality tokens during inference consistently improves performance.However, in this work, we reveal a counterintuitive phenomenon: such astrategy only benefits two-modality settings, but degrades performancein three- or more-modality scenarios, where the opposite reallocation di-rection instead leads to improvements. To understand this phenomenon,through comprehensive analysis, we show that with more modalities, thecomplex interactions between modalities make the model hard to paymore attention to tokens with more information, especially in deeperlayers, leading to improper attention allocation and the observed contra-dictory behavior. Based on these insights, we propose a training-freeattention rectification method that leverages structured shallow-layerattention as a prior to regularize entangled final-layer attention dur-ing inference, without introducing additional parameters or modifyingthe backbone model. Extensive experiments on nine datasets in MER-UniBench demonstrate that our method achieves state-of-the-art per-formance, consistently outperforming both reallocation directions acrossdiverse multimodal scenarios. Our code is available at https://github.com/yzl77/ReAR.