SiGMA: Sign-Guided Merging and Adaptation framework for Multimodal Continual Instruction Tuning

Multimodal Continual Instruction Tuning (MCIT) is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving a sequence of downstream tasks. Prior methods mostly utilize Mixtureof-Experts or expansion–merge approach, primarily focusing on catastrophic forgetting, yet they still suffer from negative interference during inference, where newly learned updates overwrite useful prior knowledge and degrade overall performance. To address this, we propose SiGMA (Sign-Guided Merging and Adaptation), a simple yet effective framework that mitigates negative interference with two components: signguided adaptive tuning during training and sign-guided merging at inference. Sign-guided adaptive tuning reduces collisions with past knowledge and learns the current task with minimal drift, mitigating severe forgetting. Sign-guided merging further improves consolidation by selectively scaling salient parameters to preserve and amplify useful taskspecific knowledge. Experiments on UCIT and DCL benchmarks show that SiGMA significantly reduces negative interference and outperforms state-of-the-art MCIT methods. Our code is available at SiGMA.