SCoT: Similarity-guided Conflict-aware Task Consolidation for Continual VQA

Continual learning in visual question answering (VQACL) re-quires a single vision–language model to acquire new multimodal reason-ing skills from a task stream while retaining prior capabilities. However,naïve sequential finetuning suffers from catastrophic forgetting. Existingcontinual VQA methods primarily rely on replay or parameter regulariza-tion but largely overlook how task-specific updates accumulate and inter-act in parameter space, particularly whether successive updates are syn-ergistic or conflicting across layers. To address this, we introduce SCoT(Similarity-guided Conflict-aware Task Consolidation), a continual learn-ing framework that represents each task as a parameter update relative to apretrained anchor model and integrates tasks through layer-wise parameter-space reasoning. For each layer, SCoT measures alignment between incom-ing and accumulated task vectors, removes only destructive components viaconditional projection when conflicts arise, and adaptively modulates con-solidation strength using similarity-guided weighting. This preserves ben-eficial transfer while suppressing harmful interference, enabling stable yetadaptive continual learning. Experiments on VQAv2 and NExT-QA demon-strate strong continual VQA performance, reducing forgetting to near-zero(0.07 and -1.90) while achieving rare positive backward transfer (+5.64 and+6.97), outperforming strong continual-learning and task-vector baselines.Project page: https://anand-patel05.github.io/SCoT