Interference-Aware Continual Vision–Language Learning via Instance-Level Expert Routing

Continual vision–language learning is increasingly required in real deploymentswhere data, skills, and objectives drift, yet models still struggle to learn new tasks withouterasing old ones and to act reliably in task-agnostic settings with limited memory. We ad-dress this gap by turning cross-task conflict into an instance-level signal that drives how themodel adapts over time. We propose an interference-aware adapter routing framework thatestimates, for each sample, a metric-consistent projection energy to quantify potential inter-ference, routes to the least interfering experts with capacity balancing, enlarges inter-expertseparation via principal-angle packing, and grows LoRA rank only along the residual princi-pal direction when current experts are insufficient. Across classification, structured conceptmatching, generative VQA, and retrieval, our approach improves over strong baselines. Anal-yses show that the conflict signal reliably predicts downstream forgetting, packing widensprincipal angles over time, and rank growth is sparse yet beneficial—especially in mid–highlayers.