Beyond Disjoint Tasks: Towards More Natural Continual Learning for Vision-Language Models

Continual learning methods for vision-language models are developed on benchmarks where each new task introduces entirely new domain knowledge. Real-world task sequences are more natural: they routinely share visual concepts, language patterns, and even training samples across stages. However, existing mixture-of-expert methods that assign one expert per task with fixed routing can split similar inputs across different experts and degrade performance. We introduce Semantic Overlap-aware Continual Learning (SOCiaL), a simple framework designed to identify and leverage shared structure across tasks. SOCiaL employs a Gaussian mixture model to estimate contextual similarity, generate synthetic replay samples, and guide expert routing. When task contexts are highly similar, their adapters are consolidated and the router is updated accordingly. To study this more realistic continual learning scenario, we also present UCIT-O, a new benchmark with three protocols that progressively increase semantic similarity across tasks. Across both disjoint and overlapping benchmarks, SOCiaL consistently outperforms existing methods, achieving 7.35, 8.04, and 3.07–9.77 points above the strongest baseline on CoIN, UCIT, and UCIT-O while reducing deployed adapters by up to threefold.