Shared LoRA Subspaces for almost Strict Continual Learning

Adapting large pretrained models to new tasks e!ciently andcontinually is crucial for real-world deployment but remains challengingdue to catastrophic forgetting and the high cost of retraining. Whileparameter-e!cient tuning methods like low-rank adaptation (LoRA) re-duce computational demands, they lack mechanisms for strict contin-ual learning and knowledge integration, without data replay or multi-adapters or MoE. We propose Share, a novel approach to parameter-e!cient continual finetuning that learns and dynamically updates a sin-gle, shared low-rank subspace, enabling seamless adaptation across multi-ple tasks and modalities. Share constructs a foundational subspace thatextracts core knowledge from past tasks and incrementally integratesnew information by identifying essential subspace directions. Knowledgefrom each new task is incorporated into this evolving subspace, enablingforward knowledge transfer while minimizing catastrophic interference.This approach achieves up to 100× reduction in trainable parametersand 281× memory savings over traditional LoRA methods, maintain-ing performance comparable to jointly trained models. A single Sharemodel can replace hundreds of task-specific LoRA adapters, support-ing scalable, asynchronous continual learning. Experiments across imageclassification, natural language understanding, 3D pose estimation, andtext-to-image generation validate its e"ectiveness, making Share a prac-tical and scalable solution for lifelong learning in large-scale AI systems.