Locality-Aware Continual Unlearning for Diffusion Models

Real-world deployment of text-to-image diffusion models re-quires continual concept removal as new privacy, copyright, or safetyobligations arise over time. Existing unlearning methods, however, aredesigned for single-step deletion and collapse after only 3–5 sequentialapplications. We trace this instability to two compounding factors: (i)coarse mapping targets that cause degradation to accumulate unnec-essarily across steps, and (ii) the absence of local protection for se-mantically neighboring concepts, whose shared internal representationsmake them the first to suffer collateral damage. Because this damageis strongest in the local semantic neighborhood of the forget concept,global replay alone cannot prevent it. Building on this analysis, we pro-pose Locality-Aware Continual Unlearning (LACU), a frameworkwith two complementary mechanisms. Locality-Aware Target Selec-tion chooses, for each forget prompt, the context-preserving mappingprompt that the diffusion model itself treats as most similar to the orig-inal prompt, measured by score-prediction distance (how differently themodel denoises the same noisy image under two text conditions), ensur-ing each update is as small and targeted as possible. Locality-AwareReplay uses the same metric to identify the retain concepts closest to theforget concept in the model’s own representation and replays them as alocal functional regularizer, directly shielding the most vulnerable neigh-borhood. Combined with teacher-student distillation and lightweight ℓ2parameter regularization, LACU maintains stable unlearning over 10 se-quential steps, preserving significantly higher related retention (RRacc )and general retention (GRacc ) than recent baselines. The code is availableat https://github.com/SonyResearch/LACU.