CS-TTA: Preserving Concept Sensitivity in Test-Time Adaptation

Test-time adaptation (TTA) methods are typically evaluated by task accuracy, leaving open the question of whether adaptation also changes which features the model relies on. We examine this question by tracking concept sensitivity throughout adaptation using TCAV directional derivatives. Across five TTA methods and five benchmarks spanning natural image and medical imaging domains, we observe recurring shifts in concept sensitivity: adaptation can reduce sensitivity to task-relevant or pathology-related concepts while increasing sensitivity to nuisance concepts such as backgrounds or institutional artifacts—a phenomenon we term Concept Sensitivity Drift, even when accuracy improves. On Waterbirds, where the causal/spurious split is defined by construction, this drift coincides with degraded worst-group accuracy; on cross-hospital medical adaptation, it coincides with increased reliance on hospital-specific artifacts rather than pathology. Motivated by this observation, we propose CS-TTA, a source-concept-supervised but target-label-free plug-in regularizer that mitigates excessive conceptsensitivity drift during adaptation. CS-TTA requires no architectural changes and can be added to any existing TTA objective. Experiments show that CS-TTA provides consistent worst-group accuracy gains on spurious-correlation benchmarks, improves AUC by 1.4–1.8 points on cross-hospital medical adaptation, and gives consistent overall accuracy gains on CIFAR-10-C and ImageNet-C.1234