Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Test-time adaptation (TTA) aims to mitigate distributionshifts by adapting models with unlabeled target data at inference time.While TTA with vision-language models (VLMs) has shown promisingresults in classification, extending it to medical image segmentation re-mains challenging. In this setting, the adaptation gains from optimizingon VLM-generated predictions are often outweighed by the degradationto the VLM’s strong pretrained features caused by noisy, update-drivenlearning, resulting in limited and unstable improvements. We thereforepropose Memory-Supported Synergistic Adaptation (MSSA), anovel training-free TTA framework for medical image segmentation.Without updating model parameters, MSSA dynamically selects reli-able image–text predictions to construct an online memory, uses themas text-guided semantic priors, and couples them with cross-image struc-tural alignment for robust adaptation. Specifically, MSSA consists of (i) anoise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment mod-ule that aligns the target sample with structurally consistent memorysamples and their reliable predictions to improve adaptation. Extensiveexperiments on multiple medical segmentation benchmarks demonstratethat MSSA consistently improves VLM-based segmentation models andoutperforms existing fine-tuning-based TTA methods by a clear mar-gin, with gains of up to 12.2% DSC and 11.7% mIoU. Project Page:https://lingrayy.github.io/MSSA/.