Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

While test-time adaptation (TTA) empowers vision-languagemodels to adapt without costly retraining, it remains highly vulnerable toout-of-distribution (OOD) outliers prevalent in real-world applications.This discrepancy motivates Noisy TTA (NTTA), an online task to fil-ter noisy OOD samples on the fly while maximizing in-distribution (ID)classification accuracy. Existing zero-shot NTTA approaches typicallyrely on test-time discriminative training, leading to overconfident mis-classifications and significantly degraded inference efficiency. To addressthese limitations, we propose a novel framework named Dual Distribu-tion Estimation (DDE), shifting the zero-shot NTTA paradigm frominstance-level learning to training-free Gaussian distribution modeling.DDE incorporates two novel modules: Positive Feature Distribution Es-timation (PFDE) and Negative Label Distribution Estimation (NLDE).PFDE explicitly models class-wise inclusion and exclusion Gaussian dis-tributions to formulate a calibrated contrastive score, robustly enhancingID accuracy. In parallel, NLDE improves OOD identification by explicitlymodeling the negative label distribution to mine highly discriminative la-bels, effectively mitigating spurious correlations. Extensive experimentsshow that on the large-scale ImageNet benchmark, DDE achieves an im-provement of 3.70% in harmonic mean accuracy and reduces the FPR95for OOD detection by 6.20%, while ensuring highly scalable and effi-cient online inference. Furthermore, DDE is zero-shot and training-free,demonstrating remarkable robustness in data-scarce scenarios. Codes areavailable at https://github.com/PolyU-VCLab/OpenOOD-VLM.