Probe, Anchor, and Amend: Active Test-Time Adaptation of Vision-Language Models

Vision-language models (VLMs) excel at zero-shot recogni-tion, yet their performance degrades under distribution shifts during in-ference. Unsupervised test-time adaptation (TTA) often relies on noisypseudo-labels or heuristics, offering limited guidance for where decisionboundaries should move; existing active TTA (ATTA) methods typi-cally treat annotations as local fixes rather than structured updates tothe shared semantics of VLMs. We introduce PAA (Probe, Anchor, andAmend), an ATTA framework that treats visual and textual prototypesas an evolving state and updates this state through a three-step cycle. Ineach cycle, the Probe step performs prototype-guided querying under aclass-balanced budget, selecting both central samples to preserve sourceknowledge and boundary samples to track distribution shifts. Using onlybinary verifications, the Anchor step applies a prototype-regularized up-date that couples cross-modal alignment with intra-class compactnessand inter-class separation, yielding a decision space that is stable yetplastic. As a slow controller, the Amend step revisits past hard casesstored in memory and converts temporal discrepancies into memory-driven self-refinement. Across VLM test-time benchmarks, PAA deliversconsistent gains over strong TTA and ATTA baselines under comparableannotation budgets and compute, turning sparse, delayed feedback intostructured evolution of the prototype-anchored decision space.