Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation

Test-Time Adaptation (TTA) has emerged as an effectivesolution for adapting Vision Transformers (ViT) to distribution shiftswithout additional training data. However, existing TTA methods oftenincur substantial computational overhead, limiting their applicability inresource-constrained real-world scenarios. To reduce inference cost, plug-and-play token aggregation methods merge redundant tokens in ViTsto lower computation demands. Albeit efficient, it suffers from signif-icant performance degradation when directly integrated with existingTTA methods. We formalize this problem as Efficient Test-Time Adap-tation (ETTA), seeking to preserve the adaptation capability of TTAwhile reducing inference latency. In this paper, we start by providingan analysis showing that token aggregation inherently leads to informa-tion loss, which cannot be fully mitigated by conventional norm-tuning-based TTA methods. Guided by this insight, we propose to NeutralizeToken Aggregation via Information Augmentation (NAVIA). Specif-ically, we propose an information augmentation mechanism that em-ploys an input-level [CLS] embedding augmentation to compensate fordomain-level information loss, and a feature-level [CLS] bias augmen-tation for fine-grained layerwise information compensation. Theoreticalanalysis provides a principled motivation that augmenting a global infor-mation carrier before aggregation can improve the learnable-informationupper bound, with entropy minimization serving as a practical surro-gate objective. Extensive experiments across various out-of-distributionbenchmarks and model backbones demonstrate that NAVIA outperformsall competing methods. Notably, NAVIA improves performance by upto 3.1% while achieving 14% to 26% wall-clock speedup over the TTAstate-of-the-art, effectively addressing the ETTA challenge. We provideour official implementation at https://github.com/Bostoncake/NAVIA.