Stabilizing Real-World Visual Active Tracking with Action-Smooth Test-Time Adaptation

Visual Active Tracking (VAT) requires an embodied agent tocontrol cameras to follow a designated target, which is essential for ap-plications like robot navigation and security monitoring. However, trans-ferring VAT trackers to the real world faces two primary bottlenecks: (1)severe visual domain shifts between simulated environments and the realworld, leading to performance collapse during deployment; and (2) tar-get distribution shift caused by similar-looking distractors in real-worldscenes. To address these, we propose VATA, a novel test time adaptationmethod with two complementary strategies. First, we introduce a CriticValue Maximization strategy that leverages value signals from a criticmodel to filter reliable test samples, enabling unsupervised model up-dates to mitigate visual shift. Second, we propose an Action Smooth-ness Regularization term that exploits the continuity of real-worldtarget motion to bound action variations, correcting irrational actionsunder target shift. Experiments demonstrate VATA achieves a 24.4%SR improvement on the EVT-Benchmark and reaches 80% TSR in 10real-world scenarios (vs. 30% by the SOTA TTA method), providing arobust and plug-and-play solution for VAT deployment.