DnA: Denoising Attention for Visual Tasks

The softmax activation in multihead attention (MHA) is thede facto standard for attention-based models in visual perception tasks.However, standard softmax can produce noisy attention patterns thatdilute relevant features and degrade its performance. In this paper, wepropose Denoising Attention or DnA, in which, first, a positive queryidentifies which image features belong to the correct class, and a neg-ative query identifies closely associated but irrelevant image features.DnA then projects these interactions into two distinct subspaces withlarger principal angles, promoting subspace separation and improved dis-criminability. Using a ViT-B backbone, our proposed DnA achieves anabsolute gain of 0.8% on ImageNet-1K compared to the baseline. Wefurther show improvements across multiple visual understanding tasks,including video understanding with video transformers (1.8%) and videoLLMs (0.5%). Our extensive empirical analyses justify the design choicesinvolving two interacting subspaces and the denoising effect of DnA. Thecode is publicly available at https://github.com/rjccv/DnA.