Towards Robustness against Typographic Attack with Training-free Concept Localization

Models trained via Contrastive Language-Image Pretrain-ing (CLIP) serve as the foundational vision encoders for many modernLarge Vision Language Models (LVLMs). Despite their widespread adop-tion, CLIP models exhibit a critical yet underexplored failure mode:irrelevant text appearing within images confounds visual representa-tions, biasing them toward lexical meaning rather than true visual se-mantics. This robustness issue, commonly described as a TypographicAttack (TA), exposes a vulnerability that poses a significant risk tosafety-critical applications such as autonomous driving. To achieve in-terpretable and effective robustness against TA, we propose a novel,training-free mechanistic interpretability method. Our method providessampling-based interpretations of hidden state representations and quan-titatively attributes semantic versus lexical focus to individual attentionheads. Through probabilistic analysis and circuit mining, we isolate spe-cific Vision Transformer (ViT) components that disproportionately en-code lexical information, thereby identifying the mechanistic source ofTA. We further show that simple interventions applied directly to theidentified circuits, without any additional training, can substantially im-prove robustness against Typographic Attacks in object classification.These interventions, such as selective adjustment of attention weights,outperform both supervised and training-free defense methods. Our ex-periments also demonstrate that applying the proposed intervention tothe vision encoders of several state-of-the-art LVLMs yields substantialgains in Visual Question Answering accuracy under Typographic At-tack interference on RIO-Bench. These results confirm both the efficacyand the generalizability of our mechanistic approach. Code is released athttps://github.com/Liu-524/SamplingTAR.