An Inverse-Adversarial and Difficulty-Adaptive Robust Vision-Language Model

Vision-language models (VLMs) pre-trained on large-scaledatasets demonstrate strong generalization in few-shot scenarios, oftensurpassing conventional visual models on natural samples. However, theyremain vulnerable to adversarial perturbations, and directly applyingadversarial training may disrupt the cross-modal alignment, degradinggeneralization to novel classes. To mitigate these issues, we proposean Inverse-Adversarial and Difficulty-Adaptive Robust Vision-LanguageModel (IADA-RVLM) for robust few-shot classification. Specifically, toconstrain the optimization trajectory, our method jointly leverages ad-versarial and inverse-adversarial samples, where inverse-adversarial sam-ples serve as semantic anchors to guide representations toward high-confidence regions. To improve robustness, we incorporate visual prompttuning and text adapters to inject task-specific knowledge into bothmodalities. To balance robustness and generalization under varying sce-narios, we quantify adversarial transfer difficulty via semantic distancesand dynamically adjust the fusion of general and specialized knowl-edge. Extensive experiments across multiple datasets and attack settingsdemonstrate that our method achieves a superior overall trade-off be-tween robustness and generalization.