PACO: Stabilizing Vision Embeddings along Local Paths for Robust Vision-Language Models

Large vision-language models with CLIP as a core compo-nent have achieved remarkable progress across a wide range of tasks,yet they remain highly vulnerable to adversarial attacks. Existing ad-versarial fine-tuning methods typically optimize CLIP under a singlefixed perturbation strength, resulting in weak robustness generalizationand semantic instability. To address this limitation, we propose Path-Consistent Fine-Tuning (PACO), a new framework for unsupervised ad-versarial fine-tuning. Rather than optimizing solely on adversarial exam-ples, PACO adopts a two-stage procedure that regularizes representa-tions on continuous local paths along adversarial directions. Specifically,it first anchors adversarial embeddings to a clean reference to preventsevere semantic drift. Building upon this, it constructs a local path be-tween the clean sample and the adversarial anchor, regularizing inter-mediate representations to encourage a linear transition, which yields amore stable visual embedding. Extensive experiments show that PACOachieves a superior robustness-accuracy trade-off: it not only mitigatesthe clean-performance degradation common to prior methods, but alsodelivers superior robustness across diverse attack settings and perturba-tion strengths. Our code is available at https://github.com/Trusted-LLM/PACO.