SAMPLe: A Sharpness Aware Minimization based Optimizer for Prompt Learning in Vision-Language Models

Pre-trained Vision-Language Models (VLMs) like CLIP haveproven highly effective as foundation models for various downstream ap-plications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve highaccuracy on seen distributions, this tuning process often undermines theirgeneralizability to unseen data. The limited set of learnable prompts,which contextualize and condition the input to steer it toward the taskwithin the pretrained VLM, tends to overfit the training data, leading toa trade-off between task-specific performance and preserving generaliza-tion. To address this dilemma, we introduce SAMPLe (Sharpness-AwareMinimization Prompt Learning), a plug-in sharpness-aware optimizerthat enhances prompt generalizability by accounting for loss landscapesharpness. Unlike conventional methods, SAMPLe balances explorationand exploitation by satisfying objective function constraints at each step,dynamically adapting to the current optimization state based on the localcurvature and gradient properties. This approach reduces overfitting onseen distributions and improves adaptability to unseen data, preservingthe generalization potential of pre-trained VLM models. We integrateSAMPLe into multiple prompt learning frameworks, including CoOp,CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectivenessacross diverse methods. Experiments show that SAMPLe elevates promptlearning frameworks and consistently outperforms existing optimizersacross diverse settings, establishing itself as a robust, model-agnosticsolution for prompt learning.