Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

Current Large Vision Language Models (LVLMs) excel atmany zero-shot tasks like image captioning, visual question answeringand OCR. However, these same models suffer from poor performance atimage classification tasks, underperforming against CLIP-based meth-ods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited byCLIP’s architecture with independent vision and text encoders. In CLIP,this separation biases classification toward class-name matching ratherthan joint visual–text reasoning. In this paper we show that, despite theirpoor raw performance, LVLMs can improve visual feature class separa-bility at inference using prompt conditioning, and LVLMs’ internal rep-resentations, especially attention heads, can outperform the model itselfat zero-shot and few-shot classification. We introduce Head EnsembleClassifiers (HEC) to bridge the performance gap between CLIP-basedand LVLM-based classification methods. Inspired by Gaussian Discrimi-nant Analysis, HEC ranks the most discriminative vision and text headsand combines them into a training-free classifier. We show that HECachieves state-of-the-art performance in few-shot and zero-shot classifi-cation across 12 datasets. Code: github.com/AdhemarDeSenneville/HEC