Variational Patch Gating for Training-Free Few-Shot Classification

Few-shot classification from a frozen backbone collapses when the target domain lies far from the pre-training distribution, because global embeddings map unrelated classes to nearly identical CLS tokens. Local patch descriptors are more resilient: an edge or texture gradient activates similar features whether it appears in a photograph or an X-ray. Yet part-level matching remains largely unexplored for few-shot recognition. The few methods that attempt it formulate matching as optimal transport or bipartite assignment, requiring iterative solvers whose cost grows rapidly with shots. We take a different approach. Instead of solving a matching optimization, we formulate patch-based classification as probabilistic evidence accumulation under a discriminative variational model. Each query patch carries a Bernoulli latent variable; minimizing the resulting assignment free energy yields a closed-form sigmoid gate and a softplus evidence score whose per-patch contribution saturates at −log(1−π), bounding clutter influence without iterative solvers or learned parameters. We evaluate extensively across twelve datasets spanning satellite, medical, fine-grained, and natural-image domains. Without any gradient updates, VPG outperforms the best fine-tuned baseline on five of ten dataset-settings and exceeds the previous training-free state of the art by +5.35 points on CDFSL. With lightweight episodic fine-tuning, VPG reaches 71.06% on CDFSL and 81.8% on four unseen Meta-Dataset domains.