Label-Free Text Prototype Adaptation for Open Vocabulary Segmentation

Vision-language models (VLMs) have recently been adoptedfor open-vocabulary segmentation by aligning visual features with textprototypes. However, this alignment often degrades under domain shift,as text prototypes encode language priors learned from natural imagesthat may not hold in the target domain. To address this challenge, wepropose VPTA, a framework that adapts text prototypes to the tar-get domain to mitigate language prior bias using only unlabeled target-domain images. VPTA gathers reliable visual evidence from confidentpixels, estimates per-class evidence reliability to drive class-dependentprototype updates, and explicitly preserves prototype separation to pre-vent collapse between similar categories. This lightweight adaptationproceeds through a small number of iterations with negligible computa-tional overhead. We conduct extensive evaluations under diverse domainshifts across three representative domains, including remote sensing, au-tonomous driving, and natural scene segmentation. VPTA achieves state-of-the-art performance on ten remote sensing datasets and consistentlyimproves segmentation performance on autonomous driving and naturalscene benchmarks. These results demonstrate the strong generalizationability of VPTA and highlight its effectiveness as a practical solution forlabel-free prototype adaptation in vision-language models.