Cross-Domain Semantic Segmentation on Inconsistent Taxonomy using VLMs

The challenge of semantic segmentation in Unsupervised Domain Adaptation (UDA) emerges not only from domain shifts between source and target images but also from discrepancies in class taxonomies across domains. Traditional UDA research assumes consistent taxonomy between the source and target domains, thereby limiting their ability to recognize and adapt to the taxonomy of the target domain. This paper introduces a novel approach, Cross-Doamin Semantic Segmentation on Inconsistent Taxonomy using Vision Language Models (CSI), which effectively performs domain-adaptive semantic segmentation even in situations of source-target class mismatches. CSI leverages the semantic generalization potential of Visual Language Models (VLMs) to create synergy with previous UDA methods. It utilizes segment reasoning obtained through traditional UDA methods, alongside the rich semantic knowledge embedded in VLMs, to perform relabeling to classes of the target domain. This approach allows for effective adaptation to changed taxonomies without requiring any ground truth label for the target domain. Our method has shown to be effective across various benchmarks in situations of inconsistent taxonomy settings, such as coarse-to-fine taxonomy and open taxonomy, and demonstrates consistent synergy effects when integrated with previous state-of-the-art UDA methods.