Towards Unsupervised Multi-modal Semantic Segmentation

Multi-modal semantic segmentation (MSS) is essential forrobust perception in complex environments, yet its potential remainslargely untapped due to the prohibitive cost of human annotations. Whileunsupervised semantic segmentation (USS) has seen success on single RGBmodality, its naive extension to multi-modal data is hamperedby fusion degradation. This is because, in the absence of explicitsupervision, existing frameworks struggle to reconcile the heterogeneousstructural patterns captured by different sensors, failing to effectivelyexploit their complementary information. In this paper, we make thefirst attempt to address the novel problem of Unsupervised Multi-modal Semantic Segmentation (UMSS), aiming to effectively exploitcomplementary sensor information in a fully label-free setting. To thisend, we propose UniM2 (Unified Multi-Modal), a novel frameworkbuilt upon DINOv3 that transforms conventional fusion methods intoconsistent performance gains. Our key idea is to learn a unified latentspace driven by Cross-modal Correspondence Synergy (CMCS) toextract intrinsic shared semantic cues, bypassing the need for label-guidedadaptive fusion. To mitigate inherent inter-modal conflicts, we introducea Cross-modal Harmonizer (CMH) that designates RGB as a stablereference, effectively suppressing inconsistent relational supervision whileguiding the model to exploit complementary structural features. Extensiveexperimental results on NYU-Depth-v2 and MFNet show that UniM2improves mIoU by 6.4% and 9.8%, respectively, demonstrating clearadvantages over existing frameworks in UMSS task.