CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation

Referring Remote Sensing Image Segmentation (RRSIS) hasachieved significant progress through the integration of VLMs and theSegment Anything Model (SAM). However, this progress largely relieson strong pre-trained capabilities, while leaving two fundamental limi-tations insufficiently addressed: (1) Architectural Weak-Coupling, wherethe unidirectional flow forces reliance on coarse VLM prompts and wastesSAM’s pixel-level structural guidance, causing localization drift; and (2)Object-Centric Semantic Bias, where models overemphasize dominantobject semantics while remaining insensitive to spatial reasoning cru-cial for RRSIS. Motivated by these observations, we propose CROSS, atightly integrated paradigm for RRSIS. First, we introduce Linguistic-Guided Cascaded Distillation (LGCD) to bridge the architectural gap,which distills SAM’s geometric affinities as soft regularizers into VLM in-termediate layers, injecting dense structural priors to refine localization.Second, Perspective-Spatial Contrastive Learning (PSCL) imposes cross-anchored constraints by mining mask-filtered deceptive distractors andspatial-linguistic counterfactuals as hard negatives, explicitly shatter-ing semantic shortcuts to enforce genuine logical consistency. Extensiveexperiments on RRSIS benchmarks demonstrate that CROSS achievesstate-of-the-art performance and maintains precise localization even un-der severe spatial description perturbations, standing as a robust newparadigm for RRSIS. https://clarence-cv.github.io/CROSS/.