RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios

Visual grounding (VG) localizes target objects in an imagefrom natural-language expressions. In real-world perception, RGB cuesoften degrade under low illumination and adverse weather, making visualgrounding substantially more challenging. However, existing VG bench-marks are largely RGB-only and provide limited, structured coverage ofsuch conditions, hindering systematic robustness evaluation and cross-spectral comparison. We present RGBT-GroundBench, the first large-scale benchmark for RGB-Thermal (TIR) visual grounding in complexenvironments. It contains over 40K images (21,535 RGB-TIR pairs) and38,760 object instances with referring expressions, bounding boxes, andfine-grained annotations at three levels: scene types, environmental con-ditions (illumination and weather), and object properties (size and oc-clusion). As a benchmark suite, RGBT-GroundBench provides not onlycurated RGB-TIR grounding annotations but also a unified evaluationprotocol supporting RGB-only, TIR-only, and RGB+TIR inputs. Underthis protocol, we benchmark 11 representative VG models across diversescenes and environmental conditions. Our results show that groundingaccuracy is strongly correlated with scene complexity, LoRA-based mod-els are more robust in complex scenes, and low-illumination conditionscause significant performance degradation that has been rarely explored.Guided by these observations, we introduce RGBT-VGNet, a simpleand reproducible reference baseline under the unified protocol, featuringAsymmetric Modality Adaptation, Language-Aware Visual Synergy, andTri-Prior Fusion for reliability-aware RGB-TIR integration. Resources,annotations, code, checkpoints, and evaluation scripts have been publiclyreleased.