RAU: Reference-based Anatomical Understanding with Vision-Language Models

Anatomical understanding, which is the ability to identify,localize, or segment anatomical structures, is critical in medical imageanalysis; however, its progress is constrained by the scarcity of expert-labeled data. A promising remedy is to leverage an annotated referenceimage to guide the interpretation of an unlabeled target. Although re-cent vision–language models (VLMs) exhibit non-trivial visual reasoning,their reference-based understanding and fine-grained localization remainlimited. We introduce RAU, a framework for reference-based anatomicalunderstanding with VLMs. We first show that a VLM learns to identifyanatomical regions through relative spatial reasoning between referenceand target images, trained on a moderately sized dataset. We validatethis capability through visual question answering (VQA) and boundingbox prediction. Next, we demonstrate that the VLM-derived spatial cuescan be seamlessly integrated with the fine-grained segmentation capabil-ity of SAM2, enabling localization and pixel-level segmentation of smallanatomical regions, such as vessel segments. Across two in-distributionand two out-of-distribution datasets, RAU consistently outperforms aSAM2 fine-tuning baseline using the same memory setup, yielding moreaccurate segmentations and more reliable localization. More importantly,its generalization ability to unseen modalities makes it scalable to unseendatasets, a property crucial for medical image applications. To the bestof our knowledge, RAU is the first to explore the capability of VLMs forreference-based identification, localization, and segmentation of anatom-ical structures in medical images. Its promising performance highlightsthe potential of VLM-driven approaches for anatomical understandingin automated clinical workflows.