Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation

Universal medical image segmentation seeks to use a singlefoundational model to handle diverse tasks across multiple imaging modal-ities. However, existing approaches often rely heavily on manual visualprompts or retrieved reference images, which limits their automationand robustness. In addition, naive joint training across modalities oftenfails to address large domain shifts. To address these limitations, wepropose Concept-to-Pixel (C2P), a novel prompt-free universal seg-mentation framework. C2P explicitly separates anatomical knowledge intotwo components: Geometric and Semantic representations. It leveragesMultimodal Large Language Models (MLLMs) to distill abstract, high-level medical concepts into learnable Semantic Tokens and introducesexplicitly supervised Geometric Tokens to enforce universal physical andstructural constraints. These disentangled tokens interact deeply withimage features to generate input-specific dynamic kernels for precisemask prediction. Furthermore, we introduce a Geometry-Aware InferenceConsensus mechanism, which utilizes the model’s predicted geometricconstraints to assess prediction reliability and suppress outliers. Extensiveexperiments and analysis on a unified benchmark comprising eight diversedatasets across seven modalities demonstrate the significant superiorityof our jointly trained approach, compared to universe- or single-modelapproaches. Remarkably, our unified model demonstrates strong general-ization, achieving impressive results not only on zero-shot tasks involvingunseen cases but also in cross-modal transfers across similar tasks. Codeis available at: https://github.com/Yundi218/Concept-to-Pixel.