Toward Robust In-Context Segmentation via Concept Guidance

In-context segmentation (ICS) requires a model to segmenttarget regions in a query image using only a few reference images andtheir corresponding masks, without updating any parameters. Despiterecent progress, prior ICS studies have largely overlooked a critical as-pect: system robustness, i.e., whether the model can produce stable seg-mentation results for the same query under different references. In thiswork, we revisit ICS from the robustness perspective and introduce anovel paradigm, Concept-Guided In-Context Segmentation (CG-ICS),which performs segmentation by extracting high-level semantic conceptsfrom references rather than relying solely on low-level visual matching.Specifically, CG-ICS introduces a concept reasoning module that usesan MLLM to propose candidates and a SAM3-driven scoring functionwith tree-search refinement to select reliable textual concepts, togetherwith a parallel visual exemplar route that provides query-side spatialgrounding via a simple context construction. Both the textual conceptand the visual exemplar are then used to activate the segmentation ca-pability of a frozen SAM3 backbone. Extensive experiments on standardICS benchmarks demonstrate that CG-ICS not only achieves state-of-the-art accuracy but also substantially improves robustness, yielding amore reliable ICS system with significantly reduced variance across di-verse reference choices.