MMAgent-R2: Learning to Rerank and Reject for Agentic mRAG

Knowledge-based Visual Question Answering (KB-VQA) re-quires models to retrieve visual entities matching the query image fromlarge-scale encyclopedic knowledge bases and answer related questions.Existing multimodal Retrieval Augmented Generation (mRAG) meth-ods rely on global visual features to match candidate entities, yet whenthe knowledge base contains numerous visually similar entities, the re-triever struggles to distinguish them, populating the candidate set withvisually similar but factually mismatched distractors. Since subsequentprocessing steps such as noise filtering are also confined to this fixedcandidate set, errors from failed retrieval inevitably propagate to the fi-nal answer. To address these challenges, we propose MMAgent-R2 , anagentic mRAG framework that integrates visual reranking and active re-jection as its internal verification mechanism. Visual reranking directlycompares query and candidate images, capturing discriminative detailsbeyond textual descriptions to precisely identify the target entity amongsimilar candidates; active rejection discards unreliable results and re-trieves additional candidates when no confident match is found, movingbeyond the fixed candidate pool. We design a composite reward functionwith step-level verification rewards and achieve joint optimization of ex-ternal retrieval, internal verification, and answer generation via GRPOtraining. Experiments on InfoSeek, E-VQA, and MMhops demonstratethat MMAgent-R2 achieves state-of-the-art performance, with particu-larly notable advantages in challenging retrieval scenarios and complexmulti-image multi-hop reasoning tasks.