CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

Knowledge-Intensive Visual Question Answering (KI-VQA)benchmarks evaluate Vision–Language Models (VLMs) as multimodalknowledge assistants by requiring external information beyond a pro-vided image to answer questions. KI-VQA involves multiple sub-problems—referring expression understanding, visual grounding, object recogni-tion, knowledge retrieval, and reasoning—yet existing benchmarks typ-ically report only end-task accuracy, obscuring where failures arise. Toanalyze the full KI-VQA pipeline, we introduce CRAG-MM-Diagnostics,a diagnostic benchmark with stage-wise data annotations that isolate1 language-based visual grounding, 2 object identification, and 3knowledge retrieval and reasoning. We evaluate fully parametric andretrieval-augmented VLMs, providing fine-grained analyses using newlycollected metadata, such as target ROIs, entity names, and visual com-plexity scores. Our results point to knowledge retrieval and reasoning asthe primary bottleneck, but also highlight issues in the other parts ofthe KI-VQA pipeline, such as the fact that VLMs struggle with targetobject identification or that image retrievers struggle to integrate textualcues. These findings expose fundamental limitations in current KI-VQAsystems and motivate stage-aware evaluation. We, lastly, leverage thesefindings to propose a grounded bimodal RAG pipeline that integrates avisual grounding module to crop targets before image retrieval, boostingGPT-5 and Qwen’s respective accuracies by 13.3 and 8.5 percentage points.