ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

Knowledge-based Visual Question Answering (KB-VQA) re-quires models to combine image understanding with external knowledge.Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive duringreasoning. We propose ProMSA, a progressive multimodal search agentfor KB-VQA. Given an image-question pair, the agent iteratively choosesimage search, text search, or stop, under explicit tool-call budgets andwith deduplication to avoid redundant retrieval. For training, we first userejection-sampling SFT to learn valid tool-use formats, then optimize theagent with TN-GSPO, a sequence-level RL objective that normalizesupdates by both generation length and tool-interaction depth. Experi-ments on E-VQA and InfoSeek show consistent gains over strong RAGand agent baselines, and improved retrieval and end-to-end accuracy.Thecode is available at https://github.com/DingWu1021/Promsa.