ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

Leveraging Multimodal Large Language Models (MLLMs)via contrastive learning has become a mainstream paradigm for improv-ing the performance of Universal Multimodal Retrieval (UMR). However,previous works have ignored the grain blindness when adapting thecontrastive paradigm into retrieval tasks. Grain blindness refers to thetendency of the model to overlook grain-level information contained inthe query, which is crucial for effectively handling complex queries. Thisstems from contrastive learning treating samples as a binary classifica-tion (positive/negative), while ignoring the different information carriedby each negative sample. To address this, we argue that negatives shouldbe treated differently according to their similarity to the positive sample,enabling the model to learn distinct grain information from each nega-tive. In this paper, we introduce a simple but effective framework, calledELVA, a novel rule-based RL framework that mitigates grain blindnessthrough ranking-driven MLLMs. 1) Instead of relying on reward mod-els, we extend Reinforcement Learning with Verifiable Rewards (RLVR)to retrieval tasks, allowing the model to explore new ranking behav-iors without explicit ranking labels. 2) By utilizing rule-based rewards,our approach jointly optimizes the ranking of negative samples whileenlarging the similarity gap between positive and negative. To more pre-cisely measure grain blindness, we further introduce MRBench, a newbenchmark specifically designed for multi-grain query scenarios. ELVAachieves state-of-the-art results across standard retrieval benchmarks,and its notable 13.1% improvement on MRBench further demonstratesits effectiveness in alleviating grain blindness. Our code is available athttps://github.com/SeerRay-Lab/ELVA.