Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning

RL (reinforcement learning) methods (e.g., GRPO) for mul-timodal LLM perception ability has attracted wide research interest ow-ing to its remarkable generalization ability. Nevertheless, existing rein-forcement learning methods still face the problem of low data quality,where data samples cannot elicit diverse responses from MLLMs, thus re-stricting the exploration scope for MLLM reinforcement learning. Somemethods attempt to mitigate this problem by imposing constraints onentropy, but none address it at its root. Therefore, to tackle this problem,this work proposes Syn-GRPO (Synthesis-GRPO), which employs anonline data generator to synthesize high-quality training data with di-verse responses in GRPO training. Specifically, Syn-GRPO consists oftwo components: (1) data server; (2) GRPO workflow. The data serversynthesizes new samples from existing ones using an image generationmodel, featuring a decoupled and asynchronous scheme to achieve highgeneration efficiency. The GRPO workflow provides the data server withthe new image descriptions, and it leverages a diversity reward to su-pervise the MLLM to predict image descriptions for synthesizing sam-ples with diverse responses. Experiment results across three visual per-ception tasks demonstrate that Syn-GRPO improves the data qualityby a large margin, achieving significant superior performance to exist-ing MLLM perception methods, and Syn-GRPO presents promising po-tential for scaling long-term self-evolving RL. Our code is available athttps: // github. com/ hqhQAQ/ Syn-GRPO .