On-Policy Diffusion Reinforcement Learning Meets Off-Policy Quality Anchoring

Reinforcement learning for diffusion models primarily relieson on-policy learning to optimize the model by using samples from itsown current policy. Despite offering a direct optimization signal, theapproach is fundamentally self-limiting and prone to reward overfittingand mode collapse due to its myopic guidance. In this paper, we pro-pose a new framework, namely DiffusionCompass, that breaks such lim-itation by strategically integrating off-policy guidance. DiffusionCom-pass establishes a dual learning objective: the model continues to ex-plore reward maximization on-policy, while simultaneously being an-chored by off-policy examples that define a stable quality benchmark.That is achieved through a novel integration of several key techniques:an off-policy objective with distribution-aware filtering to adaptively em-phasize relevant anchors, a perceptual anchored reward mechanism tomitigate reward hacking, and a reward renormalization strategy usingoff-policy samples for a stable quality baseline. The dual regimes com-plement each other: off-policy guidance prevents reward overfitting andcollapse, and on-policy exploration ensures the model does not merelyimitate a static dataset. Experiments across text-to-image/video gener-ation demonstrate that DiffusionCompass achieves superior reward per-formance while maintaining high sample diversity, shaping a robust andeffective path for fine-tuning generative models. Code is available athttps://github.com/HiDream-ai/DiffusionCompass.