Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation

Video generation is central to AI-powered content creation.The alignment with human preferences is one of the key metrics for mea-suring the quality of the generated videos. Despite significant progress invisual quality, three key challenges remain: 1) The reliability of rewardsignals is constrained by the quality of human preference data, which isoften corrupted by subjective noise and bias. 2) Standard scalar rewardmodels collapse multi-aspect human preference into a single value, lead-ing to the loss of dynamic trade-offs across multiple dimensions of humanpreference. 3) In policy optimization, the widely adopted KL-divergenceimposes only local constraints, failing to capture holistic human pref-erence. To address these challenges, we propose a unified, preference-aware learning framework for video generation. First, we propose elite-guided filtering to calibrate preference data and construct reliable su-pervision for reward-model training. We then model video quality asa multidimensional reward distribution to capture the uncertain na-ture of human preference, and use the Wasserstein distance to alignit with the empirical human preference distribution. Finally, we intro-duce Wasserstein-based distributional alignment in GRPO, guiding thepolicy’s video generation to match the global structure of human videopreference. Experiments on reward modeling and video generation showthat our approach improves the reliability of reward signals and theperceptual consistency of generated videos. Our code is available athttps://github.com/alignhs26/ahs.