Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning

Action-conditioned robot world models generate future videoframes of the manipulated scene given a robot action sequence, o!eringa promising alternative for simulating tasks that are di"cult to modelwith traditional physics engines. However, these models are optimized forshort-term prediction and break down when deployed autoregressively:each predicted clip feeds back as context for the next, causing errors tocompound and visual quality to rapidly degrade. We address this throughthe following contributions. First, we introduce a reinforcement learning(RL) post-training scheme that trains the world model on its own au-toregressive rollouts rather than on ground-truth histories. We achievethis by adapting a recent contrastive RL objective for di!usion modelsto our setting and show that its convergence guarantees carry over ex-actly. Second, we design a training protocol that generates and comparesmultiple candidate variable-length futures from the same rollout state,reinforcing higher-fidelity predictions over lower-fidelity ones. Third, wedevelop e"cient, multi-view visual fidelity rewards that combine com-plementary perceptual metrics across camera views and are aggregatedat the clip level for dense, low-variance training signal. Fourth, we showthat our approach establishes a new state-of-the-art for rollout fidelityon the DROID dataset, outperforming the strongest baseline on all met-rics (e.g., LPIPS reduced by 14% on external cameras, SSIM improvedby 9.1% on the wrist camera), winning 98% of paired comparisons, andachieving an 80% preference rate in a blind human study.