Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention

Recent advancements in video generation has shifted from bidi-rectional models for short videos to autoregressive ones for ultra long videogeneration. Previous models, which usually use sliding window attention torestrict inference cost, lack effective memory compression and retrieval forlong-term generation beyond the window size, leading to issues of forgettingand spatiotemporal inconsistencies. To enhance the retention of historicalinformation with a fixed memory budget, we additionally incorporate tem-poral recurrent neural network (RNN) layers into the diffusion transformer(DiT) model. Specifically, we found that a LSTM layer after attention at eachDiT layer achieves comparable performance to other state-of-the-art RNNblocks, such as Test-Time Training (TTT) and Mamba2. Moreover, existingdiffusion-RNN approaches often suffer from performance degradation dueto training-inference gap or the lack of overlap across windows. To addressthese limitations, we propose a novel Recurrent Autoregressive Diffusion(RAD) framework, which leverages recurrent blocks for memory update andretrieval and preserves local details by full attention on overlapping slidingwindows, with no training and inference gap. Experiments on Memory Mazeand Minecraft datasets demonstrate the superiority for long video generationby our framework with global memory and local attention.Project page: https://yeyutaihan.github.io/recurrent-autoregressive-diffusion/