UniTemp: Unlocking Video Generation in Any Temporal Order via Autoregressive Distillation

Autoregressive video diffusion models have emerged as apromising approach for long video generation, achieving strong perfor-mance in streaming settings. However, existing methods are restrictedto forward temporal generation, whereas practical video creation oftenrequires flexible generation order, e.g., conditioning on future context toextend backward, or on both past and future context for inbetween gener-ation. We bridge this gap by training a single autoregressive model thatsupports generation in arbitrary temporal directions. A key technicalchallenge arises from the Causal 3D VAE widely used in video diffusionmodels, where latents are encoded strictly conditioned on past context.While suited for forward generation, this causal structure causes inter-block discontinuities when generation proceeds backward. To addressthis, we introduce blockwise anchor latents, a set of auxiliary latentsthat aim to restore the missing past context at block boundaries dur-ing backward generation. Built on this design, we propose UniTemp, abidirectional distillation framework that trains an autoregressive studentmodel for any-direction video generation. At inference time, UniTempconditions on arbitrary past and/or future frames, improving control-lability for both bidirectional and inbetween generation. Through ex-tensive experiments, we demonstrate that UniTemp maintains compet-itive performance on short and long video generation in comparison toforward-only methods, while enabling diverse creation workflows such asbidirectional video extension, inbetween generation, looping video gen-eration, scene transition, and visual story generation. Project website:https://lzhangbj.github.io/projects/unitemp/.