DeRA: Decoupled Representation Alignment for Video Tokenization

This paper presents DeRA, a novel 1D video tokenizer thatdecouples the spatial-temporal representation learning in video tokeniza-tion to achieve better training efficiency and performance. Specifically,DeRA maintains a compact 1D latent space while factorizing video en-coding into appearance and motion streams, which are aligned withpretrained vision foundation models to capture the spatial semanticsand temporal dynamics in videos separately. To address the gradientconflicts introduced by the heterogeneous supervision, we further pro-pose the Symmetric Alignment-Conflict Projection (SACP) module thatproactively reformulates gradients by suppressing the components alongconflicting directions. Extensive experiments demonstrate that DeRAoutperforms LARP, the previous state-of-the-art video tokenizer by 25%on UCF-101 in terms of rFVD. Moreover, using DeRA for autoregressivevideo generation, we also achieve new state-of-the-art results on bothUCF-101 class-conditional generation and K600 frame prediction.