VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension

Human motion modeling must support generation from text,music, and speech, comprehension through captioning, temporal com-pletion, and multi-person coordination—yet existing methods typicallytackle each scenario with a separate, task-specific system. We arguethat discrete tokenization plus autoregressive (AR) modeling possessesunique compositional advantages for building a unified motion model:all modalities share a single vocabulary and training objective, tasks aredefined by token arrangement rather than architectural changes, andthe same model naturally supports both generation and understand-ing by simply reordering the sequence. However, realizing this potentialhas been blocked by two obstacles. First, current motion tokenizers useroot-relative, non-causal, frame-level encodings that discard world-spaceinformation, break consistency with forward-only AR decoding, and scalepoorly—confining existing discrete AR models to text–motion pairs insingle-human settings. Second, multi-modal motion data with paired textand audio annotations remains scarce. We address both obstacles. Weintroduce VerMoVQ, a causal spatio-temporal VQ-VAE that factorizeseach frame into low-dimensional body tokens while preserving world-space root trajectories, naturally enabling multi-person tokenization viaper-agent concatenation and arbitrary-length encoding through stricttemporal causality. We curate MotionHub, a \sim 380K-clip corpus in unifiedSMPL representation with large-scale multi-modal annotations spanningnine motion tasks. Built on these foundations, we train VersatileMo-tion, a single decoder-only AR Transformer that covers text\leftrightarrow motion,music\leftrightarrow dance, speech