Towards Scalable Pre-training of Visual Tokenizers for Generation

The quality of the latent space in visual tokenizers (e.g.,VAEs) is crucial for modern generative models. However, the standardreconstruction-based training paradigm produces a latent space that isbiased towards low-level information, leading to a foundational flaw: bet-ter pixel-level reconstruction accuracy does not lead to higher-qualitygeneration. This implies that pouring extensive compute into visual tok-enizer pre-training translates poorly to improved performance in gener-ation. We identify this as the “pre-training scaling problem” and suggesta necessary shift: to be effective for generation, a latent space must con-cisely represent high-level semantics. We present VTP, a unified visualtokenizer pre-training framework, pioneering the joint optimization ofimage-text contrastive, self-supervised, and reconstruction losses. Ourstudy reveals that perception-oriented tokenizer pre-training un-locks a new scaling law for generation, where generative perfor-mance scales effectively with compute, parameters, and data allocatedto the pre-training of the visual tokenizer. Our large-scale pre-trainingexperiments demonstrate the following results: (1) Without modifyingDiT training specs and FLOPs, solely scaling VTP pre-training con-sistently achieves gains in both ImageNet class-conditional and LAIONtext-to-image generation, while conventional autoencoders stagnate veryearly at 1/10 of the FLOPs. (2) VTP achieves 0.36 rFID while simultane-ously delivering 78.2% zero-shot accuracy and 85.7% linear probing accu-racy, surpassing prior unified tokenizers such as VILA-U and UniTok. (3)Furthermore, the VTP-based diffusion model exhibits exceptionally fastconvergence—reaching 2.03 gFID in only 80 epochs without guidancetricks, outperforming previous methods like VA-VAE and RAE—and ul-timately scales to achieve a remarkable 1.11 gFID on ImageNet 256×256generation. Our code and models are publicly available at VTP.