Let ViT Speak: Generative Language-Image Pre-training

In this paper, we present Generative Language-Image Pre-training (GenLIP), a simplified generative pretraining approach for Vi-sion Transformers (ViTs) tailored to multimodal large language models(MLLMs). To better align ViTs with the autoregressive nature of LLMs,GenLIP trains a ViT to predict language tokens directly from visual to-kens using a standard language modeling objective—without contrastivebatch construction or an additional text decoder. GenLIP offers three keyadvantages: (1) Simplicity: a single transformer jointly models visualand linguistic tokens; (2) Scalability: it scales efficiently with both dataand model size; (3) Performance: it achieves competitive or superiorresults across diverse multimodal benchmarks. With only 1/5 of theseen samples, GenLIP matches or surpasses strong baselines such asSigLIP2. With continued pretraining on multi-resolution images at nativeaspect ratios, GenLIP further excels at detail-sensitive tasks such as OCR,chart understanding, and visual question answering, making it a strongfoundation for vision encoders in MLLMs.