Importance-Aware Low-Rank Distillation of Diffusion Transformers

Diffusion Transformers (DiTs) have emerged as a dominantarchitecture for high-quality text-to-image generation, yet their scaleposes challenges for efficient deployment. While truncated singular valuedecomposition (SVD) is a principled tool for parameter reduction, evi-dence from large language models (LLMs) suggests that naive low-rankapproximation can cause catastrophic failure. In contrast, we find thattruncated SVD in DiTs produces smooth degradation even under sub-stantial global compression, with redundancy distributed across projec-tion matrices throughout the whole network rather than concentratedin a few transformer blocks. Building on these insights, we introduceSVDtrunc, a two-step block-level compression scheme, first allocatingranks across blocks and compressing the least important ones via trun-cated SVD under a global parameter budget, and then fine-tuning allblocks with modular knowledge distillation and a rectified-flow objec-tive. We apply SVDtrunc to FLUX.dev across compression levels rangingfrom 40–90 % of the original parameter count. Across three benchmarks,GenEval, HPSv2, and DPG, we outperform all competing approaches.Notably, and in contrast to prior work, we retain near-full performanceat 68 % and remain competitive even at 57 % of the original parameterbudget. Furthermore, we show that SVDtrunc complements step distil-lation and achieves strong results even without fine-tuning, positioningit as a practical continuation of efficiency improvements beyond diffusionstep reduction for large-scale generative models.