UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception

Recent advances in diffusion models have shown impressiveperformance in controllable image generation and dense prediction tasks.However, existing approaches typically treat diffusion-based controllablegeneration and dense prediction as separate tasks, overlooking the po-tential benefits of jointly modeling the heterogeneous distributions. Inthis work, we introduce UniGP, a framework built upon MMDiT, whichunifies controllable generation and dense prediction through simple jointtraining, without the need for complex task-specific designs or losses,while preserving the backbone’s versatile priors. By learning controllablegeneration and prediction under different conditions, our model effectivelycaptures the joint distribution of image-geometry pairs. UniGP is capableof versatile controllable generation, dense prediction, and joint generation.Specifically, the proposed UniGP consists of DUGP and a unified datasettraining strategy. The former, following the principle of Occam’s razor,uses only a copied image branch of MMDiT to model dense distributionsbeyond RGB, while the latter integrates heterogeneous datasets into aunified training framework to jointly model generation and perceptiontasks. Extensive experiments demonstrate that our unified model sur-passes prior unified approaches and performs on par with specializedmethods. Furthermore, we demonstrate that multi-task joint training pro-vides complementary benefits: generative priors enrich perceptual details,while perceptual learning improves structural alignment in generation.Project page: guoqincode.github.io/UniGP.