EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis

Text-to-image diffusion models have achieved remarkable fi-delity in synthesizing images from explicit text prompts, yet exhibita critical deficiency in processing implicit prompts that require deep-level world knowledge, ranging from natural sciences to cultural com-monsense, resulting in counter-factual synthesis. This paper traces theroot of this limitation to a fundamental dislocation of the underlyingknowledge structures, manifesting as a chaotic organization of implicitprompts compared to their explicit counterparts. In this paper, we pro-pose EruDiff, which aims to refactor the knowledge within diffusionmodels. Specifically, we develop the Diffusion Knowledge DistributionMatching (DK-DM) to register the knowledge distribution of intractableimplicit prompts with that of well-defined explicit anchors. Furthermore,to rectify the inherent biases in explicit prompt rendering, we employthe Negative-Only Reinforcement Learning (NO-RL) strategy for fine-grained correction. Rigorous empirical evaluations demonstrate that ourmethod significantly enhances the performance of leading diffusion mod-els, including FLUX and Qwen-Image, across both the scientific knowl-edge benchmark (i.e., Science-T2I) and the world knowledge bench-mark (i.e., WISE), underscoring the effectiveness and generalizability.Our code is available at https://github.com/xiefan-guo/erudiff.