The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

Deep representations across modalities are inherently inter-twined. In this paper, we systematically analyze the spectral character-istics of various semantic and pixel encoders. Interestingly, our studyuncovers a highly inspiring and rarely explored correspondence betweenan encoder’s feature spectrum and its functional role: semantic encodersprimarily capture low-frequency components that encode abstract mean-ing, whereas pixel encoders additionally retain high-frequency informa-tion that conveys fine-grained detail. This heuristic finding offers a uni-fying perspective that ties encoder behavior to its underlying spectralstructure. We define it as the Prism Hypothesis, where each datamodality can be viewed as a projection of the natural world onto ashared feature spectrum, just like the prism. Building on this insight,we propose Unified Autoencoding (UAE), a model that harmonizessemantic structure and pixel details via an innovative frequency-bandmodulator, enabling their seamless coexistence. Extensive experimentsdemonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art per-formance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanillaJIT baseline. Code is avaliable at: https://github.com/WeichenFan/UAE.