Repurposing Geometric Foundation Models for Multi-view Diffusion

The latent space of diffusion models fundamentally deter-mines their learning efficiency and generation quality. While recent ad-vances in the latent space have driven substantial progress in single-imagegeneration, the optimal latent space for novel view synthesis (NVS) re-mains largely unexplored. In particular, NVS requires geometrically con-sistent generation across viewpoints, but existing approaches typicallyoperate in a view-independent latent space. In this paper, we proposeGeometric Latent Diffusion (GLD), a framework that repurposesthe feature space of a geometric foundation model as the latent space formulti-view diffusion. We show that the features of the geometric founda-tion model not only support high-fidelity RGB reconstruction but alsoencode strong cross-view geometric correspondences, providing a well-suited latent space for NVS. Through experiments, GLD outperformsboth VAE and RAE on 2D image quality and 3D consistency metrics,accelerating training by more than 4.4× compared to the VAE latentspace. Notably, GLD remains competitive with state-of-the-art methodsthat leverage large-scale text-to-image pretraining, despite training itsdiffusion model from scratch without such generative pretraining.