Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Reconstructing physics-based 3D assets—geometry, mate-rials, and illumination—from multi-view images is a core problem incomputer graphics and vision, and a prerequisite for realistic relightingand editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong pri-ors, illumination is baked into materials, and reconstructions generalizepoorly to novel views and lighting. Data-driven diffusion models, in con-trast, predict visually plausible materials, yet their predictions rarelysatisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacingeither. Our key idea is to treat the predictions of a state-of-the-art dif-fusion model not as target material values but as a similarity kernel foroptimization: we introduce a regularization loss that penalizes deviationsin the optimized material over surface regions where the diffusion predic-tions are near-constant, while leaving the optimization free to match theinput images. Built on this regularizer, our end-to-end pipeline jointlyreconstructs geometry, materials, and illumination, yielding high-qualityassets that drop into standard rendering pipelines and relight faithfully.On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets,our method significantly outperforms state-of-the-art baselines in bothreconstruction accuracy and relighting quality.