MVGS: Multi-view Regulated Gaussian Splatting for Novel View Synthesis

Recent works in novel view synthesis, e.g., Neural RadianceField (NeRF) and 3D Gaussian Splatting (3DGS), have significantly ad-vanced rendering quality and efficiency. However, existing Gaussian-basednovel view synthesis methods typically follow a single-view optimizationparadigm. We observed that this optimization paradigm suffers fromunstable gradients, leading to suboptimal rendering quality. To tackle thisissue, we present a novel multi-view regulated Gaussian Splatting (MVGS)that fully leverages a multi-view coherent (MVC) constraint throughoutthe optimization process. Specifically, our proposed MVC enhances 3DGaussian multi-view consistency and thus ensures smoother gradient up-dates. Furthermore, since single-scale training usually leads to suboptimalsolutions, we propose a cross-intrinsic guidance scheme in a coarse-to-fine manner to improve the convergence of multi-view optimization in3DGS. In particular, by incorporating more multi-view images at the lowresolution, we can optimize 3D Gaussians with more comprehensive per-spectives. Then, finer-scale Gaussians are initialized by coarsely estimatedones instead of optimizing full-scale 3D Gaussians from scratch. Moreover,we found that 3D Gaussians usually struggle to fit 2D training views withminimal overlap. Thus, we propose a novel multi-view cross-ray densifica-tion strategy, where 3D Gaussians are dynamically split to accommodatedrastic viewpoint variations in the multi-view optimization process. Inthis way, the multi-view consistency can be further improved. Notably,our proposed MVGS method is a plug-and-play optimizer. Extensiveexperiments across various tasks demonstrate that our proposed MVGSimproves existing Gaussian-based methods and achieves state-of-the-artperformance. Project Page: https://xiaobiaodu.github.io/mvgs-project/