MoBa-GS: Learning a Spatially-Varying Motion Basis over a Dynamic Canonical Space for 4D Reconstruction

Faithfully capturing the intricate relationship between motion and geometry in dynamic scenes is essential for 4D reconstruction. Recent state-of-the-art methods rely on monolithic deformation networks or global time-basis factorizations that struggle to represent complex, non-rigid topological changes. We propose MoBa-GS, a framework that resolves this entanglement by introducing a structural inversion: a spatially-factorized motion field coupled with adaptive geometric optimization. First, our model learns a low-frequency dynamic canonical space to represent coarse scene motion. Next, it decomposes complex, non-rigid motion into a Spatially-Varying Motion Basis of local kinematics, predicted from the canonical geometry, which is then linearly combined using dynamic blending weights. This formulation directly acts as an implicit neural scaffold, recovering both geometry and motion from random initialization, thereby removing the dependency on SfM point cloud priors. This design is further augmented with motion-guided densification and positional annealing to reduce geometry overfitting. Extensive experiments show that our framework surpasses prior state-of-theart methods in reconstruction fidelity. Enabled by time-invariant caching, MoBa-GS requires an order-of-magnitude shorter training time (<18 minutes), a compact storage (∼ 11 MB), and achieves real-time rendering speeds (>163 FPS). Our work establishes a new foundation for highfidelity, efficient 4D representations without relying on explicit geometric priors. Code is available at https://github.com/tgy1221/MoBa-GS.