PASTEL: Panoramic Alignment for Monocular 4D Scene Reconstruction

Reconstructing 4D scenes from casually captured monocularvideo is vital for applications in virtual reality (VR) and embodied AI.Recent advances in 4D reconstruction and novel view synthesis havesubstantially propelled this capability. However, existing reconstructionmethods generally cannot recover regions beyond visible camera limits.Consequently, we introduce a new paradigm that achieves 4D scene syn-thesis by combining visible-region reconstruction from monocular inputwith invisible-region generation beyond observable camera boundaries.A straightforward approach is to leverage video generation models as“generative priors” for invisible-region exploration. However, their inherentstochasticity and large solution space prevent stable and consistent viewsynthesis. As a result, naively incorporating generative content into thereconstruction process often causes artifacts, especially when camera tra-jectories deviate significantly from the original video. To overcome thesechallenges, we present Panoramic Alignment for Strategic Exploitation ofGenerative Priors (PASTEL). Specifically, PASTEL proposes panoramicscene alignment, a novel representation that reformulates the intractable3D "invisible region” exploration into a tractable 2D directional trajectoryplanning. This is achieved by reducing the viewpoint planning from 6-DoFsearch to a 2D directional search with explicit visibility boundaries. Byoperating within this panoramic space, our method strategically iden-tifies camera trajectories that maximize exploration beyond observableboundaries while minimizing viewpoint deviation. Experimental resultsshow that PASTEL can not only extrapolate plausible scene contentbeyond the observable boundaries of input monocular videos, but alsosubstantially boost monocular 4D reconstruction performance. PASTELoutperforms the previous state-of-the-art method by 0.9dB in full-imagePSNR on the DyCheck IPhone dataset.