VideoSfM: Exploiting Temporal Structure for Video-Based Structure-from-Motion

Accurately recovering the camera’s calibration and metricposes for any unconstrained video would unlock large-scale training datafor navigation and scene understanding. The dominant approaches tothis problem are severely limited: Simultaneous Localization and Map-ping (SLAM) is sensitive to initialization and transient failures due to itscausal, incremental nature; it is often over-optimized for real-time oper-ation and generally requires known camera calibration; while Structure-from-Motion (SfM) typically forgoes any image ordering, enabling opti-mal initialization and global optimization, but lacks robustness to visualsymmetries and extreme motions. To bridge this gap, we introduce asystem that combines the strong sequential constraints of SLAM withthe flexibility and global optimization of offline SfM, enabling the met-ric reconstruction of arbitrary, long, uncalibrated videos. This systemleverages recent advances in wide-baseline dense image matching, treatstemporal ordering as a first-class citizen for reliable loop closure, andaugments global optimization with metric monocular depth priors. Asa result, thorough evaluations on diverse, challenging datasets that ex-hibit extreme motion and visual symmetries reveal that our approach issignificantly more robust and accurate than both state-of-the-art SLAMand SfM, classical or learned, with given or unknown camera calibration.