Stable and Scalable Bundle Adjustment of Holistic 3D Structures

Bundle Adjustment (BA) is a cornerstone of 3D computervision and has benefited from decades of advances in sparse optimiza-tion and numerical methods. It was originally developed for jointly op-timizing camera intrinsics, poses and sparse 3D points. While exten-sions incorporate lines and other primitives, integrating richer geometricstructures such as parallelism, coplanarity, or wireframes often intro-duces significantly increased computational cost and reduced numericalstability. In this paper, we propose a unified framework that extendsbundle adjustment to jointly optimize geometric features and higher-order relations. We first introduce a taxonomy that distinguishes scal-able geometric features with direct 2D measurements (e.g., points andlines), from groups encoding higher-order relations (e.g., coplanarity, par-allelism, etc.), where we show that groups can be modeled as camera-likeentities within the bundle adjustment framework. Building on this formu-lation, we propose that both group constraints and cross-feature relations(i.e., point–line associations) can be expressed through 2D reprojectionmeasurements. By formulating group-induced and cross-feature repro-jection errors, we preserve the sparsity structure of classical point-basedBA under Schur elimination, while avoiding direct 3D regularization thatdegrades the conditioning and stability. Experiments on both real-worldand synthetic datasets demonstrate runtime performance comparableto classical point-only bundle adjustment, while producing significantlyricher 3D structures and improved geometric accuracy.