Error-Driven Scene Editing for 3D Grounding in Large Language Models

Despite recent progress in 3D-LLMs, they remain limitedin accurately grounding language to visual and spatial elements in 3Denvironments. This limitation stems in part from training data that fo-cuses on language reasoning rather than spatial understanding due toscarce 3D resources, leaving inherent grounding biases unresolved. Toaddress this, we propose 3D scene editing as a key mechanism to gener-ate visual counterfactuals that mitigate these biases through fine-grainedspatial manipulation, without requiring costly scene reconstruction orlarge-scale 3D data collection. Furthermore, to make these edits targetedand directly address the specific weaknesses of the model, we introduceDEER-3D, an error-driven framework that diagnoses grounding failuresand generates targeted counterfactual training supervision via a struc-tured “Decompose, Diagnose, Edit, and Retrain” loop. Specifically, givena grounding failure, DEER-3D first identifies the predicate-level error(e.g., attribute or spatial relation). It then performs minimal predicate-aligned scene edits, such as recoloring or repositioning, and constructsaligned question–answer pairs that explicitly target the failed predicate,forming targeted counterfactual training examples. We evaluate our edit-ing pipeline across multiple benchmarks for 3D grounding and scene un-derstanding tasks, consistently demonstrating improvements across allgrounding datasets through iterative refinement (4-6% gains). DEER-3D underscores the effectiveness of targeted, error-driven scene editingin bridging linguistic reasoning with spatial grounding in 3D LLMs.