Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning

Vision-as-inverse-graphics, the concept of reconstructing im-ages into editable programs, remains challenging for Vision-LanguageModels (VLMs), which inherently lack fine-grained spatial grounding inone-shot settings. To address this, we introduce VIGA (Vision-as-Inverse-Graphics Agent), an interleaved multimodal reasoning framework wheresymbolic logic and visual perception actively cross-verify each other. VIGAoperates through a tightly coupled code-render-inspect loop: synthesizingsymbolic programs, projecting them into visual states, and inspectingdiscrepancies to guide iterative edits. Equipped with high-level semanticskills and an evolving multimodal memory, VIGA sustains evidence-basedmodifications over long horizons. This training-free, task-agnostic frame-work seamlessly supports 2D document generation, 3D reconstruction,multi-step 3D editing, and 4D physical interaction. Finally, we intro-duce BlenderBench, a challenging visual-to-code benchmark. Empirically,VIGA substantially improves accuracy compared with one-shot base-lines in BlenderGym (35.32% ), SlideBench (117.17% ) and our proposedBlenderBench (124.70% ).