TouchAnything: Diffusion-Guided 3D Reconstruction from Sparse Robot Touches

Accurate object geometry estimation is essential for manydownstream tasks, including robotic manipulation and physical interac-tion. Although vision is the dominant modality for shape perception, itbecomes unreliable under occlusions or challenging lighting conditions.In such scenarios, tactile sensing provides direct geometric informationthrough physical contact. However, reconstructing global 3D geometryfrom sparse local touches alone is fundamentally underconstrained. Wepresent TouchAnything, a framework that leverages a pretrained large-scale 2D vision diffusion model as a semantic and geometric prior for3D reconstruction from sparse tactile measurements. Unlike prior workthat trains category-specific reconstruction networks or learns diffusionmodels directly from tactile data, we transfer the geometric knowledgeencoded in pretrained visual diffusion models to the tactile domain.Given sparse contact constraints and a coarse class-level description ofthe object, we formulate reconstruction as an optimization problem thatenforces tactile consistency while guiding solutions toward shapes con-sistent with the diffusion prior. Our method reconstructs accurate ge-ometries from only a few touches, outperforms existing baselines, andenables 3D reconstruction of previously unseen object instances. Ourproject page is https://grange007.github.io/touchanything/.