Pose Anything Anywhere: Model-free Object Poses from Arbitrary References

Estimating the 6D pose of unseen objects is a fundamentalyet challenging problem for open-world robotics and embodied percep-tion. Model-based methods are accurate but depend on CAD assets orheavy onboarding, while most model-free approaches are still limitedto pairwise single-anchor matching and thus fail under occlusion andlarge viewpoint changes with low query–reference overlap. Therefore, wepresent PANY, a unified model-free framework that seamlessly sup-ports both RGB and RGB-D inputs, operates on one or sparse pose-freereference views, and generalizes effectively to novel objects. Built on amulti-view transformer geometry backbone, PANY moves beyond pair-wise matching by learning view-consistent geometry and cross-view align-ment cues that remain stable under wide baselines and limited overlap.When additional unposed assist views are available, PANY aggregatesthem via pose-graph canonical registration to increase geometric cover-age and reinforce the final pose. Extensive experiments show that PANYachieves state-of-the-art performance across multiple benchmarks, sub-stantially outperforming existing model-free methods, improving poseaccuracy by +12% on YCB-V and over +20% on LM-O. Further-more, PANY consistently performs well under both single-reference andsparse-reference settings, demonstrating strong robustness in real-worldenvironments.