PercepTax: Benchmarking Cross-Property Reasoning in Vision-Language Models

Reasoning about objects in the physical world requires understanding both the spatial structure of a scene and the properties of objects within it. To perform reasoning tasks, a model must identify candidate objects from the scene and then reason over their properties, such as material, physical attributes, affordance, and function to satisfy the task constraints. We formalize this reasoning process as Perceptual Taxonomy, a hierarchical framework that organizes visual understanding into structured scene–object–property representations. Existing benchmarks, however, address only parts of this framework: some focus on spatial understanding alone, while others evaluate individual property families—material, affordance, or physical attributes—each under a separate schema. They fail to cover a challenging yet practically important capability: cross-property reasoning, where a model must simultaneously integrate reasoning across all property families to answer questions about physical environments. To evaluate this capability, we introduce PercepTax, a benchmark for perceptual taxonomy reasoning. We annotate 3,173 object classes with 54 fine-grained attributes across four property families: material, physical attributes, affordance, and function. The benchmark contains 5,802 images from both synthetic and real domains and 28,033 questions spanning object description, singleproperty recognition, 3D spatial reasoning, and cross-property reasoning. Experiments show that state-of-the-art vision language models (VLMs) perform strongly on object description and single-property tasks, but accuracy drops substantially on cross-property reasoning that requires integrating multiple attributes simultaneously. Oracle experiments with ground-truth objects’ properties with 3D positions and taxonomy-guided in-context learning both improve performance substantially yet remain well below human levels, confirming that both visual perception of individual properties and their cross-property integration are bottlenecks for VLMs.