Human-like Object Grouping in Self-supervised Vision Transformers

Vision foundation models trained with self-supervised objec-tives achieve strong performance across diverse tasks and exhibit emer-gent object segmentation properties. However, their alignment with hu-man object perception remains poorly understood. Here, we introducea behavioral benchmark in which participants make same/different ob-ject judgments for dot pairs on naturalistic scenes, scaling up a classicalpsychophysics paradigm to over 1000 trials. We test a diverse set ofvision models using a simple readout from their representations to pre-dict subjects’ reaction times. We observe a steady improvement acrossmodel generations, with both architecture and training objective con-tributing to alignment, and transformer-based models trained with theDINO self-supervised objective showing the strongest performance. Toinvestigate the source of this improvement, we propose a metric to quan-tify the object-centric component of representations by measuring patchsimilarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately.We further show that matching the Gram matrix of supervised trans-former models, capturing similarity structure across image patches, withthat of a self-supervised model through distillation improves their align-ment with human behavior, converging with the prior finding that Gramanchoring improves DINOv3’s feature quality. Together, these resultsdemonstrate that self-supervised vision models capture object structurein a behaviorally human-like manner, and that Gram matrix structureplays a role in driving perceptual alignment. Code and data are availablehere.