Unsupervised Semantic Segmentation Facilitates Model Understanding

Self-supervised learning (SSL) has produced a diverse land-scape of vision transformers (ViTs) whose pretrained representations sup-port a wide range of downstream tasks. Towards a better understandingof these models, a body of works has assessed the mechanics of their self-attention as well as which types of information they capture across theirrepresentations, revealing, e.g., stark differences between models trainedwith contrastive learning (CL) vs. masked image modeling (MIM). How-ever, the total of these advances on model understanding has to datenot yet fully permeated a larger community, where, e.g., insights thatare specific to CL models are still at times generalized to MIM models.To make model understanding straightforward and intuitive for a broadcommunity, we propose a simple and easily interpretable visualizationprotocol. Our protocol is based on visualizing unsupervised semanticsegmentation results – yet by no means do we focus on top segmenta-tion performance. Instead, our protocol allows us to easily convey modelbehavior that consistently emerges across images. Benchmarked on a di-verse set of SSL models across layers and representations, our protocolallows us to gain novel insights into distinct positional biases and scalingbehaviors, including, e.g., strong boundary artifacts in DINOv3-Largemodel tokens. These novel insights come on top of more easily conveyinga range of previous findings. Our protocol further allows us to clearly vi-sually convey and distinguish between positional effects and the closelyrelated yet distinct locality bias, the latter being much more extensivelystudied in the literature so far. Our protocol is publicly available6 , serv-ing to catalyze further model understanding for a broad community.