Finding Distributed Object-Centric Properties in Self-Supervised Transformers

Samyak Rawlekar; Amitabh Swain; Yujun Cai; Yiwei Wang; Ming-Hsuan Yang; Narendra Ahuja

cs.CVMar 27, 2026

Finding Distributed Object-Centric Properties in Self-Supervised Transformers

Authors: Samyak Rawlekar, Amitabh Swain, Yujun Cai, Yiwei Wang, Ming-Hsuan Yang, Narendra Ahuja

Organizations: University of Illinois Urbana-Champaign · University of Queensland · UC Merced

Abstract

Self-supervised Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in [CLS] token attention maps of the final layer. However, these maps often contain spurious activations resulting in poor localization of objects. This is because the [CLS] token, trained on an image-level objective, summarizes the entire image instead of focusing on objects. This aggregation dilutes the object-centric information existing in the local, patch-level interactions. We analyze this by computing inter-patch similarity using patch-level attention components (query, key, and value) across all layers. We find that: (1) Object-centric properties are encoded in the similarity maps derived from all three components ( $q, k, v$ ), unlike prior work that uses only key features or the [CLS] token. (2) This object-centric information is distributed across the network, not just confined to the final layer. Based on these insights, we introduce Object-DINO, a training-free method that extracts this distributed object-centric information. Object-DINO clusters attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects. We demonstrate Object-DINO's effectiveness on two applications: enhancing unsupervised object discovery (+3.6 to +12.4 CorLoc gains) and mitigating object hallucination in Multimodal Large Language Models by providing visual grounding. Our results demonstrate that using this distributed object-centric information improves downstream tasks without additional training.

Figures & tables

Panel of Figure 1 — (a) Head contribution for complete network

Figure 3: Object-DINO Overview . Our training-free algorithm identifies a distributed set of object-centric heads from a pre-trained models such as DINO. First, for every head across all layers, we compute the patch similarity maps from its query ( $A_{q}$ ), key ( $A_{k}$ ), and value ( $A_{v}$ ) representations. Second, these three maps are ensembled and flattened to create a vector representing each head’s localization pattern. Third, all heads are clustered based on these pattern. Fourth, guided by the established observation that object-centric information is most prevalent in the final layer [ 4 , 28 , 23 ] , we automatically identify the object Cluster ( $c_{\text{obj}}$ ) using the criterion that it contains the highest prevalence of final-layer heads. Finally, aggregating the similarity maps from only the object-centric heads ( $\mathcal{H}_{\text{obj}}$ ) produces a high-fidelity object localization, effectively filtering noise from non-object-centric heads. For visualization, we invert the maps to show bright colors for objects.

A_{r}^{\ell,h}=\text{softmax}\!\left(\frac{\tilde{r}^{\ell,h}\cdot(\tilde{r}^{\ell,h})^{\top}}{\tau}\right)

Algorithm 1 Object-DINO: Head Clustering

Figure 4: Mitigating MLLM Hallucination with Object-DINO Visual Guidance. Our training-free decoding strategy computes two separate logit distributions. The standard branch uses the original image $u$ and a general prompt $T_{u}$ (e.g., "describe this image"). The guidance branch uses the Object-DINO’s object map $v$ and a prompt $T_{v}$ (e.g., "describe the highlighted regions"). Here, R is the text prompt generated so far. We then add these logits, $Logits(y|T_{u},R,u)+\alpha\cdot Logits(y|T_{v},R,v)$ , to amplify tokens consistent with the visual evidence, thus correcting the hallucinations (e.g., "Two" $\rightarrow$ "Three" dogs)

Model	Method	VOC07	VOC12	COCO20K
Dino-V3 [ 24 ]	TokenCut [ 28 ]	26	30.3	19.8
Dino-V3 [ 24 ]	+ Ours	30.8 ( $\uparrow$ 4.8 )	36.0 ( $\uparrow$ 5.7 )	23.4 ( $\uparrow$ 3.6 )
Dino-V2 [ 20 ]	TokenCut [ 28 ]	16.2	18.3	11.9
Dino-V2 [ 20 ]	+ Ours	25.7 ( $\uparrow$ 9.5 )	30.7 ( $\uparrow$ 12.4 )	19.7 ( $\uparrow$ 7.8 )

Table 1: Unsupervised Object Discovery. We evaluate Object-DINO by integrating it into the TokenCut [ 28 ] framework. We replace TokenCut’s baseline strategy (using all final-layer heads) with our automatically discovered set of object-centric heads. This change yields substantial performance gains (in blue ) for both DINO-V2 and DINO-V3 models across all datasets.

Figure 5: Experimental results of MME on the hallucination subset

Method	LLaVA-1.5 [ 17 ]			Instruct-BLIP [ 8 ]			Qwen-VL [ 3 ]
	Acc	P	F1	Acc	P	F1	Acc	P	F1
Regular	77.4	73.3	79.2	74.6	71.2	76.4	79.8	80.1	79.7
VCD	77.1	72.1	79.4	77.2	74.2	78.4	81.3	80.6	81.5
M3ID	78.2	73.5	80.2	77.4	73.6	79.1	82.0	81.4	82.1
RITUAL	78.8	74.4	80.5	78.7	74.5	80.3	82.8	83.1	82.7
MARINE	83.9	-	82.7	79.13	-	77.1	-	-	-

Table 2: POPE Results. Results on POPE benchmark. Higher accuracy, precision, and F1 indicate better performance. The best results in bold, and the second-best are underlined.

Method	LLaVA-1.5 [ 17 ]		Instruct-BLIP [ 8 ]
	Cs	Ci	Cs	Ci
Regular	26.2	9.4	31.2	11.1
VCD [ 13 ]	24.4	7.9	30.0	10.1
M3ID [ 10 ]	21.4	6.3	30.8	10.4
RITUAL [ 30 ]	22.4	6.9	26.6	8.9
Woodpecker [ 32 ]	24.9	7.5	31.2	10.8

Table 3: CHAIR Results. Results on CHAIR benchmark with the maximum number of new tokens set to 64. Lower CHAIRs and CHAIRi (Ci) indicate less hallucination. Best results are in bold, second-best are underlined.

Method	VOC07	VOC12	COCO20K
TokenCut [ 28 ]	26.0	30.3	19.8
+ Our Head (final layer)	27.5 ( $\uparrow$ 1.5 )	31.4 ( $\uparrow$ 1.1 )	20.5 ( $\uparrow$ 0.7 )
+ Our Head (all layers)	30.8 ( $\uparrow$ 4.8 )	36.0 ( $\uparrow$ 5.7 )	23.4 ( $\uparrow$ 3.6 )

Table 4: Layer Head Ablation. This table validates our two hypotheses. (1) Row-2 : Selecting only object-centric heads from the final layer improves over the TokenCut baseline (+1.5 CorLoc on VOC07), confirming that some final-layer heads contribute noise. (2) Row-3 : Selecting object-centric heads from all layers leads to further gains (+3.3 additional CorLoc on VOC07), proving that object-centric information is distributed across intermediate layers and accounts for the majority of the performance improvement.

Figure 6: Impact of Attention Components on Head Selection . CorLoc performance across three datasets (VOC2007, VOC2012, COCO20k) when using different attention components (

Q,K,V

) versus their ensemble to identify object-centric heads via Object-DINO. The performance ordering is

Q<V<K<\text{Ensemble}

across all datasets. This demonstrates that Ensemble leads to identification of robust object-centric heads.

Figure 7: Qualitative comparison of captions generated by LLava using regular decoding (red box) vs our method (green box).

Figure 8: Evaluation of optimal number of clusters

K

using Davies-Bouldin Score (lower is better). We select

K=5

for all our experiments.

Figure 9: Guidance weight ablation for MLLM hallucination mitigation. We observe that the

C_{s}

and

C_{i}

scores are lowest for

\alpha

= 0.4, making it our optimal choice

Figure 10: Tau ablation. We show the optimal choice of

\tau

for object centric signal is

\in(10,100)

Panel of Figure 11 — Figure 11: Evolution of localizations with temperature. For visualization, we invert the maps to show bright colors for objects.

Panel of Figure 12 — Figure 12: Final Layer Head Visualization. Ensemble similarity maps $A_{ens}$ for all 12 heads in Layer 11 of DINO-V3. For visualization, we invert the maps to show bright colors for objects.

$W_{q}$	$W_{v}$	$W_{k}$	CorLoc
0.1	0.2	0.7	17.9
0.2	0.3	0.5	19.1
0.3	0.3	0.4	19.4
0.33	0.33	0.33	19.8

Table 5: Ablation of Ensemble Weights. We report CorLoc for different weight configurations from Eq. 2 . Performance peaks at 19.8 with a uniform average (

W_{q}=W_{k}=W_{v}

= 0.33), confirming that all three components contribute valuable and complementary object signals. We therefore use this uniform average for

A_{ens}

in all experiments.

Method	Avg. Latency (s)	GPU Memory (MB)	CHAIR ${}_{\text{S}}$ ( $\downarrow$ )
Regular	3.44 (×1.00)	15,778 (×1.00)	55
VCD	6.91 (×2.01)	16,634 (×1.05)	54.4
OPERA	24.70 (×7.18)	22,706 (×1.44)	52.6
Woodpecker	10.68 (×3.10)	22,199 (×1.41)	57.6
HALC	22.61 (×6.51)	23,084 (×1.46)	51
DeGF	13.89 (×4.04)	19,119 (×1.21)	48.8

Table 6: Efficiency comparison. For each method, we show the inference latency per instance, peak GPU memory and the performance CHAIR

{}_{\text{S}}

on the LLaVA-1.5 model with max token 128

Panel of Figure 19 — (a) Head contribution for complete network

Panel of Figure 14 — Figure 14: Object-centric information in patch-level interactions from MAE [ 11 ] . We visualize the inter-patch similarity maps ( $A_{q},A_{k},A_{v}$ ) computed from the Query, Key, and Value representations of patch tokens in a MAE ViT-B/16 model. For visualization, we invert the similarity maps so objects appear bright.

Panel of Figure 21 — (a) Head contribution for complete network

Panel of Figure 16 — Figure 16: Qualitative comparison of captions generated by Regular decoding (Red) and Ours (Green)

Explore similar work

Human-like Object Grouping in Self-supervised Vision Transformers
Mar 14, 2026Hossein Adeli, Seoyoung Ahn, Andrew Luo +3Object-Centric Representation Learning Self-Supervised Visual Representation Learning
Zuckerman Mind Brain Behavior Institute, Columbia University, New York · Department of Social Science and AI, Hankuk University of Foreign Studies, Seoul · University of Hong Kong, Hong Kong +2
What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization
May 12, 2026Ryota Yoshihashi, Masahiro Kada, Satoshi Ikehata +2Visual Attention Vision Transformer
Institute of Science Tokyo · 2DENSO IT Laboratory · 3National Institute of Informatics
$A^2$ : Smaller Self-Supervised ViTs Localize Better than Larger Ones
Jun 2, 2026Sreehari Rammohan, Huy Ha, Carl VondrickVisual Attention Vision Transformer
a Columbia University · b Stanford University

Mar 14, 2026J/K move · Enter open · S save

Human-like Object Grouping in Self-supervised Vision Transformers

Hossein Adeli, Seoyoung Ahn, Andrew Luo, Mengmi Zhang+2

Zuckerman Mind Brain Behavior Institute, Columbia University, New York · Department of Social Science and AI, Hankuk University of Foreign Studies, Seoul · University of Hong Kong, Hong Kong+2

Object-Centric Representation Learning Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception remains poorly understood. Here, we introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes, scaling up a classical psychophysics paradigm to over 1000 trials. We test a diverse set of vision models using a simple readout from their representations to predict subjects' reaction times. We observe a steady improvement across model generations, with both architecture and training objective contributing to alignment, and transformer-based models trained with the DINO self-supervised objective showing the strongest performance. To investigate the source of this improvement, we propose a novel metric to quantify the object-centric component of representations by measuring patch similarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately. We further show that matching the Gram matrix of supervised transformer models, capturing similarity structure across image patches, with that of a self-supervised model through distillation improves their alignment with human behavior, converging with the prior finding that Gram anchoring improves DINOv3's feature quality. Together, these results demonstrate that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.

Abstract

Explore similar work

Human-like Object Grouping in Self-supervised Vision Transformers

What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization

A2A^2A2: Smaller Self-Supervised ViTs Localize Better than Larger Ones

$A^2$ : Smaller Self-Supervised ViTs Localize Better than Larger Ones