Objects as Addresses: Instance Binding in Vision Transformer Attention
Organizations: Polytechnique Montréal · Université TÉLUQ
Abstract
Humans naturally parse visual scenes into coherent individual objects rather than scattered features, a capability cognitive science calls object binding. While recent work shows that pretrained Vision Transformers (ViTs) intrinsically acquire this ability, how native self-attention computations mechanistically keep object instances separate remains unresolved. By studying this across eight pretrained ViTs, we found that instance separation is expressed more strongly in attention routing than in token content: selected query-key relations distinguish individual objects more reliably than value similarity or residual representations. Inside heads, patches belonging to one individual tend to read from a common image region, which we term their attention address. Visually distinct parts of one person share an address, whereas identical duplicated instances receive separate, movable addresses. These shared destinations arise in part because query interactions with the spatial structure of keys direct attention toward consistent regions for one object and elsewhere for another. In DINOv2 and most other backbones we tested, editing this query component at the selected head redirects its attention address toward another individual. Grouping patches by this query component separates same-category instances, reaching .615 FG-ARI versus .289 for the same clustering of final-layer features, which tend to merge them.
Figures & tables
| Across backbones, held-out COCO | Different parts of one person | |||||||
| Original | Request | Requested | Address | |||||
| Backbone | Head | Q/K | Readout | Dim. | AUC | |||
| DINO | L7h3 | .623 | .644 | .661 | .660 | Vector similarity | ||
| DINOv2 | L10h2 | .738 | .823 | .821 | .807 | Residual | 768 | .651 |
| DINOv2-reg | L11h3 | .730 | .805 | .806 | .804 | Value | 64 | .720 |
| DINOv3 | L10h1 | .765 | .815 | .814 | .801 | Query | 64 | .856 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Image set | Size and role | Experiments using it |
|---|---|---|
| VOC development | 102 images; 62,362 pairs | Select layers, heads and relations; fit spatial bases without object labels; select grouping radii |
| VOC held-out | 101 images; 71,390 pairs | Native relations, shared reading, spatial requests and controls |
| COCO pair test | 1,414 images; 938,606 pairs | Transfer the same frozen native relations and spatial bases |
| ADE20K pair test | 530 images; 176,732 pairs | Transfer the native routing–value comparison |
| COCO parts | 215 images; 35,100 pairs | Compare different body parts; predict addresses; intervene once per image with five controls |
| COCO grouping | 1,587 images | Group all patches; evaluate instance assignments and object masks |
| Set | Backbone | Residual | Query | Key | Query–key | Attention | Value |
|---|---|---|---|---|---|---|---|
| VOC | DINO | .593 | .644 | .620 | .642 | .627 | .598 |
| DINOv2 | .563 | .740 | .734 | .741 | .795 | .636 | |
| DINOv2-reg | .584 | .749 | .770 | .786 | .791 | .605 | |
| DINOv3 | .605 | .801 | .747 | .782 | .810 | .596 | |
| MAE | .621 | .777 | .764 | .794 | .812 | .700 | |
| CLIP | .627 | .715 | .722 | .755 | .779 | .648 |
| Model | Checkpoint |
|---|---|
| DINO | vit_base_patch16_224.dino |
| DINOv2 | vit_base_patch14_dinov2.lvd142m |
| DINOv2-reg | vit_base_patch14_reg4_dinov2.lvd142m |
| DINOv3 | vit_base_patch16_dinov3.lvd1689m |
| MAE | vit_base_patch16_224.mae |
| OpenCLIP | vit_base_patch16_clip_224.openai |
| Held-out VOC | COCO | |||||
| Full attention | Direct | Full attention | Direct | |||
| Backbone | pattern | attention | Gain | pattern | attention | Gain |
| DINO | .667 | .627 | .631 | .608 | ||
| DINOv2 | .873 | .795 | .818 | .738 | ||
| DINOv2-reg | .853 | .791 | .792 | .730 | ||
| DINOv3 | .852 | .810 | .812 | .765 | ||
| Address from | DINO | DINOv2 | DINOv2-reg | DINOv3 | MAE | CLIP | SigLIP | Supervised |
|---|---|---|---|---|---|---|---|---|
| Real attention | .708 | .870 | .859 | .847 | .876 | .744 | .636 | .616 |
| Distance only | .523 | .518 | .520 | .518 | .514 | .519 | .507 | .509 |
| Another image | .520 | .523 | .516 | .529 | .505 | .524 | .508 | .499 |
| Key fit | Same-instance AUC | Address | Movement | Grouping | |||
|---|---|---|---|---|---|---|---|
| Spatial description | VOC | Person parts | error | FG-ARI | mIoU | ||
| Linear | .669 | .871 | .909 | 2.11 | .646 | .556 | .301 |
| Quadratic | .907 | .872 | .940 | 1.90 | .821 | .615 | .429 |
| Cubic | .941 | .869 | .939 | 1.92 | .827 | .594 | .406 |
| Backbone | Current request | Wrong image | Source position |
|---|---|---|---|
| DINO | 1.32 | 1.95 | 2.88 |
| DINOv2 | 1.74 | 2.59 | 3.36 |
| DINOv2-reg | 1.55 | 2.58 | 3.38 |
| DINOv3 | 1.42 | 2.77 | 3.34 |
| MAE | 1.17 | 2.04 | 3.79 |
| CLIP | 1.79 | 2.65 | 3.10 |
| Condition | Edited L10h2 | Selected L11h6 | Selected L12h11 |
|---|---|---|---|
| Intact | |||
| swap | |||
| Matched controls |
| Person parts AUC | Address | B A | Movement | Grouping FG-ARI | |||||
|---|---|---|---|---|---|---|---|---|---|
| Backbone | Head | Request | Best other | Swap | Control | Request | Features | ||
| DINO | L7h3 | .634 | .625 | .84 | .84 | .398 | .194 | ||
| DINOv2 | L10h2 | .942 | .856 | .62 | .82 | .615 | .289 | ||
| DINOv2-reg | L11h3 | .949 | .883 | .72 | .89 | .585 | .265 | ||
| DINOv3 | L10h1 | .944 | .860 | .79 | .88 | .585 | .380 | ||
| MAE | L12h8 | .942 | .762 | .76 | .89 | .617 | .177 | ||
| Grouping rule | FG-ARI | mIoU | mBO i |
|---|---|---|---|
| Spatial request | .615 | .429 | .451 |
| Full attention pattern | .649 | .403 | .436 |
| Final features | .289 | .317 | .356 |
| Final features + coordinates | .403 | .375 | .403 |
| Patch coordinates only | .325 | .229 | .251 |
| SPOT (trained, seven slots) | .356 | .288 | .350 |
| 1,587 images with repeated objects | All 5,000 COCO validation images | |||||||
| Grouping rule | FG-ARI | mIoU | mBO i | mBO c | FG-ARI | mIoU | mBO i | mBO c |
| Spatial request | .548 | .307 | .322 | .302 | .410 | .291 | .300 | .319 |
| Final features | .264 | .227 | .253 | .381 | .264 | .282 | .295 | .374 |
| Final features + coordinates | .364 | .264 | .282 | .308 | .309 | .267 | .278 | .317 |
| SPOT | .336 | .222 | .266 | .441 | .368 | .328 | .349 | .448 |
| Different-instance negatives | ||
|---|---|---|
| Readout | Same category | Different category |
| Full attention pattern | .862 | .915 |
| Attention address | .867 | .850 |
| Spatial request | .866 | .845 |
| Requested destination | .858 | .811 |
| Li et al. (2025) | Ours | |
|---|---|---|
| Question | Does object binding naturally emerge in pretrained ViTs? | How is same-category individuation expressed and used in native attention? |
| Problem | Binding: do two patches belong to the same object? | Individuation: which of two same-category objects does a patch belong to? |
| Pairs | All pairs among patches sampled within an image, including background and different categories | Foreground pairs, with negatives from another instance of the same category |
| Spatial control | No displacement matching | Match distance and direction exactly |
| Main readout | Learned quadratic probe | Native Q/K relations and the spatial request |
| Goal | Characterize binding geometry and effects of editing it | Identify specific layers and heads, then explain and edit their native routing |
Vision Transformer
Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.
See all 384 papers