I Walk the Line: Examining the Role of Gestalt Continuity in Object Binding for Vision Transformers

Alexa R. Tartaglini; Michael A. Lepori

cs.CVApr 10, 2026

I Walk the Line: Examining the Role of Gestalt Continuity in Object Binding for Vision Transformers

Authors: Alexa R. Tartaglini, Michael A. Lepori

Organizations: Stanford University · Brown University

Abstract

Object binding is a foundational process in visual cognition, during which low-level perceptual features are joined into object representations. Binding has been considered a fundamental challenge for neural networks, and a major milestone on the way to artificial models with flexible visual intelligence. Recently, several investigations have demonstrated evidence that binding mechanisms emerge in pretrained vision models, enabling them to associate portions of an image that contain an object. The question remains: how are these models binding objects together? In this work, we investigate whether vision models rely on the principle of Gestalt continuity to perform object binding, over and above other principles like similarity and proximity. Using synthetic datasets, we demonstrate that binding probes are sensitive to continuity across a wide range of pretrained vision transformers. Next, we uncover particular attention heads that track continuity, and show that these heads generalize across datasets. Finally, we ablate these attention heads, and show that they often contribute to producing representations that encode object binding.

Figures & tables

Panel of Figure 1 — Figure 1: Object binding probe stimuli and results. (a) An example stimulus pair from the dataset used to train and test the binding probes (Object and $\text{ Scrambled}_{\textit{location}}$ ). More example images from the Object and Scrambled datasets can be found in Appendix A . (b) Difference between object binding probe test accuracy on Blobs Object stimuli vs. Scrambled stimuli (taking the maximum probe test accuracy over $\{\text{Scrambled}_{\textit{orientation}},\text{ Scrambled}_{\textit{location}}\}$ ), averaged across models. See Appendix B for results on the Curves dataset. (c) Maximum object binding probe test accuracy for Object stimuli from the Blobs and Curves datasets across model layers. The quantities in the parentheses show the delta with maximum probe test accuracy on the Scrambled datasets; for example, (+6.1) indicates that maximum probe test accuracy on the Object dataset is 6.1 points higher than the probe test accuracy on either of the Scrambled datasets. Scrambled dataset accuracies are reported explicitly in Appendix B . Overall, we find that binding probes achieve higher performance when the stimuli obey continuity, even when controlling for proximity and surface similarity of the patches. This indicates that continuity plays a role in generating binding representations across vision transformer models. See Appendix B for full results.

Panel of Figure 2 — Figure 2: Procedure for identifying Gestalt continuity heads. (a) Top row: We parametrically vary continuity by translating a target perimeter patch. At $t=0$ , the patch is maximally aligned with its neighbors, creating a continuous curve. Bottom row: We create a control condition where the target patch at $t=0$ contains the same pixel information but randomly rotated as to disrupt continuity between neighboring patches. b At each $t$ and for each attention head, we measure attention from neighboring patches containing the curve (black squares) to the target patch (blue square). We then divide the maximum neighbor attention for the continuity condition by the same quantity for the control condition to get $S(t)$ , the continuity sensitivity index. c $S(t)$ for an attention head from CLIP ViT-B/16 with a high continuity sensitivity index (Layer 1, Head 10). Attention peaks at the maximally-aligned time step $t=0$ . See Appendix C for all models.

Figure 3: Attention ablation results. We plot the impact of ablating Gestalt continuity heads (

\Delta_{\text{{ablate}}}

) on either the Object dataset or the Scrambled datasets. We plot the most selective ablations across layers for each model on the Blobs dataset, where selectivity is defined as

\Delta_{\text{{ablate}}}^{\text{Object}}-\Delta_{\text{{ablate}}}^{\text{Scrambled}}

. We find that ablating Gestalt continuity heads impacts probe performance more on the Object dataset than the Scrambled datasets. See Appendix F for full results (including Curves results) and additional controls.

Figure 4: Difference between object binding probe test accuracy on Curves Object stimuli vs. Scrambled stimuli. These results follow the Blobs results shown in Figure 1 b.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Randomly selected stimuli from the Blobs dataset (“Object” version).

Figure 6: Randomly selected stimuli from the Curves dataset (“Object” version).

Figure 7: Randomly selected stimuli from ImageNet (with corresponding object segmentations).

Figure 8: Randomly selected triplets of stimuli showing the Object (top row), $\text{Scrambled}_{\textit{orientation}}$ (middle row), and $\text{Scrambled}_{\textit{location}}$ (bottom row) datasets for Blobs stimuli. Specifically, these examples show Scrambled stimuli for the $16\times 16$ image patch size condition.

Panel of Figure 9 — Figure 9: Randomly selected stimuli from the binding datasets used to train object binding probes in Section 4 .

Figure 10: Object binding probe test accuracy for Object, $\text{{Scrambled}}_{\text{{orientation}}}$ , and $\text{{Scrambled}}_{\text{{location}}}$ datasets across layers for each model on the Blobs dataset.

Figure 11: Object binding probe test accuracy for Object, $\text{{Scrambled}}_{\text{{orientation}}}$ , and $\text{{Scrambled}}_{\text{{location}}}$ datasets across layers for each model on the Curves dataset.

Figure 12: Maximum object binding probe test accuracy on Object vs. Scrambled datasets. These results are exactly the results shown in Figure 1 c, except the Scrambled results are shown explicitly. For the Scrambled results, we select the maximal probe test accuracy over

\{\text{Scrambled}_{\textit{orientation}},\text{ Scrambled}_{\textit{location}}\}

Figure 13: Tuning curves for attention heads with the top-5 highest continuity sensitivity index scores (left panels) and the bottom-5 continuity sensitivity index scores (right panels) computed on the Blobs dataset for each model. Tuning curves

S(t)

are computed following the procedure described in Figure 2 .;

S(t)

essentially represents how many more times the attention head is sensitive to the curve-completing condition vs. the non curve-completing control, thereby giving a measure of sensitivity to continuity across patches (as opposed to mere pixel-level similarity or proximity). The

x

-axis shows

t

, i.e. the amount of displacement of the moving image patch (in pixels) from its original, maximally-aligned position. The

y

-axis shows the continuity sensitivity index,

S(t)

. The timestep

t=0

gives the scores shown in the heatmaps in Figure 14 . Above each tuning curve is the layer (L) and head (H) index, followed by

S(0)

in parentheses.

Figure 14: Continuity sensitivity index scores computed on the Blobs dataset at

t=0

for all heads, layers, and models. Examples of stimuli from the Blobs dataset are shown in Figure 5 . Heads with the highest continuity scores tend to be clustered within the first four model layers.

Figure 15: Continuity sensitivity index scores computed on the Curves dataset at

t=0

for all heads, layers, and models. Examples of stimuli from the Curves dataset are shown in Figure 6 .

Figure 16: Continuity sensitivity index scores computed on the ImageNet dataset at

t=0

for all heads, layers, and models. Examples of stimuli from the ImageNet dataset are shown in Figure 7 .

Figure 17: Pearson’s

r

(left) and Spearman’s

\rho

for continuity sensitivity index scores at

t=0

between Blobs, Curves, and ImageNet datasets. See Appendix A for details of how the

t=0

stimuli are generated for each dataset.

Figure 18: Attention ablation results for Blobs (left) and Curves (right) datasets. As in Figure 3 , each bar shows the

\Delta_{ablate}

(i.e. decrement in object binding probe test accuracy as a result of the ablation). We visualize the most selective ablation per model (i.e., the layer at which ablation impacts the Object dataset more than the Scrambled datasets). While Figure 3 only shows the maximum control ablation (taken across both Scrambled datasets), here we show results for both Scrambled datasets separately.

Figure 19: Selectivity (i.e.

\Delta_{ablate}\text{Object}-\Delta_{ablate}\text{Scrambled}

) for the

\text{Scrambled}_{\textit{orientation}}

dataset (Blobs). Here, the continuity heads are ablated on the Object version of the dataset to compute

\Delta_{ablate}\text{Object}

; the same set of heads are ablated on the

\text{Scrambled}_{\textit{orientation}}

dataset to compute

\Delta_{ablate}\text{Scrambled}

Figure 20: Selectivity (i.e.

\Delta_{ablate}\text{Object}-\Delta_{ablate}\text{Scrambled}

) for the

\text{Scrambled}_{\textit{location}}

dataset (Blobs). Here, the continuity heads are ablated on the Object version of the dataset to compute

\Delta_{ablate}\text{Object}

; the same set of heads are ablated on the

\text{Scrambled}_{\textit{location}}

dataset to compute

\Delta_{ablate}\text{Scrambled}

Figure 21: Selectivity (i.e.

\Delta_{ablate}\text{Object}-\Delta_{ablate}\text{Scrambled}

) for the

\text{Scrambled}_{\textit{orientation}}

dataset (Curves). Here, the continuity heads are ablated on the Object version of the dataset to compute

\Delta_{ablate}\text{Object}

; the same set of heads are ablated on the

\text{Scrambled}_{\textit{orientation}}

dataset to compute

\Delta_{ablate}\text{Scrambled}

Figure 22: Selectivity (i.e.

\Delta_{ablate}\text{Object}-\Delta_{ablate}\text{Scrambled}

) for the

\text{Scrambled}_{\textit{location}}

dataset (Curves). Here, the continuity heads are ablated on the Object version of the dataset to compute

\Delta_{ablate}\text{Object}

; the same set of heads are ablated on the

\text{Scrambled}_{\textit{location}}

dataset to compute

\Delta_{ablate}\text{Scrambled}

Explore similar work

Jun 2, 2026

Formalizing the Binding Problem

Representations of the world, arguably, contain information about features (e.g. something is blue, something is a circle) but also information about which features are part of the same object (e.g. the circle is blue), which we call binding information. Any system with the ability to understand scenes with multiple objects must be able to solve the binding problem: it needs to know which features belong together. However, despite work showing that Vision Transformers (ViTs) know which patches belong together, it is not known whether current deep learning models learn to exhibit binding information, i.e., for features. We may believe that there is not much binding information, after all misattributing features to wrong objects is a common failure of ViT-based architectures, especially in scenes with objects sharing features. Here we formalize the binding problem with an information-theoretic approach, and introduce a probing method to measure binding information in model representations. We perform experiments on ViTs, measuring binding from different components of the architecture, such as the image summary token [CLS] or the spatial tokens. We use datasets with different binding challenges, such as feature sharing, occlusion, and natural features, while comparing the performance of several pre-trained ViTs. Overall, our research demonstrates binding as a key ingredient to strong visual recognition and reasoning.

Lianghuan Huang, Yihao Li, Saeed Salehi +3

Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA · Department of Computer and Information Science, University of Pennsylvania · Machine Learning Group, Technical University of Berlin, Berlin, Germany +3

Visual Representation Learning Vision Transformer

Ask PDF

May 25, 2026

Binding Visual Features Point by Point

Despite success on standard benchmarks, vision language models display persistent failures on tasks involving processing of multi-object scenes, including many tasks that are relatively easy for humans. Recent work has found that these failures may stem from a basic inability to accurately bind object features in-context, a challenge that is referred to as the "binding problem" in cognitive science and neuroscience. The human visual system is thought to solve this binding problem via serial processing, attending to individual objects one at a time so as to avoid interference from other objects. Recent work has proposed "pointing" -- the use of explicit spatial coordinates to refer to objects -- as an analogous solution for vision language models, and found that it improves performance on challenging multi-object tasks. However, it is unclear

\textit{why}

(i.e., on a mechanistic or representational level) this approach improves performance, and how directly this relates to serial processing in human vision. Here, we investigate this question. We find that learning to point-via-text induces an internal visual search routine, and we characterize the mechanisms that support this procedure. We also find that pointing behavior can be generalized to new tasks via fine-tuning, and that doing so eliminates binding errors and enables compositional generalization. These results provide a proof-of-principle that serial processing can solve the binding problem for vision language models just as it does for biological vision.

Udith Haputhanthri, Declan Campbell, Rim Assouel +2

Princeton University · Mila – Quebec AI Institute · Université de Montréal

Vision-Language Models VLM Interpretability Vision-Language Grounding Visual Search

Ask PDF

May 29, 2026

How can embedding models bind concepts?

Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP's binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization. Code is publicly available at https://github.com/oshapio/binding-concepts-complexity.

Arnas Uselis, Darina Koishigarina, Seong Joon Oh

University of Tübingen, Tübingen · 2KAIST AI

Multimodal Embedding Vision-Language Alignment Compositional Generalization

Ask PDF