Recent research has shown that contrastive vision-language models such as CLIP often lack fine-grained understanding of visual content. While a growing body of work has sought to address this limitation, we identify a distinct failure mode in the CLIP family, which we term center bias, that persists even in recent model variants. Specifically, CLIP tends to disproportionately focus on the central region of an image, overlooking important objects located near the boundaries. This limitation is fundamental as failure to recognize relevant objects makes it difficult to perform any sophisticated tasks that depend on those objects. To understand the underlying causes of the limitation, we conduct analyses from both representation and attention perspectives. Using interpretability methods, i.e., embedding decomposition and attention map analysis, we find that relevant concepts especially those associated with off-center objects vanish from the model's embedding in the final representation due to information loss during the aggregation of visual embeddings, particularly the reliance on pooling mechanisms. Finally, we show that this bias can be alleviated with training-free strategies such as visual prompting and attention redistribution by redirecting models' attention to off-center regions.
Figures & tables
Figure 1: A representative example from What’sUp . When both objects are centrally aligned (left), CLIP correctly assigns high probability to “a pot and a chair”. However, when the pot is moved off-center (right), CLIP disproportionately focuses on the central chair, leading to a sharp drop in probability for the correct answer and a preference for “a chair.”
Figure 2: Examples of center and off-center configurations in the GRID dataset.
Model
center ( ↑ )
off-center ( ↑ )
center bias ( ↓ )
OpenAI CLIP ViT-B/32
62.9
31.9
31.0
OpenAI CLIP ViT-L/14
94.2
76.7
17.5
OpenCLIP ViT-B/32
65.7
10.5
55.2
OpenCLIP ViT-L/14
87.6
46.0
41.6
RoBERTaCLIP ViT-B/32
64.8
15.7
49.1
CoCa ViT-B/32
39.0
6.70
32.3
Table 1: Performance of CLIP-based models on the center and off-center subsets of What’sUp . Despite strong performance on centrally located objects, all models show significantly lower accuracy on off-center examples, indicating a systematic center bias.
Figure 3: Mean top-1 accuracy vs. object size on a 7×7 GRID . Left: OpenCLIP ViT-B/32. Right: OpenCLIP ViT-L/14. Despite improvements with increasing object size, a persistent center–off-center performance gap remains.
Figure 4: Comparison of top-weighted concepts from CLIP when the dog is placed at the center (left) versus off-center (right). When centered, CLIP assigns meaningful weights to “dog”-related concepts, whereas these concepts disappear when the dog is placed off-center
Figure 5: Attention maps for the [CLS] token and other visual tokens in the final layer. Brighter pixels indicate higher attention. While the [CLS] token focuses mainly on the table, patch tokens capture information from other important objects (the bottle and the socket). This suggests that center bias arises not from a lack of available information, but from information loss during pooling.
Figure 6: Effect of visual prompting on an off-center object. Without prompting (left), the model fails to assign meaningful “dog”-related concepts. By adding a visual prompt (red circle) around the object (right), relevant concepts such as pets and dog reappear, indicating improved attention to the off-center region.
Model
center ( ↑ )
off-center ( ↑ )
center bias ( ↓ )
improv. off-center ( ↑ )
Visual Prompting
OpenAI CLIP ViT-B/32
54.2
21.8
32.4
-10.1
OpenAI CLIP ViT-L/14
87.6
76.7
10.9
0.00
OpenCLIP ViT-B/32
59.0
23.3
35.7
+12.8
OpenCLIP ViT-L/14
81.0
56.5
24.5
+10.5
RoBERTaCLIP ViT-B/32
52.3
21.7
30.6
+6.00
Table 2: Performance comparison of visual prompting (VP) and attention redistribution (AR) across various CLIP variants on What’sUp . VP generally improves off-center performance but exhibits inconsistent effects across models. In contrast, AR improves off-center performance and overall accuracy more consistently.
Model
center ( ↑ )
off-center ( ↑ )
center bias ( ↓ )
OpenAI CLIP ViT-B/32
19.1
15.7
3.40
+ VP
17.7
13.2
4.50
+ AR
19.2
15.7
3.50
OpenAI CLIP ViT-L/14
30.5
26.0
4.50
+ VP
30.4
25.2
5.2
+ AR
31.2
26.7
4.50
Table 3: Mean performance across CIFAR10, FashionMNIST, and Food101 comparing visual prompting (VP) and attention redistribution (AR) on the synthetic 7×7 GRID with s=1 . AR improves both off-center performance and overall accuracy.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Position
Mean Distance
Median Distance
Left
0.273
0.288
Right
0.261
0.276
Under
0.192
0.198
On
0.064
0.060
Appendix
Table 10
Model
Center
Off-center
Center-bias
OpenAI CLIP ViT-B/32
62.9
32.9
30.00
OpenAI CLIP ViT-L/14
94.2
80.4
13.8
OpenCLIP ViT-B/32
65.7
8.0
57.7
OpenCLIP ViT-L/14
87.6
41.9
45.7
Appendix
Table 11
Figure 7: Additional examples of concept vanishing under different object positions.
Figure 8: Additional examples of concept vanishing under different object positions.