cs.CVApr 2, 2026
SaveVLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
Organizations: University of California, Riverside 1 · MBZUAI2
Abstract
Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perception, even when the required information is still present in their internal representations. Prior work has attributed this ``hidden-in-plain-sight'' gap to the language model, but the cause remains unexplained. In this work, we demonstrate that this gap arises from the language model's lack of semantic labels for fine-grained visual details: when visual entities can be mapped to known concepts, VLMs bypass visual comparison and reason through language; when they cannot, VLMs resort to brittle and hallucinated descriptions. We verify this across semantic correspondence, synthetic shape matching, and face matching, and find that VLMs perform much better when the relevant entities are nameable than when they are unnamable. Mechanistically, Logit Lens analysis confirms that VLMs explicitly recover semantic labels for nameable entities and surface more unique tokens compared to unnameable entities. Furthermore, we show that this limitation can be addressed: teaching completely arbitrary names for unknown entities improves performance. More importantly, task-specific finetuning yields even stronger generalization without relying on language priors, i.e., through real visual perception. Our findings suggest that current VLM failures on visual tasks reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning. Code and datasets are available at https://github.com/Patchwork53/VLMs-Need-Words-COLM2026.
Figures & tables
Figure 1: Overview of the correspondence task framed as multiple-choice VQA. The VLM must match the reference point “REF” in the first image to one of four candidate regions (A, B, C, D) in the second image. Regions can either be Named or have No-Name . We evaluate VLMs in three setups: Direct answer, Chain-of-Thought, and Representation Probing.
| Model | Size | Subset | Direct (D) | CoT (C) | C D | Rep. Probe (R) | R max(D,C) |
| Qwen3VL | 2B | Named | 36.6 | 57.4 | 20.8 | 68.7 | 11.3 |
| No-Name | 32.4 | 42.2 | 9.8 | 60.2 | 18.0 | ||
| 4B | Named | 52.5 | 62.9 | 10.4 | 68.4 | 5.5 | |
| No-Name | 37.2 | 46.7 | 9.5 | 62.1 | 15.4 | ||
| 8B | Named | 53.8 | 65.3 | 11.5 | 68.6 | 3.3 | |
| No-Name | 40.3 | 47.9 | 7.6 | 61.6 | 13.7 |
Table 1: Semantic correspondence results on SPairs71k. R denotes the representation-probing accuracy from the best-performing layer of the language model. We report layer-wise performance in Appendix Fig. 8 . The R max(D, C) column shows the gap between Representation Probing and the best textual output.
Figure 2: Example of our 2D shape correspondence task. Face correspondence example is presented in Appendix Fig. 11 .
| Model | Known | Faces | 2D Shapes | |||||||||
| Direct (D) | CoT (C) | C D | Rep. Probe (R) | R max(D,C) | Direct (D) | CoT (C) | C D | Rep. Probe (R) | R max(D,C) | |||
| Qwen3VL | 2B | ✓ | 77.1 | 59.2 | 17.9 | 97.2 | 20.1 | 54.1 | 97.3 | 43.2 | 100 | 2.7 |
| 41.1 | 37.4 | 3.7 | 70.8 | 29.7 | 29.0 | 27.3 | 1.7 | 74.2 | 45.2 | |||
| 4B | ✓ | 84.2 | 83.2 | 1.0 | 93.2 | 9.0 | 93.5 | 99.4 | 5.9 | 100 | 0.6 | |
| 56.9 | 55.7 | 1.2 | 73.4 | 16.5 | 48.4 | 40.0 | 8.4 | 91.7 | 43.3 | |||
| 8B | ✓ | 83.9 | 85.2 | 1.3 | 92.8 | 7.6 | 99.7 | 99.9 | 0.2 | 100 | 0.1 | |
Table 2: Comparison of Direct, Chain-of-Thought, and Representational Probe accuracy across tasks, models, and subsets. R denotes the representation-probing accuracy from the best-performing layer of the language model. We report layer-wise performance in Appendix B . columns show the gain of CoT over Direct (C D) and of the representational probe over the stronger of the two baselines (R max(D,C)). Bold entries indicate that the probing substantially exceeds verbal output for unknown entities.
Figure 3: Logit Lens analysis. Top row: layerwise decoded tokens for a known shape (star) and a known face (Jungkook) in Gemma3-12B, showing the progression from semantically unrelated tokens to exact labels and encyclopedic associations. For the unknown shape, the Logit Lens probe surfaces a hallucinated token (“legs”). Bottom row: Mean Jaccard Distance across layers for shapes and faces in Gemma3-12B and Qwen3VL-8B. Known entities ( blue ) yield consistently higher Mean Jaccard Distance than unknown entities ( orange ), confirming greater semantic discernibility in the hidden representations.
Figure 5: Set of procedurally-generated shapes tested. denotes the shape complexity variable: number of anchor points for squiggles and grid size for mazes. Orange denotes the train dataset (Squiggles n=30) and blue denotes the test dataset.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Chain-of-Thought reasoning allows the VLM to recover the semantic label for each point explicitly and essentially convert the task to a verbal problem. However, when the points lack a concrete semantic label, the transcription process becomes harder, leading to hallucinations, as shown by the VLM mislabeling point B as the top of the rear window instead of the bottom.
Figure 8: Layer-wise Representation Probing Performance for Nameable and No-Name reference points from the Semantic Correspondence ( Min et al., 2019 ) task.
Figure 9: Layer-wise Representation Probing Performance for Known (Celebrity) and Unknown (AI-Generated) Faces .
Figure 10: Layer-wise Representation Probing Performance for 2D Known and Procedurally Generated (Squiggle) Shapes .
Figure 11: Face correspondence task. Known celebrities (left) and AI-generated faces (right).
| VLM | Recognized Faces |
| All | Bruce Lee, BTS J-Hope, BTS Jimin, BTS Jin, BTS Jungkook, BTS RM, BTS Suga, BTS V, Jack Ma, Jackie Chan, Masayoshi Son, Song Heungmin, Steven Yeun, Yao Ming |
| Qwen-2B | BTS Jimin, BTS Jin, BTS Jungkook, BTS Suga, BTS V, Bruce Lee, Jack Ma, Jackie Chan, Song Heungmin, Yao Ming |
| Qwen-4B | BTS J-Hope, BTS Jimin, BTS Jin, BTS RM, BTS Suga, BTS V, Bruce Lee, Jack Ma, Jackie Chan, Steven Yeun, Yao Ming |
| Qwen-8B | BTS Jimin, BTS Jin, BTS Jungkook, BTS Suga, BTS V, Bruce Lee, Jack Ma, Jackie Chan, Steven Yeun, Yao Ming |
| Gemma-4B | BTS Jimin, BTS Jin, BTS Jungkook, BTS RM, BTS V, Bruce Lee, Jack Ma, Jackie Chan, Masayoshi Son, Steven Yeun |
| Gemma-12B | BTS Jimin, BTS Jin, BTS Jungkook, BTS RM, BTS Suga, BTS V, Jackie Chan, Steven Yeun |
Table 5: VLM-specific Face Correspondence dataset.
| Ground Truth | InternVl3.5 14B Predictions (Images 1–4) | Correct |
| Barack Obama | Barack Obama, Barack Obama, Barack Obama, Barack Obama | 4/4 |
| Cristiano Ronaldo | IDK , Cristiano Ronaldo, Paul Walker, IDK | 1/4 |
| Donald Trump | Donald Trump, Donald Trump, Donald Trump, Donald Trump | 4/4 |
| Lionel Messi | Cristiano Ronaldo, Cristiano Ronaldo, Chris Hemsworth, Aaron Hernandez | 0/4 |
| BTS J-Hope | Park Yohan, Park Yoo-chun, Jungkook, Lee Min-ho | 0/4 |
| BTS Jimin | Lee Seung-gi, Yohan Lemoine, Park Yohan, Park Yohan | 0/4 |
Table 6: InternVL 3.5 14B face recognition probing results. IDK means the VLM responded that it was unable to identify the person. We test additional faces of very famous celebrities to confirm InternVL-3.5 failure in face recognition.
| Model | Known | 2D Shapes | |||||
| Direct (D) | CoT (C) | C D | Rep. Probe (R) | R max(D,C) | |||
| InternVL 3.5 | 2B | ✓ | 46.1 | 76 | 29.9 | 92 | 16 |
| 26.1 | 27.7 | 1.6 | 77.7 | 50 | |||
| 8B | ✓ | 66.4 | 89.9 | 23.5 | 89.8 | -0.1 | |
| 27.7 | 27.6 | -0.1 | 86.8 | 59.1 | |||
| 14B | ✓ | 55.1 | 90.9 | 35.8 | 89.5 | -1.4 | |
Table 7: Comparison of Direct, Chain-of-Thought, and Representational Probe accuracy for InternVL-3.5. R denotes the representation-probing accuracy from the best-performing layer of the language model. We report layer-wise performance in Appendix Fig. 10 . columns show the gain of CoT over Direct (C D) and of the representational probe over the stronger of the two baselines (R max(D,C)). Bold entries indicate that the probe substantially exceeds the best prompted baseline for Unknown 2D-shapes.
| Model | Size | Subset | Direct (D) | CoT (C) | C D | Rep. Probe (R) | R max(D,C) |
| Gemma-3 | 4B | Named | 25.7 | 26.3 | 0.6 | 38.6 | 12.3 |
| No-Name | 24.8 | 26.2 | 1.4 | 34.6 | 8.4 | ||
| 12B | Named | 31.2 | 35.9 | 4.7 | 39.4 | 3.5 | |
| No-Name | 27.5 | 28.4 | 0.9 | 33.5 | 5.1 |
Table 8: Semantic correspondence results for Gemma-3 on SPairs71k. Gemma-3-4B performs near chance and therefore does not exhibit the trends observed in more capable VLMs. In contrast, Gemma-3-12B shows a larger gap between Representation Probing and the best textual strategy, R max(D, C) , on No-Name keypoints, consistent with the pattern observed for the Qwen3-VL and InternVL3.5 families in Table 1 .
Figure 12: Qualitative Logit Lens example from Gemma-3-12B. The results show that some decoded tokens are highly informative of the person’s identity and affiliations, while Fig. 12(f) shows an example of a visual token that encodes generic semantic information.
Figure 13: All Gemma3-12B Logit Lens tokens for (Unknown Shape 1).
Figure 14: All Gemma3-12B Logit Lens tokens for . (Unknown Shape 2)
| Name Set | Names |
| Random | 0QK2Z2, 5F1FT3, OZ0W0M, ALCTDF, DNXXB0, ION17F, K0XQNF, UTNWY7, JT1GWQ, 1VZS0M |
| Human | John, Mary, Charles, Elizabeth, William, Margaret, James, Catherine, Robert, Dorothy |
| Ordinary | cup, brick, anchor, fork, bell, shield, blade, horn, nest, arrow |
| Task | Image Type | Example Prompt | Expected Answer |
| Naming | Single Image | “What is this shape called?” | Mary |
| Yes/No | Single Image | “Is this a cup?” | Yes. |
| Choice | Single Image | “Which of the following: fork, cup, bell, arrow?” | cup |
| Comparison | Single Image (Two Shapes) | “Which object is the arrow, A or B?” | A |
| Description | Single Image | “Can you describe the shape called John?” | A name derived from the Latin ‘Iohannes’… |
Figure 15: Representation Probing accuracy increases after teaching VLM names through finetuning. For Qwen, the improvement is consistent across all layers, matching the gap reported in Figure 10(a) . For Gemma, there is a sharp decline for all models after Layer 4, followed by a sharp recovery at Layer 19. This matches the behavior for Known shapes shown in Figure 10(d) .
Figure 16: Chain-of-thought example on unknown shapes. Without a semantic anchor, the VLM generates hallucinated descriptions that actively mislead its final answer.
| Faces | Sem. Corr. | |||||
| Model | Split | Base | FT | Split | Base | FT |
| Acc. | Acc. | Acc | Acc | |||
| Qwen3VL-2B | Known | 77.1 | 89.8 | Named | 36.6 | 52.3 |
| Unknown | 41.1 | 60.4 | No-Name | 32.4 | 38.1 | |
| Gemma3-4b | Known | 49.8 | 60.8 | Named | 26.1 | 35.50 |
| Unknown | 32.4 | 38.8 | No-Name | 25.7 | 27.80 | |
Table 9: Semantic and face correspondence performance after finetuning on squiggles ( ). Base Acc. is the pre-finetuning baseline. FT Acc. is after finetuning.
Figure 17: Distribution of Qwen3-8B judge scores on open-ended CLEVR-Change responses produced by Qwen3VL-2B before and after finetuning on squiggle correspondence. The finetuned model achieves a higher mean judge score (5.087 vs. 4.507) and produces more high-scoring responses (scores 8–10: 33.1% vs. 24.7%).
Figure 18: Art Style task example. The ground truth answer is B — both the reference portrait and option B are by Paul Cézanne.
| Textual Evaluation | Representation Probing | ||||
| Model | Subset | Direct | CoT | MaxSim | Gram Dist. |
| Qwen3VL-2B | Known | 65.4 | 59.4 | 65.2 | 62.0 |
| Unknown | 48.6 | 45.8 | 51.4 | 51.6 | |
| Qwen3VL-8B | Known | 72.0 | 73.0 | 68.2 | 62.6 |
| Unknown | 47.8 | 37.2 | 51.2 | 51.8 | |
| Gemma3-4B | Known | 35.6 | 37.8 | 53.6 | 47.8 |
Table 10: Art style matching accuracy on model-specific Known and Unknown subsets. Known artworks are created by painters that the corresponding VLM can identify, while Unknown artworks are created by painters that it does not identify. Bold indicates the better-performing subset for each model and evaluation strategy.
More in this topic
VLM Reasoning
Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.
See all 846 papers