Vision language models (VLMs) have achieved remarkable success in broad visual understanding, yet they remain challenged by object-centric reasoning on rare objects due to the scarcity of such instances in pretraining data. While prior efforts alleviate this issue by retrieving additional data or introducing stronger vision encoders, these methods are still computationally intensive during finetuning VLMs and don't fully exploit the original training data. In this paper, we introduce an efficient plug-and-play module that substantially improves VLMs' reasoning over rare objects by refining visual tokens and enriching input text prompts, without VLMs finetuning. Specifically, we propose to learn multi-modal class embeddings for rare objects by leveraging prior knowledge from vision foundation models and synonym-augmented text descriptions, compensating for limited training examples. These embeddings refine the visual tokens in VLMs through a lightweight attention-based enhancement module that improves fine-grained object details. In addition, we use the learned embeddings as object-aware detectors to generate informative hints, which are injected into the text prompts to help guide the VLM's attention toward relevant image regions. Experiments on two benchmarks show consistent and substantial gains for pretrained VLMs in rare object recognition and reasoning. Further analysis reveals how our method strengthens the VLM's ability to focus on and reason about rare objects.
Figures & tables
Figure 1: Comparison on rare object recognition: (a) shows that LLaVA tends to predict the “bollard” as a common object “traffic light”, while (b) demonstrates that our method corrects LLaVA by predicting “bollard” and providing reasoning through visual enhancement and text prompt enrichment with object hints, both based on the learned multi-modal class embeddings.
Figure 2: Visual attention on the object “bollard” from the CODA-LM dataset. The attention weights across layers show that LLaVA-1.5-7B allocates less attention to the target object region. Brighter colors indicate higher attention weights.
Figure 3: Overview of the model framework, which consists of three main components: (a) a multi-modal class embedding learning module, which fuses object visual features with synonym-augmented text features; (b) a visual token enhancement module, which applies a cross-attention mechanism between class embeddings and image visual tokens in VLMs; and (c) a text hints injection module, which leverages the learned multi-modal class embeddings for object identification and enriches the text prompt with object hints.
Model / Metrics
Barrier ↑
Other ↑
Cone ↑
Light ↑
Sign ↑
Vehicle ↑
VRU ↑
All ↑
LLaVA-1.5-7B [ 20 ]
39.3
40.2
54.5
54.4
48.8
48.9
40.5
46.5
LLaVA-1.5-7B + Ours
68.3
68.3
84.9
61.4
48.2
73.0
56.1
72.8
Qwen2.5-VL-7B [ 3 ]
70.9
62.5
84.9
48.8
52.1
66.5
54.6
67.9
Qwen2.5-VL-7B + Ours
79.8
73.8
91.7
64.3
54.3
71.0
58.4
75.4
InternVL3-8B [ 45 ]
59.7
65.5
73.3
64.4
52.3
66.9
59.6
65.4
InternVL3-8B + Ours
76.4
69.3
85.8
67.5
55.1
73.8
66.2
74.2
Table 1: GPT score comparison on the CODA-LM dataset. “+ Ours” denotes our parameter-efficient refinement built on frozen baseline VLMs. Models marked with † are task-specific finetuned models on CODA-LM, and models marked with ‡ are training-free methods.
Model / Metrics
Aerial ↑
Maritime ↑
Vehicle ↑
Sports ↑
Construction ↑
All ↑
LLaVA-1.5-7B [ 20 ]
16.5
29.8
14.0
15.5
12.2
20.9
LLaVA-1.5-7B + Ours
21.5
49.4
12.0
34.5
10.0
33.2
Qwen2.5-VL-7B [ 3 ]
18.0
33.5
26.5
27.9
13.3
27.4
Qwen2.5-VL-7B + Ours
26.7
52.3
34.0
36.7
14.2
39.3
InternVL3-8B [ 45 ]
26.0
46.1
38.0
35.2
20.0
37.4
InternVL3-8B + Ours
35.4
61.4
42.1
45.7
24.3
48.2
Table 2: GPT score comparison on the GeoBench-VLM dataset. “+ Ours” denotes our parameter-efficient refinement built on frozen baseline VLMs. The model marked with † is finetuned on GeoBench-VLM, and models marked with ‡ are training-free methods.
Figure 4: Ablation study of visual refinement and text hints for LLaVA-1.5-7B on the CODA-LM dataset.
Figure 5: Comparison of different k for LLaVA-7B on CODA-LM. “Detection Accuracy” is the top- k detection accuracy of multi-modal class embeddings for objects. “VLM Accuracy” measures how VLMs recognize objects with/without our hints. “Trust Rate” is the ratio of VLMs’ output that aligns with our hints.
Figure 6: Attention weights comparison between (a) LLaVA-1.5-7B and (b) LLaVA-1.5-7B + Ours.
Figure 7: Interpretation of image hidden states for object class “bus” in LLaVA-1.5-7B via logit lens [ 12 ] on CODA-LM dataset.
How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.
Vision Language Models (VLMs) are well known for hallucinating non-existent objects in images. Objects with missing parts present a unique challenge for VLMs, stemming from both real-world knowledge bias and the scarcity of such images in training data. We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario: when vision-language models fail to recognize that an essential component of an object has been removed. Across ten leading models, we observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception. We further ask whether granting models access to image processing tools (e.g., cropping, contrast adjustment) enables autonomous inspection to resolve these failures. We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement, indicating that this failure mode cannot be addressed through current prompting or post-hoc correction techniques. Our findings highlight a fundamental limitation of current VLM for inspection and monitoring tasks and underscore the need for architectural or training-level interventions that enable models to override internal expectations when confronted with contradictory evidence.
Wenqi Marshall Guo, Qingyun Qian, Shiyu Zhou +2
Department of CMPS, University of British Columbia, Canada · Weathon Software, Canada · Department of Computer Science, University of British Columbia, Canada
Recent advances in Vision-Language Models (VLMs) have benefited from Reinforcement Learning (RL) for enhanced reasoning. However, existing methods still face critical limitations, including the lack of low-level visual information and effective visual feedback. To address these problems, this paper proposes a unified multimodal interleaved reasoning framework ForeSight, which enables VLMs to See Further with low-level visual cues and Think Deeper with effective visual feedback. First, it introduces a set of low-level visual tools to integrate essential visual information into the reasoning chain, mitigating the neglect of fine-grained visual features. Second, a mask-based visual feedback mechanism is elaborated to incorporate visual reflection into the thinking process, enabling the model to dynamically re-examine and update its answers. Driven by RL, ForeSight learns to autonomously decide on tool invocation and answer verification, with the final answer accuracy as the reward signal. To evaluate the performance of the proposed framework, we construct a new dataset, Character and Grounding SalBench (CG-SalBench), based on the SalBench dataset. Experimental results demonstrate that the ForeSight-7B model significantly outperforms other models with the same parameter scale, and even surpasses the current SOTA closed-source models on certain metrics.
Zhiheng Wu, Tong Wang, Shuning Wang +2
1Baidu Inc. · †Independent researcher. · 2Zhejiang University +1