cs.CVOct 9, 2026
SaveIn-Context Learning Can Help Vision-Language Models Overcome Training Priors
Organizations: Princeton University
Abstract
Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples ( for Qwen3-VL 235B and for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from to . We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.
Figures & tables
Figure 1: Visual ICL overcomes training prior dominance. (A) A VLM answers “2” for a three-legged peacock under zero-shot prompting; label-balanced demonstrations shift the answer to the visually grounded “3.” (B) Scaling ICL shots yields a pp accuracy gain on counterfactual images while real images remain near ceiling. (C) Mechanistically, demonstrations redirect attention from the prior cue (“peacock”) to the grounding cue (“leg”).
Figure 2: Data design. (A) Our benchmark covers five vision-centric tasks, each contrasting a prior answer with the grounded visual answer. (B) Each task contains complex, simple, and no-prior settings to separate prior override from ordinary visual task learning. (C) We probe the training prior by comparing counterfactual examples, where the prior and visual answers conflict, with real examples, where they agree.
Figure 3: Qwen3-VL scales with in-context demonstrations. Accuracy vs. number of in-context demonstrations for Qwen3-VL model family on complex, simple, and no-prior settings across tasks.
| Model | 0-shot | 16-shot | |
|---|---|---|---|
| Qwen3-VL 32B | 69.9% | 79.6% | |
| Qwen2.5-VL 32B | 49.1% | 64.7% | |
| Qwen3.5 27B | 44.1% | 74.5% | |
| InternVL3.5 38B | 52.4% | 70.6% | |
| Gemma-4 31B | 75.2% | 86.6% |
Table 1: In-context scaling of 30B model families on the complex setting. Accuracies and deltas are reported with % and micro-averaged over query entries.
Figure 4: Scaling and diagnostic analysis of Qwen3-VL across model sizes on the complex setting. (a) Aggregate accuracy on the real and counterfactual subsets with varying numbers of in-context learning (ICL) shots, for the 4B, 8B, 32B, and 235B Qwen3-VL models. (b) Per-task 0-to-16-shot accuracy change ( Acc) on the overall, counterfactual, and real subsets, broken down by task (counting, color, shape, direction, spatial relation) and model size. Positive values (blue) indicate tasks where ICL improves accuracy; negative values (red) indicate degradation.
Figure 5: Two complementary routes to visual grounding. (A) Masking visual tokens that encode prior cues weakens the learned-training prior pathway. (B) In-context demonstrations strengthen attention to image-grounded evidence. Both interventions shift the prediction from a prior-driven error toward the visually grounded answer.
Figure 6: Demonstrations reduce reliance on prior cues. (a) From 0-shot to 16-shot, VLMs shift answer-token attention away from prior cues and toward grounding cues. (b) On the complex setting with greedy decoding, prior-cue masking improves accuracy over both zero-shot and matched random-token masking for Qwen3-VL 32B and Gemma-4 31B on all five tasks. Table 13 reports all attention shifts and full masking accuracies.
Figure 7: ICL gains weaken for harder counting cases. The plot shows the accuracy change from 0-shot to 16-shot prompting.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Prompts used for querying models across tasks and settings.
| Task | Setting | Source | Labels | Count | Real | CF | Edited | HV |
|---|---|---|---|---|---|---|---|---|
| Counting | Simple | Procedural (Pillow) | 320 | |||||
| Counting | No-prior | Procedural (Pillow) | 320 | |||||
| Counting | Complex | VLMs are Biased (regenerated) [ 2 ] | 200 | 90 | 110 | ✓ | ✓ | |
| Color ID | Simple | Procedural (Pillow) | {G, O, Br, Bk, Pk, R} | 240 | ||||
| Color ID | No-prior | Procedural (Pillow) | {G, O, Br, Bk, Pk, R} | 240 | ||||
| Color ID | Complex | Generated + Edited; task design [ 1 ] | {G, O, Br, Bk, Pk, R} | 240 | 120 | 120 | ✓ | ✓ |
Table 2: Dataset statistics. (a) Original benchmark sources, labels, and counts, including real and counterfactual (CF) subsets. The real/CF split applies only to complex examples. Citations identify image sources or, for color and shape, task designs. Procedural images use Pillow or SVG; generated, regenerated, and edited images use Gemini. Regeneration reproduces a source image; HV denotes human validation. (b) Counting datasets used in Section 4.3 ; high-count shapes have 50 images per label.
| Task | Input | Real | Counterfactual | Overall |
|---|---|---|---|---|
| Object Direction | Image | 93.3 | 45.9 | 77.5 |
| Caption Only | 42.1 | 2.7 | 29.0 | |
| Spatial Relation | Image | 83.2 | 41.2 | 69.2 |
Table 3: Configuration-prior diagnostics. Zero-shot accuracy (%) for Qwen3-VL 32B on the original complex setting. Caption-only input replaces the direction image with object-identity text from its source caption. Overall accuracy weights the 160 real and 80 counterfactual examples in each task.
Figure 9: Dataset construction. For each task, images are rendered, generated, regenerated, or edited across the simple, no-prior, and complex settings, then paired with closed answer labels and validation controls for label-balanced ICL evaluation.
| Task | Accepted | Rejected | Candidates | Acceptance (%) |
|---|---|---|---|---|
| Counting | 110 | 192 | 302 | 36.4 |
| Color ID | 120 | 15 | 135 | 88.9 |
| Shape ID | 120 | 13 | 133 | 90.2 |
| Object Direction | 80 | 78 | 158 | 50.6 |
| Spatial Relation | 80 | 128 | 208 | 38.5 |
| All Tasks | 510 | 426 | 936 | 54.5 |
Table 4: Human validation statistics for the original counterfactual portion of the complex setting. Acceptance requires both annotators to select the intended label independently and confirm that the image meets the quality criteria.
| Task | Stage | Prompt |
|---|---|---|
| Counting | Edit | {new_total_legs} leg(s) total. Edit this image so the {animal_name} has exactly {new_total_legs} leg(s). {arrangement hint}. Every leg must match the size and texture of the original legs. The result must be photo-realistic with all {new_total_legs} leg(s) clearly visible and distinct. Preserve the same background, facial features, lighting, and camera angle. |
| Direction | Edit | Edit this image so the {object_name} faces {target_direction}ward. {front_part} should point toward the {top|bottom} of the frame, with the {object_name} {pose_desc}. The {object_name} must remain clearly visible and recognizable. The result must be photo-realistic. Preserve the same background, lighting, object size, and camera angle. |
| Spatial | Caption | You generate counterfactual captions by changing ONLY the spatial relation between the same two objects. Caption: {original_caption}. Choose a preposition from {allowed_preps} that makes the resulting caption feel the most unnatural. Produce {num_variants} candidate captions. |
| Spatial | Edit | Edit this image. From the viewer’s perspective, the {obj1} is {relation} the {obj2}. Both objects must remain fully visible. Maintain photorealistic appearance. Preserve the same background, lighting, and object appearance. |
| Color ID | Generate | Generate a photorealistic image of a {animal_name}. The {animal_name} should be the central object in the image. Include realistic natural scenery around the animal. The animal must be clearly visible. |
| Color ID | Edit | Edit this image: change the color of the {animal_name} from {original_color} to {target_color}. The {animal_name} must remain clearly recognizable. Preserve the same background, lighting, and camera angle. |
Table 5: Generation and editing prompts used for image construction in the complex setting.
| Mode | Model family | Max tokens | Temp. | Top- | Top- | Rep. penalty | Presence penalty |
|---|---|---|---|---|---|---|---|
| Standard non-thinking | Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5 | vLLM: ; HF: omitted | |||||
| Standard non-thinking | Gemma-4 | vLLM: ; HF: omitted | |||||
| Thinking mode | Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5 | N/A | N/A | ||||
| Thinking mode | Gemma-4 | N/A | N/A |
Table 6: Sampling parameters for VLM benchmark evaluation. Values follow each model card’s recommended decoding settings and are used for the vLLM behavioral runs. Mechanistic experiments (Hugging Face Transformers) instead use greedy decoding and therefore do not use the temperature, top- , top- , repetition, or presence parameters listed here.
Figure 10: ICL scaling across model families and settings. Rows: model families (aggregated, then Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5, Gemma-4). Columns: settings. Qwen3-VL scales consistently; Qwen2.5-VL less so. Qwen3.5 and InternVL3.5 degrade beyond 8–16 shots. Gemma-4 achieves the strongest and most stable scaling.
| InternVL3.5 | Qwen2.5-VL | Qwen3-VL | Qwen3.5 | Gemma-4 | |||||||
| Setting | Shots | 4B | 38B | 3B | 32B | 4B | 32B | 4B | 27B | E4B | 31B |
| Counting | |||||||||||
| No-prior | 0 | 43.1 | 28.8 | 35.6 | 41.9 | 32.5 | 43.8 | 58.8 | 49.7 | 35.0 | 23.8 |
| 16 | 49.8 ( ) | 76.5 ( ) | 19.9 ( ) | 70.8 ( ) | 35.6 ( ) | 71.9 ( ) | 16.8 ( ) | 80.4 ( ) | 28.6 ( ) | 72.4 ( ) | |
| +6.6 | +47.8 | -15.7 | +28.9 | +3.1 | +28.2 | -41.9 | +30.8 | -6.4 | +48.5 | ||
| Simple | 0 | 92.5 | 63.8 | 87.5 | 84.4 | 88.1 | 93.8 | 95.0 | 95.6 | 63.8 | 54.4 |
Table 7: Cross-family accuracies by task and setting. Accuracies are percentages; standard deviations and are in percentage points. For 16-shot rows, parentheses report standard deviations over sampled demonstration contexts. Best 0- and 16-shot entries per row are bold; for the largest per-row improvement is bold. A dash indicates that a valid run was unavailable, and is omitted unless both endpoints are available.
| Subset | Shots | Both correct (%) | Gemma-4 only (%) | Qwen3-VL only (%) | Both incorrect (%) | Agreement (%) | Agreement (pts) |
|---|---|---|---|---|---|---|---|
| Real | 0 | 82.4 | 5.9 | 4.5 | 7.2 | 89.6 | — |
| 16 | 80.9 | 6.6 | 4.5 | 8.1 | 88.9 | 0.7 | |
| Counterfactual | 0 | 51.7 | 15.4 | 9.2 | 23.7 | 75.4 | — |
| 16 | 72.6 | 13.5 | 3.4 | 10.4 | 83.1 | +7.7 |
Table 8: Paired correctness of Gemma-4 31B and Qwen3-VL 32B. Both models answer the same complex queries, split into real and counterfactual examples, at 0 and 16 shots. Agreement is the share of cases where both models are correct or both are incorrect. Percentages use the same aggregation as Figure 4 (a), and entries are rounded independently.
Table 9: Expanded complex-setting dataset and partial results. Exact expanded counterfactual and total counts are unavailable. The partial evaluation covers Qwen3-VL 32B at two shot counts; 16-shot entries are mean accuracy standard deviation across ten fixed demonstration contexts, while zero-shot has no context variance. Accuracies are percentages; standard deviations and are in percentage points (pp).
Table 10: Masking and length-matched control ablations. Panels (a), (b), and (c) report accuracy (%) for Qwen3-VL 32B on the complex setting with greedy decoding. Ground mask removes grounding-cue tokens from the query image; Blank replaces each demonstration image with a same-size blank image. Unrelated uses demonstrations from another split of the same task, while Vision only removes question text from both the demonstrations and query. Panel (d) reports simple -setting endpoints from the shuffled-label sweep in Figure 11 ; is the change from zero-shot accuracy in percentage points. Compute limits restrict panel (c) to counting.
Figure 11: Qwen3-VL unlearns under label shuffling. In the simple setting, shuffled demonstration labels reduce performance below the zero-shot baseline, with larger drops for larger models.
| Demonstration Task | Images | Shots | Accuracy (%) |
| None | — | 0 | 32.1 |
| Counting | Complex (standard) | 16 | 54.4 |
| Counting | 2D shapes | 16 | 43.5 |
| Color ID | Animals | 16 | 34.6 |
| Color ID | 2D shapes | 16 | 20.2 |
| Color ID | Mixed splits | 16 | 27.1 |
Table 11: Cross-task demonstrations. Accuracy on complex counting queries for Qwen3-VL 32B. Mixed color demonstrations are sampled uniformly across splits.
| Counting | Original | Explicit Language Prior |
|---|---|---|
| 0-shot | 33.0 | 28.9 |
| 16-shot | 53.7 | 51.0 |
Table 12: Counting under an explicit language prior. Accuracy (%) for standard counting prompts and prompts prepended with an explicit prior statement. The explicit prior lowers zero-shot accuracy, but the 0-to-16-shot gain remains comparable to the original prompt.
Figure 12: Reasoning (thinking) vs. non-reasoning ICL accuracy. Overall accuracy for Qwen3-VL-4B (Instruct vs. Reasoning), Qwen3-VL-32B (Instruct vs. Reasoning), Gemma-4-E4B-it, and Gemma-4-31B-it. Solid lines denote non-thinking mode; dotted lines denote thinking mode. Enabling thinking does not consistently improve ICL scaling.
Figure 13: Answer-label distribution shifts under ICL in the complex setting. For Qwen3-VL-32B-Instruct and Gemma4-31B, we plot the probability assigned to the correct label at 0-shot and 16-shot for color identification (top row) and counting (bottom row), separated into counterfactual and real entries. Probability mass on counterfactual answers increases substantially with demonstrations, while probabilities on real entries remain largely intact.
Table 13: Mechanistic results across Qwen3-VL and Gemma-4. (a) Changes in answer-token attention to prior cues, grounding cues, and other query-image tokens. Attention is normalized over query-image visual tokens and averaged per token within each group. (b) Zero-shot accuracy, zero-shot masking interventions, and 16-shot accuracy on all five tasks. The random mask matches the prior-cue mask’s token budget. Object direction and spatial relation use object identity as a proxy prior cue.
Figure 14: Prior-cue masking examples. Two queries from the complex setting with their identified prior-cue tokens (red overlay on the query image) and the candidate-normalized answer distributions at 0-shot, after prior-cue token ablation, and at 16-shot. Left: a one-legged swan, where the 0-shot model places nearly all mass on the prior-default answer “2”; ablating the prior-cue tokens shifts mass onto the ground-truth “1”; and 16-shot demonstrations produce a similar shift. Right: a pink raven, where 0-shot mass concentrates on the training prior default “black”, and both prior-cue ablation and 16-shot ICL move probability mass onto the ground-truth “pink”. Prior-cue ablation reproduces the qualitative effect of in-context demonstrations.