How much scene context a single object carries is a well-studied question in human scene perception, yet how this capacity is organized in vision-language models (VLMs) remains poorly understood, with direct implications for the robustness of these models. We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects. Presenting VLMs with single objects on masked backgrounds, we probe their ability to infer both fine-grained scene category and coarse superordinate context (indoor vs. outdoor). We found that single objects support above-chance inference at both levels, with performance modulated by the same object properties that predict human scene categorization. Object identity, scene, and superordinate predictions are partially dissociable: accurate inference at one level neither requires nor guarantees accurate inference at the others, and the degree of coupling differs markedly across models. Mechanistically, object representations that remain stable when background context is removed are more predictive of successful contextual inference. Scene and superordinate schemas are grounded in fundamentally different ways: scene identity is encoded in image tokens throughout the network, while superordinate information emerges only late or not at all. Together, these results reveal that the organization of contextual inference in VLMs is more complex than accuracy alone suggests, with behavioral and mechanistic signatures
Figures & tables
Figure 1: Experimental design. Each image condition (full scene or object-only) is paired with three prompt types: (A) scene category, (B) superordinate category, and (C) object identity. We analyze the intermediate image token representations corresponding to the objects (shown in black).
Figure 2: Classification accuracy across image conditions and tasks. Bars show accuracy under the full-scene (original image), and object-only (single foreground object on a masked background) conditions. For scene classification, solid bars indicate normal accuracy (exact label match) and hatched bars indicate relaxed accuracy (any semantically valid label accepted). Error bars denote the standard error of the mean.
Figure 3: Log-odds coefficients from multivariate logistic regression predicting classification accuracy in the object-only condition, as a function of object size, frequency (Freq.), specificity (Spec.), and object type (anchor vs. local). Error bars denote 95% confidence intervals. Significance levels: ∗∗p<.01 , ∗∗∗p<.001 .
Model
Task
Obj. correct
Obj. incorrect
Marginal
LLaVA
Scene
65.6% (+7.0%)
41.8% (-16.8%)
58.6%
Scene (relaxed)
90.0% (+4.5%)
74.6% (-10.9%)
85.5%
Superordinate
96.0% (+2.2%)
88.7% (-5.1%)
93.8%
InternVL
Scene
60.4% (+7.0%)
31.0% (-22.4%)
53.4%
Scene (relaxed)
90.6% (+5.9%)
65.7% (-19.0%)
84.7%
Superordinate
90.6% (+5.6%)
67.1% (-17.9%)
85.0%
Table 1: Scene and superordinate accuracy conditioned on object identification correctness in the object-only condition.
Figure 4: Representational stability of object-patch tokens across transformer layers. Left: Mean cosine similarity between hidden-state activations of object patches under the full-scene and object-only conditions, averaged across all images, for LLaVA and InternVL. Higher values indicate that object representations are less modulated by the presence of background context. Center and right: Difference in mean cosine similarity between correctly and incorrectly classified trials ( Δ cosine similarity) at each layer. Values above zero indicate that correctly classified trials have more stable object representations. Shaded regions denote 95% confidence intervals.
Figure 5: ROC-AUC between top-3 patch logit strength and binary classification accuracy at each transformer layer. Values above the dashed line indicate that logit strength in image patch tokens is predictive of classification accuracy at that layer. Markers indicate statistically significant layers.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Scene classification accuracy in the object-only condition by scene category, for LLaVA and InternVL. Solid bars indicate normal accuracy (exact label match) and hatched bars indicate relaxed accuracy (any semantically valid label accepted). Orange bars correspond to indoor scene categories (bathroom, bedroom, kitchen, living room) and blue bars to outdoor categories (skyline, coast, forest, mountain). Error bars denote the standard error of the mean.
Figure 7: Scene and superordinate classification accuracy for the object-only condition compared to label-based baselines. VLM Object Only shows VLM performance when only the foreground object is visible. VLM Mean Tokens replaces the image with a semantically uninformative average visual token while providing the object name in the prompt. LLM receives the object name but no image. For scene classification, solid bars indicate normal accuracy and hatched bars indicate relaxed accuracy. Superordinate classification uses normal scoring only. Error bars denote the standard error of the mean.
Model
Task
Predictor
β^
SE
z
p
LLaVA
Scene
Cosine sim.
0.487
0.051
9.54
<.001
Object size
0.385
0.061
6.30
<.001
Superordinate
Cosine sim.
1.017
0.109
9.36
<.001
Object size
1.344
0.302
4.46
<.001
Object
Cosine sim.
0.728
0.058
12.57
<.001
Object size
-0.373
0.056
-6.71
<.001
Appendix
Table 2: Logistic regression predicting task accuracy from cosine similarity and object size (both z-scored). Coefficients are log-odds. Object size is included as a control for the number of patches contributing to the cosine similarity estimate.
Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45% to 58.51%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.
Yuxing Cheng, Yuan Wu, Yi Chang
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples (+27.2% for Qwen3-VL 235B and +15.1% for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from +9.7% to +30.3%. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.
Visual inputs are often assumed to improve language understanding in multimodal models. We examine this assumption by asking whether vision-language models (VLMs) can distinguish useful visual evidence from incidental image context in lexical judgments. We use human concreteness and imagery ratings because they span words with varying expected visual relevance, from abstract and low-imagery words to concrete and high-imagery words. We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings, most sharply when visual evidence is least relevant. Through probing and canonical correlation analysis, complemented by an attribution case study, we find that real-image contexts are associated with representational shifts and greater sensitivity to spurious visual cues, coinciding with weaker recoverability of the targeted lexical properties. We further show that instructing models to focus solely on textual content at inference time can reduce this degradation, with the clearest gains on these vulnerable subsets. Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.