Organizations: Sussex Centre for Consciousness Science, University of Sussex · AI Alignment Network · The University of Tokyo · RIKEN · Program on Brain, Mind, and Consciousness, Canadian Institute for Advanced Research · Tohoku University
Bistable images such as the duck-rabbit are classic stimuli in which one image supports multiple mutually incompatible interpretations, typically reported one at a time in humans. We ask whether multimodal large language models (MLLMs) show similar report behavior and what internal computations support it. Using the LLaVA family, we study two tractable dimensions: modulability, whether reports can be biased by bottom-up visual cues and top-down linguistic priors, and exclusivity, whether responses commit to a single interpretation. We test both on the canonical duck-rabbit and on synthetic Visual Anagrams to mitigate memorization confounds. Behaviorally, both visual and linguistic manipulations systematically shift reports in human-consistent ways, while responses remain predominantly exclusive. Mechanistically, these effects arise from competing image-token representations, distinct pathways for bottom-up and top-down modulation, and a link between exclusive reporting and object-count encoding. Code and data are available at https://github.com/rtakatsky/mllm-bistable-images.
Figures & tables
Figure 1: Overview of our framework. We test whether MLLM reports on the duck–rabbit show modulability and exclusivity , and then analyze how these behaviors are implemented along the vision-to-language pipeline.
Figure 2: Bottom-up modulation by rotation in LLaVA-1.5-7B under the default query and prefix. Small changes in image orientation shift the relative preference for duck- and rabbit-related outputs for Harper’s version (a) and the generated ambiguous stimuli (b), demonstrating modulability. The two-animal controls (c) show no such reversal: the duck-related first token remains dominant across the same rotation range. In (b), (c), and all aggregated plots in this paper, lines indicate the mean across valid input images and shading indicates ±1 standard deviation.
Figure 3: Bottom-up modulation by local visual cueing: rabbit-vs-duck difference maps for red-circle logit effects in LLaVA-1.5-7B (a) and human fixation density (b), adapted from Hsu and Chen (2025) . Red-circle placement shifts the logit difference in regions qualitatively aligned with the human fixation-density pattern.
Figure 4: Top-down modulation by linguistic priors in LLaVA-1.5-7B . Both prefix-based and semantic cues bias the reported interpretation toward the intended animal, with prefix cues producing the stronger effect.
Figure 5: Exclusivity under beam search ( LLaVA-1.5-7B ). (a) Harper’s version with the bird interpretation forced into the prefix: single-interpretation continuations ( "." , " in the image." ) outweigh enumerating ones ( " and a rabbit" ); the rabbit-prefixed condition behaves alike (Figure 21 ). (b, c) Under the default prefix, each beam continuation is classified as naming a single or multiple animals (Appendix A.2.2 ); curves are the mean ± SD across stimuli: ambiguous stimuli are reported almost exclusively as a single animal, the two-animal controls as multiple.
Figure 6: Layerwise direct logit attribution (DLA) for "_bird" and "_rabb" in LLaVA-1.5-7B . Both interpretation logits rise in later layers, with image-token attention contributing most strongly in middle-to-late layers.
Figure 7: Attention maps of heads contributing most strongly to the bird–rabbit logit difference (blue: bird-side, red: rabbit-side), on the canonical duck–rabbit image. These heads attend to local regions supporting different interpretations.
Figure 8: Example dominant patch map from image-token logit lens, on the same image (blue: bird-dominant, red: rabbit-dominant patches).
Figure 9: Bottom-up modulation tracks image-token evidence. Across rotation and red-circle conditions, the bird–rabbit logit difference broadly follows the balance of bird- and rabbit-dominant patches.
Figure 10: Layerwise resampling ablation for top-down cue runs in LLaVA-1.5-7B , averaged over the model’s valid Visual Anagram stimuli. We patch clean runs (e.g., "It starts with ’d’." ) with corrupted runs (e.g., "It starts with ’x’." ) and measure the effect on the final duck–rabbit logit difference.
Figure 11: Query-side patching explains very little of the modulation effect. This suggests that neither bottom-up nor top-down modulation is primarily driven by changes in text-to-image attention queries.
Figure 12: A small subset of image tokens affects both object-count encoding and exclusive reporting. Each point shows the effect of patching a single image-token embedding from the matched two-animal control into the bistable-image run.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Ten randomly selected stimuli from the 50-stimulus synthetic Visual Anagram set, rendered as in the experiments. Top: duck view ( +45∘ ); bottom: rabbit view ( −45∘ ).
Figure 14: Illustration of the red-circle manipulation. A fixed-radius red circle is swept across Harper’s version on a regular grid, and the model output is recorded at each circle center.
Model
Valid stimuli (of 50)
LLaVA-1.5-7B
44
LLaVA-1.5-13B
42
LLaVA-v1.6-Vicuna-7B
40
LLaVA-v1.6-Mistral-7B
40
Llama3-LLaVA-Next-8B
41
Qwen2-VL-7B-Instruct
41
Appendix
Table 1: Number of Visual Anagram stimuli with a valid decision-boundary fit per model (default prompt).
Figure 15: Rotation-based modulability across all models for Harper’s version. All models show a duck-dominant interpretation near 0∘ , but the sharpness and location of the reversal differ by model.
Figure 16: Rotation-based modulability across all models for the aggregated Visual Anagram stimuli. The same qualitative reversal pattern appears across models, though the reversal profile varies in sharpness and stability.
Figure 17: Red-circle modulation on Harper’s version across LLaVA-family models: rabbit-vs-duck difference maps (as in Figure 3 ), with the human fixation-density map ( Hsu and Chen, 2025 ) shown for reference (right).
Figure 18: Top-down modulation across all models for prefix cues. Prefix cues consistently bias the output toward the intended interpretation.
Figure 19: Top-down modulation across all models for semantic cues. The qualitative effect is similar to the prefix condition, but weaker.
Figure 20: Beam-search results across all models for the bird-prefixed condition. In all models, single-interpretation continuations are preferred to continuations that would naturally enumerate both animals.
Figure 21: Beam-search results across all models for the rabbit-prefixed condition. The same qualitative exclusivity pattern is observed when the opposite interpretation is forced into the prefix.
Figure 22: Beam-continuation classes on aggregated Visual Anagrams under the default prefix, across LLaVA-family models (mean ± SD across stimuli).
Figure 23: Rotation-based modulation on Harper’s duck–rabbit across non-LLaVA models.
Figure 24: Rotation-based modulation on aggregated Visual Anagram stimuli across non-LLaVA models.
Figure 25: Red-circle modulation on Harper’s version across non-LLaVA models: rabbit-vs-duck difference maps, with the human fixation-density map ( Hsu and Chen, 2025 ) shown for reference (right).
Figure 26: Top-down modulation by prefix cues across non-LLaVA models.
Figure 27: Top-down modulation by semantic cues across non-LLaVA models.
Figure 28: Beam-continuation classes on aggregated Visual Anagrams under the default prefix, across non-LLaVA models (mean ± SD across stimuli).
Figure 29: Beam-continuation classes on aggregated two-animal controls (default prefix) across LLaVA-family models (mean ± SD across stimuli).
Figure 30: Prefix cues that name an interpretation ( "It starts with ‘d’." , "It starts with ‘r’." ) applied to stimuli offering no visual evidence for the cued percept (the image shows the other animal) or ambiguous evidence. Curves show the cued percept’s probability mass with the cue (solid) and on the same stimuli with no cue (dashed), mean ± SD across stimuli, for all nine models. Without supporting evidence the cue leaves the cued percept near zero.
Figure 31: Semantic cues, plotted as in Figure 30 , averaged over the four semantic cues on each side (duck: "avian" , "an egg-layer" , "nesting" , "singing" ; rabbit: "terrestrial" , "viviparous" , "herbivorous" , "Easter" ).
Figure 32: Raven–bear beam continuations under the default prefix, classified by the animals they name (Appendix A.2.2 ), across LLaVA-family models. Here bird only collects the bird reading (bird, raven, crow, swan), bear/dog only the mammal reading (bear, dog, panda, polar bear), and both percepts continuations naming one of each. Llama3-LLaVA-Next-8B needs a longer continuation budget on this stimulus before it names an animal (Table 2 ).
Figure 33: Raven–bear top-down single-letter prefix cues at the centre angle, across LLaVA-family models. First-token probabilities of the two percept tokens, as in the duck–rabbit top-down analyses.
Stimulus
1.5-7B
1.5-13B
v1.6-Vicuna
v1.6-Mistral
Llama3-NeXT
( n valid VA)
(44)
(42)
(40)
(40)
(41)
Object-count query: probability of the answer "1" / "2"
Harper
0.88 / 0.10
0.97 / 0.03
0.99 / 0.00
0.99 / 0.01
0.99 / 0.01
Raven–Bear
0.18 / 0.66
0.35 / 0.60
0.60 / 0.38
0.94 / 0.05
0.84 / 0.15
VA
0.88 / 0.09
0.89 / 0.10
0.99 / 0.01
0.96 / 0.03
0.96 / 0.04
Two-animal ctrl.
0.01 / 0.87
0.01 / 0.97
0.00 / 0.98
0.03 / 0.95
0.00 / 0.99
Appendix
Table 2: Object count and exclusivity across stimuli, for the five LLaVA-family models. Top block: first-token probability of the answers "1" and "2" to "How many objects are in the image? Answer with a number only." , read at the model’s own answer step (the remainder falls on other numbers). Bottom block: shares of beam mass whose continuation names a single vs. multiple animals under the default query and prefix (Appendix A.2.2 ; the remainder names no animal). The two blocks are separate measurements: an answer of "1" in the top block corresponds to exclusive reporting in the bottom one. Beam continuations use a 12-token budget, except Llama3-NeXT on Raven–Bear, which uses 24 tokens because at 12 tokens 67% of its beam mass had not yet named an animal. Harper and Raven–Bear at the original orientation; Visual Anagrams (VA) at each model’s boundary angle and the matched two-animal controls at 0∘ , means over the model’s valid stimuli ( n per column).
Figure 34: General description ( "Describe this image." ), averaged over the valid Visual Anagram stimuli, across all nine models. Each generation is classified by the animals it names (Appendix A.2.2 ): single animal , multiple animals or no animal , with bird only / rabbit only the subsets of single naming each percept and both percepts the subset of multiple; 2 samples per stimulus and angle, i.e. 74–88 generations per angle.
Figure 35: Zero-shot chain-of-thought, averaged over the valid Visual Anagram stimuli, across all nine models. Classes as in Figure 34 ; 2 samples per stimulus and angle, i.e. 74–88 generations per angle.
Figure 36: Beam-continuation classes on aggregated Visual Anagrams under the "... There are two animals." prompt (mean ± SD across stimuli).
Figure 37: Forced-choice prompts ( "Is it a bird or a rabbit?" / "Is it a rabbit or a bird?" , prefix "It is a" ) on aggregated Visual Anagrams across all nine models (mean ± SD across stimuli). Solid: bird named first; dashed: rabbit named first.
Figure 38: Prefix cues and their negations ( "It starts with ‘d’." versus "It doesn’t start with ‘d’." ) on aggregated Visual Anagrams across all nine models. Curves show the paired shift in the duck–rabbit preference relative to the no-cue run for the same stimulus (mean ± SD across stimuli).
Figure 39: Semantic cues and their negations ( "It’s avian." versus "It’s not avian." ), plotted as in Figure 38 .
Figure 40: Layerwise direct logit attribution for the two output tokens, across all LLaVA-family models.
Figure 41: Bottom-up modulation: difference in number of bird/rabbit dominant patches vs. final-token logit difference, across all LLaVA-family models.
Figure 42: Top-down resampling ablation results for resid_post across all LLaVA-family models, averaged over each model’s valid Visual Anagram stimuli.
Figure 43: Top-down resampling ablation results for attn_out across all LLaVA-family models, averaged over each model’s valid Visual Anagram stimuli.
Figure 44: Top-down resampling ablation results for mlp_out across all LLaVA-family models, averaged over each model’s valid Visual Anagram stimuli.
Figure 45: Query-patching results: image-attention contribution to top-down vs. bottom-up modulation, across LLaVA-family models. LLaVA-v1.6-Mistral-7B and Llama3-LLaVA-Next-8B are omitted (grouped-query attention; see text).
Figure 46: Exclusivity probe (object-count vs. continuation effect of single-token image-embedding patching) across all LLaVA-family models.
Wittgenstein's duck-rabbit poses a question for vision-language models: when a model captions an ambiguous image, where in the model is the commitment to one aspect made? We address this with a 3,320-generation behavioral baseline over 83 bistable stimuli that surfaces three regimes (default-dominant, force-dominant, force-balanced) under neutral vs forced-choice prompting, then probe the underlying representations using a TopK sparse autoencoder we train on the CLIP layer that LLaVA-1.6-7B actually consumes (validation EV 0.93). Across 69 bistable stimuli with both per-aspect feature pools available, 72% (50/69) show simultaneous activation of both pools at the vision tower, including 12/12 default-dominant duck/rabbit and 7/8 force-balanced young/old. Causal steering at CLIP layer 22 flips captions on default-dominant stimuli (33% rabbit-flip rate under a fluency guard) but cannot flip captions on force-balanced young/old at any tested coefficient, despite their vision-side superposition. The dominance bottleneck lives downstream of the vision tower; the gap between vision-side representation and language-side commitment is an empirical handle on the seeing/seeing-as distinction. We also flag a methodological note: rank-based statistics on TopK SAE outputs require tie-corrected ranking to avoid silent row-order bias.
Visual search is a fundamental cognitive ability. This study investigates whether Multimodal Large Language Models (MLLMs) exhibit human-like difficulty signatures in visual search tasks. We compared search performance of humans (n = 1,250) and MLLMs using identical 2D and 3D stimuli across different set sizes. Both groups showed efficient performance in feature searches, most clearly when the target had a unique color, but performance degradation in conjunction searches as set sizes increased. Additionally, we found strong correlations between human and MLLM error rates (ρ=0.82), which suggests that MLLMs are sensitive to similar objective complexities, such as stimulus heterogeneity. However, differences were found as well: whereas humans invested extra search time to respond accurately on target-absent trials, MLLMs exhibited extreme present/absent response biases in complex searches. We conclude that MLLMs replicate high-level human performance signatures, yet their underlying computations differ significantly.
Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou +2
Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands
Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.
Jiaang Li, Chengzu Li, Zhaochong An +4
University of Copenhagen · University of Cambridge · 3ETH Zürich +1