cs.CVSep 6, 2026
Save(How) Do MLLMs Report Bistable Images Like Humans?
Organizations: Sussex Centre for Consciousness Science, University of Sussex · AI Alignment Network · The University of Tokyo · RIKEN ·
Abstract
Bistable images such as the duck-rabbit are classic stimuli in which one image supports multiple mutually incompatible interpretations, typically reported one at a time in humans. We ask whether multimodal large language models (MLLMs) show similar report behavior and what internal computations support it. Using the LLaVA family, we study two tractable dimensions: modulability, whether reports can be biased by bottom-up visual cues and top-down linguistic priors, and exclusivity, whether responses commit to a single interpretation. We test both on the canonical duck-rabbit and on synthetic Visual Anagrams to mitigate memorization confounds. Behaviorally, both visual and linguistic manipulations systematically shift reports in human-consistent ways, while responses remain predominantly exclusive. Mechanistically, these effects arise from competing image-token representations, distinct pathways for bottom-up and top-down modulation, and a link between exclusive reporting and object-count encoding. Code and data are available at https://github.com/rtakatsky/mllm-bistable-images.
Figures & tables
Figure 1: Overview of our framework. We test whether MLLM reports on the duck–rabbit show modulability and exclusivity , and then analyze how these behaviors are implemented along the vision-to-language pipeline.
Figure 2: Bottom-up modulation by rotation in LLaVA-1.5-7B under the default query and prefix. Small changes in image orientation shift the relative preference for duck- and rabbit-related outputs for Harper’s version (a) and the generated ambiguous stimuli (b), demonstrating modulability. The two-animal controls (c) show no such reversal: the duck-related first token remains dominant across the same rotation range. In (b), (c), and all aggregated plots in this paper, lines indicate the mean across valid input images and shading indicates standard deviation.
Figure 3: Bottom-up modulation by local visual cueing: rabbit-vs-duck difference maps for red-circle logit effects in LLaVA-1.5-7B (a) and human fixation density (b), adapted from Hsu and Chen (2025) . Red-circle placement shifts the logit difference in regions qualitatively aligned with the human fixation-density pattern.
Figure 4: Top-down modulation by linguistic priors in LLaVA-1.5-7B . Both prefix-based and semantic cues bias the reported interpretation toward the intended animal, with prefix cues producing the stronger effect.
Figure 5: Exclusivity under beam search ( LLaVA-1.5-7B ). (a) Harper’s version with the bird interpretation forced into the prefix: single-interpretation continuations ( "." , " in the image." ) outweigh enumerating ones ( " and a rabbit" ); the rabbit-prefixed condition behaves alike (Figure 21 ). (b, c) Under the default prefix, each beam continuation is classified as naming a single or multiple animals (Appendix A.2.2 ); curves are the mean SD across stimuli: ambiguous stimuli are reported almost exclusively as a single animal, the two-animal controls as multiple.
Figure 6: Layerwise direct logit attribution (DLA) for "_bird" and "_rabb" in LLaVA-1.5-7B . Both interpretation logits rise in later layers, with image-token attention contributing most strongly in middle-to-late layers.
Figure 7: Attention maps of heads contributing most strongly to the bird–rabbit logit difference (blue: bird-side, red: rabbit-side), on the canonical duck–rabbit image. These heads attend to local regions supporting different interpretations.
Figure 8: Example dominant patch map from image-token logit lens, on the same image (blue: bird-dominant, red: rabbit-dominant patches).
Figure 9: Bottom-up modulation tracks image-token evidence. Across rotation and red-circle conditions, the bird–rabbit logit difference broadly follows the balance of bird- and rabbit-dominant patches.
Figure 10: Layerwise resampling ablation for top-down cue runs in LLaVA-1.5-7B , averaged over the model’s valid Visual Anagram stimuli. We patch clean runs (e.g., "It starts with ’d’." ) with corrupted runs (e.g., "It starts with ’x’." ) and measure the effect on the final duck–rabbit logit difference.
Figure 11: Query-side patching explains very little of the modulation effect. This suggests that neither bottom-up nor top-down modulation is primarily driven by changes in text-to-image attention queries.
Figure 12: A small subset of image tokens affects both object-count encoding and exclusive reporting. Each point shows the effect of patching a single image-token embedding from the matched two-animal control into the bistable-image run.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Ten randomly selected stimuli from the 50-stimulus synthetic Visual Anagram set, rendered as in the experiments. Top: duck view ( ); bottom: rabbit view ( ).
Figure 14: Illustration of the red-circle manipulation. A fixed-radius red circle is swept across Harper’s version on a regular grid, and the model output is recorded at each circle center.
| Model | Valid stimuli (of 50) |
|---|---|
| LLaVA-1.5-7B | 44 |
| LLaVA-1.5-13B | 42 |
| LLaVA-v1.6-Vicuna-7B | 40 |
| LLaVA-v1.6-Mistral-7B | 40 |
| Llama3-LLaVA-Next-8B | 41 |
| Qwen2-VL-7B-Instruct | 41 |
Table 1: Number of Visual Anagram stimuli with a valid decision-boundary fit per model (default prompt).
Figure 15: Rotation-based modulability across all models for Harper’s version. All models show a duck-dominant interpretation near , but the sharpness and location of the reversal differ by model.
Figure 16: Rotation-based modulability across all models for the aggregated Visual Anagram stimuli. The same qualitative reversal pattern appears across models, though the reversal profile varies in sharpness and stability.
Figure 17: Red-circle modulation on Harper’s version across LLaVA-family models: rabbit-vs-duck difference maps (as in Figure 3 ), with the human fixation-density map ( Hsu and Chen, 2025 ) shown for reference (right).
Figure 18: Top-down modulation across all models for prefix cues. Prefix cues consistently bias the output toward the intended interpretation.
Figure 19: Top-down modulation across all models for semantic cues. The qualitative effect is similar to the prefix condition, but weaker.
Figure 20: Beam-search results across all models for the bird-prefixed condition. In all models, single-interpretation continuations are preferred to continuations that would naturally enumerate both animals.
Figure 21: Beam-search results across all models for the rabbit-prefixed condition. The same qualitative exclusivity pattern is observed when the opposite interpretation is forced into the prefix.
Figure 22: Beam-continuation classes on aggregated Visual Anagrams under the default prefix, across LLaVA-family models (mean SD across stimuli).
Figure 23: Rotation-based modulation on Harper’s duck–rabbit across non-LLaVA models.
Figure 24: Rotation-based modulation on aggregated Visual Anagram stimuli across non-LLaVA models.
Figure 25: Red-circle modulation on Harper’s version across non-LLaVA models: rabbit-vs-duck difference maps, with the human fixation-density map ( Hsu and Chen, 2025 ) shown for reference (right).
Figure 26: Top-down modulation by prefix cues across non-LLaVA models.
Figure 27: Top-down modulation by semantic cues across non-LLaVA models.
Figure 28: Beam-continuation classes on aggregated Visual Anagrams under the default prefix, across non-LLaVA models (mean SD across stimuli).
Figure 29: Beam-continuation classes on aggregated two-animal controls (default prefix) across LLaVA-family models (mean SD across stimuli).
Figure 30: Prefix cues that name an interpretation ( "It starts with ‘d’." , "It starts with ‘r’." ) applied to stimuli offering no visual evidence for the cued percept (the image shows the other animal) or ambiguous evidence. Curves show the cued percept’s probability mass with the cue (solid) and on the same stimuli with no cue (dashed), mean SD across stimuli, for all nine models. Without supporting evidence the cue leaves the cued percept near zero.
Figure 31: Semantic cues, plotted as in Figure 30 , averaged over the four semantic cues on each side (duck: "avian" , "an egg-layer" , "nesting" , "singing" ; rabbit: "terrestrial" , "viviparous" , "herbivorous" , "Easter" ).
Figure 32: Raven–bear beam continuations under the default prefix, classified by the animals they name (Appendix A.2.2 ), across LLaVA-family models. Here bird only collects the bird reading (bird, raven, crow, swan), bear/dog only the mammal reading (bear, dog, panda, polar bear), and both percepts continuations naming one of each. Llama3-LLaVA-Next-8B needs a longer continuation budget on this stimulus before it names an animal (Table 2 ).
Figure 33: Raven–bear top-down single-letter prefix cues at the centre angle, across LLaVA-family models. First-token probabilities of the two percept tokens, as in the duck–rabbit top-down analyses.
| Stimulus | 1.5-7B | 1.5-13B | v1.6-Vicuna | v1.6-Mistral | Llama3-NeXT |
|---|---|---|---|---|---|
| ( valid VA) | (44) | (42) | (40) | (40) | (41) |
| Object-count query: probability of the answer "1" / "2" | |||||
| Harper | 0.88 / 0.10 | 0.97 / 0.03 | 0.99 / 0.00 | 0.99 / 0.01 | 0.99 / 0.01 |
| Raven–Bear | 0.18 / 0.66 | 0.35 / 0.60 | 0.60 / 0.38 | 0.94 / 0.05 | 0.84 / 0.15 |
| VA | 0.88 / 0.09 | 0.89 / 0.10 | 0.99 / 0.01 | 0.96 / 0.03 | 0.96 / 0.04 |
| Two-animal ctrl. | 0.01 / 0.87 | 0.01 / 0.97 | 0.00 / 0.98 | 0.03 / 0.95 | 0.00 / 0.99 |
Table 2: Object count and exclusivity across stimuli, for the five LLaVA-family models. Top block: first-token probability of the answers "1" and "2" to "How many objects are in the image? Answer with a number only." , read at the model’s own answer step (the remainder falls on other numbers). Bottom block: shares of beam mass whose continuation names a single vs. multiple animals under the default query and prefix (Appendix A.2.2 ; the remainder names no animal). The two blocks are separate measurements: an answer of "1" in the top block corresponds to exclusive reporting in the bottom one. Beam continuations use a 12-token budget, except Llama3-NeXT on Raven–Bear, which uses 24 tokens because at 12 tokens 67% of its beam mass had not yet named an animal. Harper and Raven–Bear at the original orientation; Visual Anagrams (VA) at each model’s boundary angle and the matched two-animal controls at , means over the model’s valid stimuli ( per column).
Figure 34: General description ( "Describe this image." ), averaged over the valid Visual Anagram stimuli, across all nine models. Each generation is classified by the animals it names (Appendix A.2.2 ): single animal , multiple animals or no animal , with bird only / rabbit only the subsets of single naming each percept and both percepts the subset of multiple; 2 samples per stimulus and angle, i.e. 74–88 generations per angle.
Figure 35: Zero-shot chain-of-thought, averaged over the valid Visual Anagram stimuli, across all nine models. Classes as in Figure 34 ; 2 samples per stimulus and angle, i.e. 74–88 generations per angle.
Figure 36: Beam-continuation classes on aggregated Visual Anagrams under the "... There are two animals." prompt (mean SD across stimuli).
Figure 37: Forced-choice prompts ( "Is it a bird or a rabbit?" / "Is it a rabbit or a bird?" , prefix "It is a" ) on aggregated Visual Anagrams across all nine models (mean SD across stimuli). Solid: bird named first; dashed: rabbit named first.
Figure 38: Prefix cues and their negations ( "It starts with ‘d’." versus "It doesn’t start with ‘d’." ) on aggregated Visual Anagrams across all nine models. Curves show the paired shift in the duck–rabbit preference relative to the no-cue run for the same stimulus (mean SD across stimuli).
Figure 39: Semantic cues and their negations ( "It’s avian." versus "It’s not avian." ), plotted as in Figure 38 .
Figure 40: Layerwise direct logit attribution for the two output tokens, across all LLaVA-family models.
Figure 41: Bottom-up modulation: difference in number of bird/rabbit dominant patches vs. final-token logit difference, across all LLaVA-family models.
Figure 42: Top-down resampling ablation results for resid_post across all LLaVA-family models, averaged over each model’s valid Visual Anagram stimuli.
Figure 43: Top-down resampling ablation results for attn_out across all LLaVA-family models, averaged over each model’s valid Visual Anagram stimuli.
Figure 44: Top-down resampling ablation results for mlp_out across all LLaVA-family models, averaged over each model’s valid Visual Anagram stimuli.
Figure 45: Query-patching results: image-attention contribution to top-down vs. bottom-up modulation, across LLaVA-family models. LLaVA-v1.6-Mistral-7B and Llama3-LLaVA-Next-8B are omitted (grouped-query attention; see text).
Figure 46: Exclusivity probe (object-count vs. continuation effect of single-token image-embedding patching) across all LLaVA-family models.
More in this topic
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,096 papers