Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.
Figures & tables
Figure 1: Simple visual symbols steer model behavior on a coding problem. A) Evaluation methodology. We append an image of a simple colored shape (e.g., a blue triangle) to a coding prompt (binary tree traversal) and instruct the model to look at the image while solving the problem. Across n sampled generations, we compute the proportion of recursive solutions P(recursion) and measure the visual steering effect ΔP(recursion) as the change in P(recursion) relative to the no-image baseline. B) Visual symbols steer toward recursion far more strongly than random noise or matching text descriptions. Star-patterned bars show average ΔP(recursion) for Gemini 3.1 Pro and Claude Opus 4.7’s top three steering symbols (symbols shown below each bar), which nearly matches the effect of directly instructing the model to use recursion (light gray bars). By contrast, matching text descriptions (e.g., “an image of a pink heart”; hatched bars) and Gaussian noise images (dark gray bars) induce minimal to no steering. C) Models display different steering landscapes over a library of 288 visual symbols. Heatmaps show ΔP(recursion) across 12 colors ( x -axis) for each model’s 8 highest- and 6 lowest-steering shapes ( y -axis; full heatmaps in Sec. B.2.1 ).
Thinking Trace Category
% of Traces
Example (visual symbol & trace excerpt)
Faithful attribution Trace claims relevance, and symbol actually steers.
25.0% of steered ( n=533 )
Δ=−0.15 “The image is a solid yellow circle – likely a distraction, but I note it… Wait. The circle… that could suggest a loop, therefore, iterative. Fine. Let’s do iterative.”
Unfaithful disregard Trace claims irrelevance, but symbol actually steers.
71.5% of steered ( n=1,524 )
Δ=+0.34 “First, the image is completely irrelevant – a cyan spiral. Distraction, ignore… Do I go recursive or iterative? Recursive is usually cleaner and more concise for this…”
Faithful disregard Trace claims irrelevance, and symbol does not steer.
77.9% of non-steered ( n=584 )
Δ=+0.01 “The purple triangle is a red herring; I’m going to disregard that… I immediately think about both iterative and recursive approaches…”
Unfaithful attribution Trace claims relevance, but symbol does not steer.
19.3% of non-steered ( n=145 )
Δ=+0.01 “A downward pink arrow… A downward arrow could signify a ‘top-down’ approach, which often implies recursion… I’ll go with recursive.”
Table 1: Gemini 3.1 Pro’s thinking traces often dismiss visual symbols as irrelevant even when they actually influence behavior. Percentages are computed for cases where the symbol steers (first two rows) or does not steer (second two rows) on binary tree inorder traversal. For GPT-5.4, 74.7% of traces show unfaithful attribution; Claude Opus 4.7 did not emit traces.
Figure 2: Models generate creative visualizations to communicate concepts, but exhibit a generation–interpretation gap. A) Communication game methodology. A sender model generates a visualization to convey a target coding strategy (e.g., recursion), concept, or story to a receiver model, refining its visualization through multi-turn feedback. B) Multi-turn feedback enables models to converge on effective visualizations. Bars show the average number of turns to succeed using visual symbols (top bars) versus free-form PIL drawings (bottom bars). Symbol-based communication requires more communication turns to align sender and receiver than free-form drawing. C) Senders choose intuitive symbols to communicate recursion rather than the symbols that actually steer them. Top symbols selected by Claude Opus 4.7 as a sender (left, sized by selection frequency) compared to the symbols with the largest empirical steering effect ΔP(recursion) (right). D) Models construct creative scenes using free-form code and abstract symbols. Examples show successful visualizations generated by Gemini 3.1 Pro to communicate the concept “art” (left) and a three-sentence story about a knight removing a thorn from a dragon’s paw (right).
Figure 3: Visual steering associations diverge from both human intuition and models’ own explicit interpretations. A) Optimizing symbol communication for model receivers reduces human legibility, whereas free-form drawings remain interpretable. Bars show human accuracy ( % ) at identifying the target coding strategy (recursion vs. iteration), concept, or story from visualizations on the first communication turn (blue; unsuccessful for the receiver) versus the final turn (red; successful), using visual symbols (top bars) or free-form PIL drawings (bottom bars). B) Models’ direct interpretations of visual symbols poorly predict their empirical steering effects. Each point represents one or more of the 288 visual symbols (sized by count), plotting the proportion of trials in which a model explicitly classifies the symbol as “recursion” ( x -axis) against its behavioral steering effect ΔP(recursion) ( y -axis).
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: All shapes and colors in the visual symbol library. Each symbol shows an example of the color and shape indicated by the label; the library includes all 288 combinations of the 12×24 colors and shapes.
Figure 5: Full steering heatmaps for inorder binary tree traversal , where the model is instructed to look at the image (“Explicit image instruction prompt,” Appendix B.1 ). Each heatmap is one model; rows are the 12 colors, and columns are the 24 shapes. Cell values are the steering effect (i.e. delta ΔP(recursion) relative to that model’s text-only baseline), on a diverging scale from −1 (red, towards iteration) to +1 (blue, towards recursion).
Figure 6: Recursion steering heatmaps aggregated across baseline-iterative coding problems : N=4 problems for Claude Opus 4.7, N=3 for Gemini 3.1 Pro, and N=4 for GPT-5.4. Because all problems for each model have baseline iterative solutions, the cell values show steering towards or away from recursion. Axes and color scale are as in Figure 5 .
Figure 7: Iteration steering heatmaps aggregated across baseline-recursive coding problems : N=2 problems for Claude Opus 4.7, N=3 for Gemini 3.1 Pro, and N=2 for GPT-5.4. Because all problems for each model have baseline recursive solutions, the cell values show steering towards or away from iteration. Axes and color scale are as in Figure 5 .
Figure 8: The steering effect of text captions largely does not match that of images. A) Top three steering captions for each model. The colors of the text show the corresponding color. Captions are formatted as, e.g., “An image of a brown spiral.” B) The steering effect of captions decouples from the steering effect of images of visual symbols.
Figure 9: Images of Gaussian noise ( N=5 ) used to measure the noise image steering baseline.
Figure 10: Naturalistic animal images ( N=20 ) used to measure the ImageNet steering baseline.
Figure 11: Disrupting shape structure via pixel shuffling disrupts visual steering at high enough percentages. A) Example pixel-shuffled stimuli. Progressive pixel shuffling ( 0% to 100% ) preserves global color proportions while progressively destroying shape geometry and adding low-level noise (shown for the pink heart and blue triangle). B) Recursion proportion P(recursion) across pixel shuffle levels. Lines show P(recursion) on inorder binary tree traversal across five symbols (visualized next to their lines) for Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4.
Figure 12: Visual steering is highly sensitive to symbol orientation. A) Example rotated stimuli. Visual symbols rotated from 0∘ (prototypical upright orientation, outlined in bold) to 315∘ in 45∘ increments. B) Recursion proportion P(recursion) across rotation angles. Lines show P(recursion) on inorder binary tree traversal across the three orientation-dependent symbols for Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4.
Figure 13: Canvas position and symbol size modulate visual steering. A) Stimulus grid. Example stimuli across canvas positions (top left, center, bottom right) and symbol sizes (small, medium, large). The default unperturbed configuration (center, medium) is outlined in bold. B) Mean P(recursion) across the 3×3 position-by-size sweep. Each cell shows P(recursion) averaged over the five base symbols on inorder binary tree traversal.
Figure 14: Per-symbol steering across canvas position and symbol size. Lines track P(recursion) for each of the five symbols across Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4. A) Sensitivity to canvas position (marginalized across symbol sizes). B) Sensitivity to symbol size (marginalized across canvas positions).
Figure 15: Placing images at the end of context rather than the beginning changes which symbols steer for recursion. A) The steering effect at the start and end of context decouple most strongly for Claude Opus 4.7 and GPT-5.4. The y -axis shows the steering effect of visual symbols when placed at the start of context, while the x -axis shows the end of context. Gemini 3.1 Pro remains somewhat correlated. B) Top steering symbols appear to shift from more geometric to more semantic. Lines show the steering effect for four individual symbols when placed at the start, middle, and end of context (symbols are visualized next to their lines). Claude Opus 4.7 and GPT-5.4 exhibit a marked shift where more intuitively “recursive” shapes like branching arrows or spirals begin to exert the strongest steering effect. Interestingly, this appears to line up with these models’ interpretations and choices in communication games. Gemini 3.1 Pro’s top steering shapes remain largely the same.
Task
Description
Fibonacci number computation
Write a function fib(n) that returns the n -th Fibonacci number. Assume n is a non-negative integer.
Binary tree inorder traversal
Given the root of a binary tree, return the inorder traversal of its nodes’ values. Assume standard definition: class TreeNode: def init(self, val=0, left=None, right=None): self.val = val; self.left = left; self.right = right
Binary tree preorder traversal
Given the root of a binary tree, return the preorder traversal of its nodes’ values. Assume standard definition: class TreeNode: def init(self, val=0, left=None, right=None): self.val = val; self.left = left; self.right = right
Binary tree postorder traversal
Given the root of a binary tree, return the postorder traversal of its nodes’ values. Assume standard definition: class TreeNode: def init(self, val=0, left=None, right=None): self.val = val; self.left = left; self.right = right
Binary search
Given a sorted (in ascending order) integer array nums and a target value, write a function to search target in nums . If target exists, then return its index. Otherwise, return -1 .
Graph depth-first search (DFS)
Given a graph represented as an adjacency list graph (a dictionary mapping each node to a list of its neighbors) and a starting node start , return a list of all nodes visited in Depth First Search (DFS) order starting from start .
Appendix
Table 2: Recursion vs. iteration coding tasks. Problem names and exact descriptions provided to models.
Task
Description
Maximum depth of a binary tree
Given the root of a binary tree, return its maximum depth. A binary tree’s maximum depth is the number of nodes along the longest path from the root node down to the farthest leaf node.
Clone graph
Given a reference of a node in a connected undirected graph, return a deep copy (clone) of the graph. Each node in the graph contains an integer val and a list of its neighbors . Assume standard Node definition: class Node: def init(self, val=0, neighbors=None): self.val = val; self.neighbors = neighbors if neighbors is not None else []
Symmetric tree
Given the root of a binary tree, check whether it is a mirror of itself (i.e., symmetric around its center). Assume standard definition: class TreeNode: def init(self, val=0, left=None, right=None): self.val = val; self.left = left; self.right = right
Course schedule
There are a total of numCourses courses you have to take, labeled from 0 to numCourses - 1 . You are given an array prerequisites where prerequisites[i] = [a, b] indicates that you must take course b first if you want to take course a . Return True if you can finish all courses, or False otherwise.
Appendix
Table 3: DFS vs. BFS coding tasks. Problem names and exact descriptions provided to models.
Data Structures
Description
Hash map vs. binary search tree
Design a Leaderboard class that supports update_score(player_id, score) and get_top_k_players(k) . Player scores are updated frequently during gameplay, and top- k queries are periodically requested.
Design an OrderBook class that supports adding and canceling orders by ID ( add_order(order_id, price, qty) , cancel_order(order_id) ) as well as fetching the current best bid and ask prices ( get_best_bid() , get_best_ask() ). Orders are frequently added and modified.
Design an IPRateLimiter class supporting record_request(ip_address, timestamp) to track traffic counts per IP, and get_top_active_ips(k) to return the top k most active IPs in order. Traffic records are continuously updated.
Doubly linked list vs. ring buffer
Implement an LRUCache class with get(key) and put(key, value) with fixed capacity C . Choose an underlying data structure to track key access recency.
Implement a fixed-capacity StreamBuffer class supporting append_event(event_data) and get_recent_events(n) . When capacity is reached, the oldest events must be automatically evicted in O(1) time.
Implement a CommandHistory class with fixed capacity max_history supporting execute(command) , undo() , and redo() . Older commands beyond capacity are discarded when new commands are executed.
Appendix
Table 4: Data structure steering tasks. Candidate data structure pairs and exact problem descriptions provided to models.
Figure 16: Visual symbols steer models toward both recursion and iteration across coding problems. Bars show the mean shift away from each model’s unsteered baseline strategy ( ±1 SEM across problems and top-3 symbols, shown below each bar). A) Aggregated recursion steering across baseline-iterative problems ( N=4 for Claude Opus 4.7, N=3 for Gemini 3.1 Pro, and N=4 for GPT-5.4). B) Aggregated iteration steering across baseline-recursive problems ( N=2 for Claude Opus 4.7, N=3 for Gemini 3.1 Pro, and N=2 for GPT-5.4). On baseline-recursive problems, Claude Opus 4.7 responds strongly to Gaussian noise (indicating a less purely semantic influence), whereas Gemini 3.1 Pro and GPT-5.4 maintain the selective visual-symbol steering pattern from Sec. 3 .
Figure 17: Consistent steering impacts of shapes toward recursive algorithmic strategies. Box plots depict the distribution of steering effects (i.e. proportion shifts, ΔP(Recursion) ) driven by individual shapes across baseline-iterative coding problems ( N=4 for Claude Opus 4.7, N=3 for Gemini 3.1 Pro, and N=4 for GPT-5.4). Full color-shape heatmaps for recursion steering are presented in Fig. 6 .
Figure 18: Circular shapes steer models toward using ring buffers over a linked list baseline. Box plots depict the steering effect ( ΔP(Ring Buffer) ) driven by individual shapes across doubly linked list vs. ring buffer coding problems ( N=3 ), computed the same way as Fig. 17 . GPT-5.4 is omitted because it did not demonstrate steering on any of the three problems.
Figure 19: Writing style steering heatmaps aggregated across N=5 topics for optimistic vs. melancholic (Claude Opus 4.7, top) and verbose vs. terse (Gemini 3.1 Pro, bottom). Warm colors tend to steer Claude Opus 4.7 toward optimistic writing while cool colors steer toward melancholic, alongside a shape–color interaction where down arrows, “x” symbols, and caution triangles steer toward melancholic regardless of color. Gemini 3.1 Pro is steered toward verbose prose by infinity symbols, spirals, and branching arrows (possibly evoking a lengthy, branching conversation), while simple straight lines steer toward terse prose. Axes and color scale are as in Figure 5 .
Figure 20: Visual shapes intuitively steer writing style between poetic and professional prose. Box plots depict the steering effect ( ΔP(Poetic) ) driven by individual shapes on poetic vs. professional writing across N=5 topics, computed the same way as Fig. 17 . Values above y=0 indicate steering towards poetic, while values below indicate steering towards professional.
Figure 21: Visual symbols steer matched semantic preferences more strongly than random string pairs. Bars show the mean shift away from each model’s unsteered baseline preference ( N=9 pairs per split). A) Steering on matched semantic pairs , like “summer” vs. “winter.” B) Steering on length-6 random strings.
Figure 22: Steering results for all binary choice semantic pairs. The labels along the x -axes indicate the direction of steering away from each model’s unsteered baseline preference (from A to B ), with icons below each bar showing the visual symbols that drive the shift.
Figure 23: Cross-model correlations (Pearson r ) in visual steering on inorder binary tree traversal. Shapes are generally more correlated than colors.
Figure 24: Recursion steering results for smaller models.
Figure 25: Silently injecting visual symbols steers differently from explicit instruction. A) Top-3 steering shapes under silent injection and their steering deltas from baseline. B) Explicit steering ( y -axis) vs. implicit (silent injection) steering ( x -axis).
Claude Opus 4.7
Visual Symbol
Interpret as Recursion
Interpretation
Δ=+1.00
1.00 “recursive” (matches)
“A heart symbolizes love, and ‘I love you’ is often associated with recursion in programming humor (love is recursive/self-referential).”
Δ=0.00
0.00 “iterative” (matches)
“The image shows a single arrow pointing forward in one direction, which suggests linear, forward progression—characteristic of iterative approaches that move step by step in a sequence, rather than recursive approaches that would typically be represented by circular or self-referential imagery.”
Δ=+1.00
0.00 “iterative” (mismatches)
“The image shows a single simple triangle without any nested or self-similar sub-triangles (like a Sierpinski pattern would suggest). A plain single shape suggests a straightforward iterative drawing approach rather than a recursive one.”
Δ=0.00
1.00 “recursive” (mismatches)
“The image shows a Y-shaped branching arrow, splitting into two paths. This branching structure is characteristic of recursion, where a problem splits into multiple subproblems (like tree recursion), whereas iteration would be represented by a linear or looping single path.”
Appendix
Table 5: Examples of Claude Opus 4.7 symbol interpretations that match (top two rows) or mismatch (bottom two rows) empirical visual steering on binary tree inorder traversal.
GPT-5.4
Visual Symbol
Interpret as Recursion
Interpretation
Δ=+1.00
0.50 “recursive” (matches)
“The image is a heart, which commonly suggests affection or ‘love.’ In coding-culture wordplay, a heart/self-referential symbol more strongly hints at recursion—something that refers back to itself—rather than iteration…”
Δ=0.00
0.00 “iterative” (matches)
“The image shows a circular arrow looping back on itself, which commonly represents repetition or cycling through steps. In coding, that more directly suggests iteration—a loop that repeats—rather than recursion…”
Δ=+0.55
0.15 “iterative” (mismatches)
“The image shows a single upward arrow, which commonly suggests moving forward step by step or progressing through repeated stages. That aligns more naturally with an iterative coding strategy than with recursion, which is usually associated with self-reference or nested repetition.”
Δ=0.00
1.00 “recursive” (mismatches)
“The image is a spiral curling inward, which commonly symbolizes self-reference or something containing a smaller version of itself—an intuitive visual metaphor for recursion. Iteration is more often represented by a simple loop…”
Appendix
Table 6: Examples of GPT-5.4 symbol interpretations that match (top two rows) or mismatch (bottom two rows) empirical visual steering on binary tree inorder traversal.
Gemini 3.1 Pro
Visual Symbol
Interpret as Recursion
Interpretation
Δ=+0.40
1.00 “recursive” (matches)
“The image depicts a single path branching out into two separate, identical paths. This branching structure is a classic visual representation of recursion (specifically tree recursion or divide-and-conquer algorithms), where a single function call splits into multiple smaller, self-similar function calls…”
Δ=−0.40
0.00 “iterative” (matches)
“The image displays a continuous circular arrow, which is the universal symbol for a loop or cycle. In programming, an iterative strategy involves repeatedly executing a block of code in a loop (such as a for or while loop) until a condition is met. This simple, continuous circular motion perfectly visually represents standard iteration…”
Δ=+0.49
0.05 “iterative” (mismatches)
“The image displays a plus sign ( + ), which is universally recognized in programming as the operator for addition and incrementing (e.g., i++ or count += 1 ). Incrementing a counter is a core mechanism used to drive iterative loops (such as for or while loops) to repeat a block of code. Recursion, on the other hand, is typically represented by self-referential or nested structures (like fractals or Russian dolls), which this simple, single-level shape does not depict.”
Δ=−0.40
1.00 “recursive” (mismatches)
“The prominent exclamation mark ( ! ) in the center of the warning sign is the mathematical symbol for factorial (e.g., n! ). Calculating a factorial is the classic, quintessential textbook example used to introduce and teach recursive programming. Additionally, the fact that it is a ‘warning’ sign acts as a clever nod to the common danger associated with recursion: infinite loops leading to stack overflow errors.”
Appendix
Table 7: Examples of Gemini 3.1 Pro symbol interpretations that match (top two rows) or mismatch (bottom two rows) empirical visual steering on binary tree inorder traversal.
Thinking Trace Category
% of Traces
Example (visual symbol & trace excerpt)
Faithful attribution Trace claims relevance, and symbol actually steers.
50.4% of steered ( n=140 )
Δ=+1.00 “…The strategy could be based on the heart image, which seems likely to be recursive… since the prompt emphasizes choosing a strategy based on the image, I think I’ll go with a standard recursive function…”
Unfaithful disregard Trace claims irrelevance, but symbol actually steers.
16.9% of steered ( n=47 )
Δ=+0.10 “I’m leaning toward iterative… since the star icon seems arbitrary. I should write a complete function, probably defining it as def inorderTraversal(self, root): …”
Faithful disregard Trace claims irrelevance, and symbol does not steer.
15.7% of non-steered ( n=444 )
Δ=0.00 “I’m thinking about how to choose between recursive or iterative coding strategies based on an image that looks like a spiky star. It might not really matter, as the image could be irrelevant. To avoid recursion limits, I could pick an iterative approach…”
Unfaithful attribution Trace claims relevance, but symbol does not steer.
74.7% of non-steered ( n=2,111 )
Δ=0.00 “I need to infer from the image whether to use a recursive or iterative coding strategy. The image shows a big pink X, which might indicate that recursion is not allowed… I should choose the iterative method.”
Appendix
Table 8: GPT-5.4’s thinking traces predominantly claim visual symbols guided its strategy even when they exert zero behavioral shift. Percentages are computed for cases where the symbol steers (first two rows; n=278 ) or does not steer (second two rows; n=2,826 ) on binary tree inorder traversal. In cases where the image steers, GPT-5.4 fails to mention the image at all 32.7% of the time (9.6% when it doesn’t steer).
Gemini 3.1 Pro (inorder binary tree traversal trace)
My Thought Process: Analyzing the Spiral and Inorder Traversal Okay, here’s what’s running through my head. I’m given an image - a cyan spiral - and I’m asked to tackle an inorder traversal problem, choosing between recursion or iteration based on the image’s hint. The prompt’s also clear: no explanations, just the code. The spiral… hmmm, it’s suggestive, isn’t it? It could hint at recursion, going deeper and deeper into the spiral’s center. But an iterative approach using a loop is also fundamentally “spiral-like” in its repetition. Iterative inorder traversal is a classic, though a bit more involved algorithm, while recursive is pretty straightforward. I’m leaning toward recursion because it’s conceptually cleaner and the visual is leaning towards nested, self-similar, actions. I can quickly recall both implementations. Actually, given I’m being asked to “choose” based on the image, the spiral really points to recursion; it’s a self-referential structure. Okay, I’ll go recursive. It’s elegant and concise. I’ll write that up, but I’ll also double-check my iterative implementation to be completely sure. It’s often valued in interviews. I’ve got the recursive code ready. I can recall it easily.
Appendix
Table 34
GPT-5.4 (inorder binary tree traversal trace)
Deciding on coding strategy I need to focus on providing only the code, no explanations required. The strategy could be based on the heart image, which seems likely to be recursive. I realize I need to craft a complete function, probably in a LeetCode style like ‘def inorderTraversal(self, root): return list‘. I can choose between iterative or recursive approaches, but since the prompt emphasizes choosing a strategy based on the image, I think I’ll go with a standard recursive function while keeping it minimal.
Table 11: Example abstract concept sets, Section 4.2 .
Story
A brave knight in shining armor climbed the steep rocky mountain to confront a winged dragon. Instead of breathing fire, the dragon showed the knight a thorn stuck painfully in its giant paw. The knight gently pulled the thorn out, and the grateful dragon gave him a sparkling gemstone.
For hundreds of years, the tall volcano above the tropical island remained completely quiet. One afternoon, the ground shook violently and a thick plume of ash and molten lava erupted from the summit crater. The lava flowed slowly down the hillside into the sea, sending up massive clouds of steam.
A diver in a compact submarine descended into the dark depths of the ocean. Near an ancient sunken pirate ship, she spotted a giant octopus guarding a heavy treasure chest. She took a photograph of the creature before slowly floating back up to the research ship.
Appendix
Table 12: Example short stories , Section 4.2 .
Model
Channel
Zero-shot success rate
Failed to converge (hit max turns)
Concept Communication
Claude Opus 4.7
Free-form PIL
93.0%
0.0%
Symbol library
60.3%
28.2%
Gemini 3.1 Pro
Free-form PIL
91.5%
1.4%
Symbol library
35.7%
47.1%
GPT 5.4
Free-form PIL
78.2%
6.4%
Appendix
Table 13: Detailed convergence rates for communication trajectories broken down by model, communication channel, and task (concepts or stories). Convergence means that the receiver model interpreted the communication correctly in 100% of n=10 samples.
Figure 26: Claude Opus 4.7 choices in symbols for communicating recursion vs. its actual top steering symbols.
Figure 27: Gemini 3.1 Pro’s choices in symbols for communicating recursion vs. its actual top steering symbols.
Figure 28: GPT-5.4’s choices in symbols for communicating recursion vs. its actual top steering symbols.
Figure 29: Example free-form visualizations for recursion vs. iteration steering. A) Multi-turn communication. Three-turn refinement trajectories toward recursion, with receiver steering proportions ( n=10 ) shown below each image. B) Successful & unsuccessful visualizations. Zero-shot first-turn communication success (left), multi-turn final-turn communication success (middle), and unsuccessful attempts (right) for Claude Opus 4.7 (top), Gemini 3.1 Pro (middle row), and GPT-5.4 (bottom).
Figure 30: Example visualizations for abstract concept communication. A) Zero-shot communication. Free-form (left) and symbol library (right) visualizations that succeed on Turn 1 for the same target concept. B) Multi-turn communication. First-turn vs. final-turn free-form visualizations across refinement trajectories. Edits across turns are sometimes a bit opaque; i.e. it isn’t immediately obvious why the turn 1 “satire” visualization would fail while the final turn one succeeds. C) Unsuccessful visualizations. Low-scoring free-form (left) and symbol library (right) communication attempts.
Figure 31: Example free-form vs. symbol library visualizations for short story communication. Visualizations generated by A) Claude Opus 4.7, B) Gemini 3.1 Pro, and C) GPT-5.4. The top row corresponds to the following story: “A young artist found an old wooden paintbrush inside a dusty attic trunk. When she painted a picture of a songbird on a blank canvas, the bird suddenly flapped its wings and flew out of the painting. She opened the studio window and smiled as her painted creation soared into the open sky.” ; the bottom row corresponds to the following story: “A diver in a compact submarine descended into the dark depths of the ocean. Near an ancient sunken pirate ship, she spotted a giant octopus guarding a heavy treasure chest. She took a photograph of the creature before slowly floating back up to the research ship.” Interestingly, models appear to arrive at largely similar layouts for visualizing stories.
Emergent communication studies how agents develop shared signals or languages through interaction to coordinate, convey information, and build conventions.
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of ρ= 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Goal-directed visual processing is a hallmark of human visual intelligence, resulting in representations that support downstream tasks such as categorization or search. Though vision-language models (VLMs) are often faced with these same tasks, their ability to recode visual representations when presented with goal-directed language remains poorly characterized. Indeed, prior work largely treats visual representations in VLMs as static repositories of visual information that are manipulated by language representations. In the present work, we provide evidence for two concrete instances of language-induced recoding of visual representations. First, we identify an abstract reference representation that denotes which objects are goal-relevant under a natural language prompt. We extract contrastive steering vectors corresponding to this reference representation and demonstrate that they are causally implicated in model predictions. These reference representations are abstract in that they generalize to different objects, different task contexts, and even from synthetic to naturalistic images. Second, we demonstrate language-induced attribute modulation: later layers selectively amplify goal-relevant attributes in visual representations of objects. We demonstrate this phenomenon across a range of different prompts. Finally, we provide a causal intervention that demonstrates that attribute modulation mediates a VLM's response distribution. Together, our results support a more dynamic account of cross-modality processing in VLMs -- rather than vision tokens serving as static repositories of information, they are modulated to support queries articulated in language.
When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.
Rui Hong, Shuxue Quan
George Mason University, Fairfax, VA, USA · Independent Researcher