Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements, particularly those represented by nodes and directed edges (e.g., arrows and lines). To investigate the underlying causes of this limitation, we probe the internal representation of LVLMs using a carefully constructed synthetic diagram dataset based on directed graphs. Our probing experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model. In contrast, node information and global structural features are already linearly encoded in individual hidden states of the vision encoder. These findings suggest that the stage at which linearly separable representations are formed varies depending on the type of visual information. In particular, the delayed emergence of edge representations may help explain why LVLMs struggle with relational understanding, such as interpreting edge directions, which require more abstract, compositionally integrated processes.
Figures & tables
Figure 1: Overview of this study. We analyze internal representations in LVLMs using probing on a synthetic diagram dataset. We find that node information (e.g., node color) and global information (e.g., node count) are linearly encoded in a single image patch within the vision encoder, whereas edge information (e.g., edge color) is linearly encoded in a single text token within the language model.
Table 1: The 3 categories, 11 aspects, and their corresponding label sets Y .
Figure 2: Examples of synthetic diagrams. Each diagram contains five nodes, and we control evaluation aspects such as node color, shape, and edge connectivity. We provide two variants: Drand , which uses random node layouts (left part), and Dfix , which uses fixed layouts (right part).
Node Color
Node Shape
In-degree Count
Out-degree Count
Edge Color
Edge Style
Edge Existence
Edge Direction
Multi-hop Path
Node Count
Edge Count
Chance level
12.5
20.0
20.0
20.0
12.5
50.0
50.0
50.0
50.0
20.0
20.0
Qwen3-VL 8B
91.4
76.6
40.3
34.7
57.3
73.5
69.6
49.3
58.3
40.3
21.6
Table 2: AccVQA for Qwen3-VL 8B. Accuracy exceeded chance level for all aspects except Edge Direction.
Figure 3: Layer-wise maximum accuracy in the vision encoder of Qwen3-VL 8B. The x-axis denotes the relative layer position (0 is the input layer and 1 is the final layer), and the y-axis denotes accuracy. Aspects sharing the same threshold are drawn with the same line style, which also matches the style of the corresponding black threshold lines.
Figure 4: Position-wise accuracy in the vision encoder of Qwen3-VL 8B. Each heatmap shows accuracy by patch position for a specific layer and aspect. The node layout of the evaluation diagrams is the same as that of the diagrams in the right part of Figure 2 .
Figure 5: MaxAccl per layer in the language model of Qwen3-VL 8B. The x-axis denotes the layer position (0 is the input layer and 1 is the final layer), and the y-axis denotes accuracy. Aspects sharing the same threshold are drawn with the same line style, which also matches the style of the corresponding black threshold lines.
Figure 6: accl,t(dfix) in the language model of Qwen3-VL 8B for a representative subset dfix . The node layout of the evaluation diagrams is the same as that of the diagrams in the right part of Figure 2 .
Figure 7: Probing results for text tokens in the language model of Qwen3-VL 8B. The x-axis shows token position in the question, and the y-axis shows layer position.
Figure 8: Overview of causal intervention. Hidden states of patches with probing accuracy exceeding the threshold are replaced with the mean of hidden states of other patches. If VQA accuracy decreases with this operation, it suggests that the replaced patch hidden states contributed to inference.
Node Color
Node Shape
In-degree Count
Out-degree Count
Edge Color
Edge Style
Edge Existence
Edge Direction
Multi-hop Path
Node Count
Clean run
91.4
76.6
40.3
34.7
57.3
73.5
69.6
49.3
58.3
40.3
Patched run
11.7
20.4
26.5
27.7
32.1
68.1
69.4
48.8
57.0
25.7
Controlled
83.4
73.6
36.8
34.3
53.8
72.3
68.6
50.3
57.1
31.8
Chance level
12.5
20.0
20.0
20.0
12.5
50.0
50.0
50.0
50.0
20.0
Δ (Clean run - Patched run)
79.7
56.2
13.8
7.0
25.2
5.4
0.2
0.5
1.3
14.6
Δ (Clean run - Controlled)
8.0
3.0
3.5
0.4
3.5
1.2
1.0
-1.0
1.2
8.5
Table 3: Results of causal intervention on Qwen3-VL 8B. Clean run shows VQA accuracy without intervention. Patched run shows accuracy when all patches exceeding the threshold in probing accuracy are replaced. Controlled refers to the controlled experiment setting described in section 6.1 .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Aspect
Question
Node Color
What color is node A?
Node Shape
What shape is node A?
In-degree
What is the in-degree of node A?
Out-degree
What is the out-degree of node A?
Edge Color
What color is the edge between node A and node B?
Edge Style
What style is the edge between node A and node B?
Appendix
Table 4: Questions used to evaluate each aspect.
Parameter
Image Part
Text Part
Epochs
1000
1000
Batch size *
{1000, 2000, 4000, 8000, 16000}
{10, 20, 40, 80, 160}
Learning rate *
[0.0001, 0.01] (log scale)
[0.0001, 0.01] (log scale)
Optimizer
AdamW
AdamW
betas
[0.9, 0.999]
[0.9, 0.999]
eps
1e-08
1e-08
Appendix
Table 5: Hyperparameters used for probe training. Batch size and learning rate were subject to hyperparameter search using Optuna. Optimizer parameters use PyTorch’s AdamW default values. * Subject to hyperparameter search
Figure 9: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 10: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 11: Probing results for text tokens in Qwen3-VL 8B. The x-axis shows token position in the question, and the y-axis shows layer position.
Figure 12: Example diagrams with layouts distinct from those in Figure 2 .
Figure 13: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B vision encoder (node A placed on the left side). Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 14: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B language model (node A placed on the left side). Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 15: Layer-wise maximum accuracy in the vision encoder of Qwen2.5-VL 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.
Figure 16: Accuracy for each hidden state in the image input part of the Qwen2.5-VL 7B vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 17: Layer-wise maximum accuracy in the language model of Qwen2.5-VL 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.
Figure 18: Accuracy for each hidden state in the image input part of the Qwen2.5-VL 7B language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 19: Probing results for text tokens in Qwen2.5-VL 7B. The x-axis shows token position in the question, and the y-axis shows layer position.
Figure 20: Layer-wise maximum accuracy in the vision encoder of LLaVA1.5 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.
Figure 21: Accuracy for each hidden state in the image input part of the LLaVA1.5 7B vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 22: Layer-wise maximum accuracy in the language model of LLaVA1.5 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.
Figure 23: Accuracy for each hidden state in the image input part of the LLaVA1.5 7B language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 24: Probing results for text tokens in LLaVA1.5 7B. The x-axis shows token position in the question, and the y-axis shows layer position.
Figure 25: Layer-wise maximum accuracy in the vision encoder of Gemma3-4B-IT. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.
Figure 26: Accuracy for each hidden state in the image input part of the Gemma3-4B-IT vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 27: Layer-wise maximum accuracy in the language model of Gemma3-4B-IT. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.
Figure 28: Accuracy for each hidden state in the image input part of the Gemma3-4B-IT language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.
Figure 29: Probing results for text tokens in Gemma3-4B-IT. The x-axis shows token position in the question, and the y-axis shows layer position.
Node Color
Node Shape
In-degree Count
Out-degree Count
Edge Color
Edge Style
Edge Existence
Edge Direction
Multi-hop Path
Node Count
Edge Count
Clean run
91.9
78.0
37.0
28.8
54.9
71.0
50.0
46.0
65.0
83.6
22.2
Patched run
11.9
20.4
25.0
21.8
40.9
73.0
50.5
47.0
51.0
17.2
N/A
Controlled
93.4
78.0
37.4
26.8
56.6
69.0
50.0
45.0
69.5
21.0
N/A
Chance level
12.5
20.0
20.0
20.0
12.5
50.0
50.0
50.0
50.0
20.0
20.0
Patched ratio
28.5
22.3
37.9
31.6
16.0
8.20
3.50
1.56
28.1
93.0
100
Appendix
Table 6: Results of the causal intervention for Qwen2.5-VL 7B. “Patched” reports the results when all hidden states whose probing accuracy exceeded the threshold were replaced. “Controlled” reports the results when the same number of patches as those whose probing accuracy exceeded the threshold were randomly selected and replaced.
Node Color
Node Shape
In-degree Count
Out-degree Count
Edge Color
Edge Style
Edge Existence
Edge Direction
Multi-hop Path
Node Count
Edge Count
Clean run
40.5
39.6
34.0
30.8
38.1
49.5
60.5
50.0
50.0
20.0
27.2
Patched run
49.6
28.8
71.2
52.8
16.0
50.5
59.5
50.0
50.0
21.8
N/A
Controlled
9.20
19.6
20.0
22.4
13.0
50.0
59.0
50.0
50.0
21.2
N/A
Chance level
12.5
20.0
20.0
20.0
12.5
50.0
50.0
50.0
50.0
20.0
20.0
Patched ratio
89.8
83.5
94.4
93.9
75.3
11.1
0.174
0
65.3
99.8
100
Appendix
Table 7: Causal intervention results for LLaVA1.5 7B. “Patched run” denotes the results when all patches whose probing accuracy exceeds the threshold are replaced. “Controlled” denotes the results when the same number of patches as those exceeding the threshold are randomly selected and replaced.
Node Color
Node Shape
In-degree Count
Out-degree Count
Edge Color
Edge Style
Edge Existence
Edge Direction
Multi-hop Path
Node Count
Edge Count
Clean run
69.2
50.0
26.4
30.0
41.2
57.2
56.6
50.3
50.0
33.9
20.1
Patched run
35.8
33.6
25.1
27.0
41.0
56.9
57.0
50.1
50.0
9.40
20.5
Controlled
68.2
48.3
25.7
29.8
40.8
57.2
56.9
49.9
50.0
23.9
20.6
Chance level
12.5
20.0
20.0
20.0
12.5
50.0
50.0
50.0
50.0
20.0
20.0
Patched ratio
7.55
5.44
7.06
6.28
0.708
1.34
0.457
0.425
11.0
72.3
82.4
Appendix
Table 8: Causal intervention results for Gemma3-4B-IT. “Patched run” reports the results when all hidden states whose probing accuracy exceeded the threshold are replaced. “Controlled” reports the results when the same number of hidden states as those exceeding the probing-accuracy threshold is randomly selected and replaced.
Vision-Language Models (VLMs) achieve strong performance in visual reasoning tasks, but it remains unclear whether they understand visual relations, or simply employ shortcuts such as language cues or priors. To investigate this, we use the Qwen3-VL-4B (Bai et al., 2025), a modern VLM, to decode how visual information is encoded across depths. For this, we propose a synthetic dataset of simple geometric shapes for controlled analysis, along with queries crafted to precisely test language cues. Furthermore, the dataset is modified to test causal reliance on visual evidence. Our results show that current VLMs combine genuine visual reasoning with shortcut strategies primarily rooted in language cues.
Vision Language Models (VLMs) are increasingly applied to software engineering artifacts, especially UML class diagrams whose meaning depends on visual notation. Yet, it is unclear whether VLMs actually read such diagrams or instead answer from pretrained priors about how classes typically relate. We introduce a controlled UML benchmark in which each prior-conforming diagram is paired with its prior-conflicting counterpart that (1) preserves the same class names and layout while (2) reverses only the relation arrow. We evaluate eight open-source VLMs from two model families, InternVL3.5 and Qwen3, alongside two closed-source frontier models GPT-5.4 and GPT-5.4 Mini. Across the eight open-source models, reversing the arrow reduces relation-direction accuracy by 33.48% on average, while GPT-5.4 Mini retains a 10% gap. In the harder three-class condition, accuracy drops sharply by 45.28% for open-source models, and even 18.62% for the GPT-5.4 family on average. Scaling provides only limited improvements and is family-dependent. Our benchmark presents a diagnostic prior-driven failure in diagram-grounded software understanding. Our artifact is available at https://anonymous.4open.science/r/UMLKnowledgeConflict-8461.
Zaiyu Cheng, Khai-Nguyen Nguyen, Antonio Mastropaolo
Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning over structured layouts. Although recent LVLMs have shown progress on VDU benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.