Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models

Haruto Yoshida; Keito Kudo; Yoichi Aoki; Ryota Tanaka; Itsumi Saito; Keisuke Sakaguchi; Kentaro Inui

cs.CLMar 3, 2026

Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models

Authors: Haruto Yoshida, Keito Kudo, Yoichi Aoki, Ryota Tanaka, Itsumi Saito, Keisuke Sakaguchi, Kentaro Inui

Organizations: Tohoku University · Human Informatics Labs., NTT, Inc.

Abstract

Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements, particularly those represented by nodes and directed edges (e.g., arrows and lines). To investigate the underlying causes of this limitation, we probe the internal representation of LVLMs using a carefully constructed synthetic diagram dataset based on directed graphs. Our probing experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model. In contrast, node information and global structural features are already linearly encoded in individual hidden states of the vision encoder. These findings suggest that the stage at which linearly separable representations are formed varies depending on the type of visual information. In particular, the delayed emergence of edge representations may help explain why LVLMs struggle with relational understanding, such as interpreting edge directions, which require more abstract, compositionally integrated processes.

Figures & tables

Figure 1: Overview of this study. We analyze internal representations in LVLMs using probing on a synthetic diagram dataset. We find that node information (e.g., node color) and global information (e.g., node count) are linearly encoded in a single image patch within the vision encoder, whereas edge information (e.g., edge color) is linearly encoded in a single text token within the language model.

Category	Aspect $a$	Label set $\mathcal{Y}$
Single	Node Color	{red, green, yellow, blue, brown, orange, pink, purple}
	Node Shape	{circle, square, pentagon, hexagon, septagon}
	In-degree	{0, 1, 2, 3, 4}
	Out-degree	{0, 1, 2, 3, 4}
Multiple	Edge Color	{red, green, yellow, blue, brown, orange, pink, purple}
	Edge Style	{solid, dashed}

Table 1: The 3 categories, 11 aspects, and their corresponding label sets

\mathcal{Y}

Figure 2: Examples of synthetic diagrams. Each diagram contains five nodes, and we control evaluation aspects such as node color, shape, and edge connectivity. We provide two variants: $\mathcal{D}_{\mathrm{rand}}$ , which uses random node layouts (left part), and $\mathcal{D}_{\mathrm{fix}}$ , which uses fixed layouts (right part).

	Node Color	Node Shape	In-degree Count	Out-degree Count	Edge Color	Edge Style	Edge Existence	Edge Direction	Multi-hop Path	Node Count	Edge Count
Chance level	12.5	20.0	20.0	20.0	12.5	50.0	50.0	50.0	50.0	20.0	20.0
Qwen3-VL 8B	91.4	76.6	40.3	34.7	57.3	73.5	69.6	49.3	58.3	40.3	21.6

Table 2:

\mathrm{Acc}_{\mathrm{VQA}}

for Qwen3-VL 8B. Accuracy exceeded chance level for all aspects except Edge Direction.

Figure 3: Layer-wise maximum accuracy in the vision encoder of Qwen3-VL 8B. The x-axis denotes the relative layer position (0 is the input layer and 1 is the final layer), and the y-axis denotes accuracy. Aspects sharing the same threshold are drawn with the same line style, which also matches the style of the corresponding black threshold lines.

Figure 4: Position-wise accuracy in the vision encoder of Qwen3-VL 8B. Each heatmap shows accuracy by patch position for a specific layer and aspect. The node layout of the evaluation diagrams is the same as that of the diagrams in the right part of Figure 2 .

Figure 5:

\mathrm{MaxAcc}_{l}

per layer in the language model of Qwen3-VL 8B. The x-axis denotes the layer position (0 is the input layer and 1 is the final layer), and the y-axis denotes accuracy. Aspects sharing the same threshold are drawn with the same line style, which also matches the style of the corresponding black threshold lines.

Figure 6: $\mathrm{acc}_{l,t}(d_{\mathrm{fix}})$ in the language model of Qwen3-VL 8B for a representative subset $d_{\mathrm{fix}}$ . The node layout of the evaluation diagrams is the same as that of the diagrams in the right part of Figure 2 .

Figure 7: Probing results for text tokens in the language model of Qwen3-VL 8B. The x-axis shows token position in the question, and the y-axis shows layer position.

Figure 8: Overview of causal intervention. Hidden states of patches with probing accuracy exceeding the threshold are replaced with the mean of hidden states of other patches. If VQA accuracy decreases with this operation, it suggests that the replaced patch hidden states contributed to inference.

	Node Color	Node Shape	In-degree Count	Out-degree Count	Edge Color	Edge Style	Edge Existence	Edge Direction	Multi-hop Path	Node Count
Clean run	91.4	76.6	40.3	34.7	57.3	73.5	69.6	49.3	58.3	40.3
Patched run	11.7	20.4	26.5	27.7	32.1	68.1	69.4	48.8	57.0	25.7
Controlled	83.4	73.6	36.8	34.3	53.8	72.3	68.6	50.3	57.1	31.8
Chance level	12.5	20.0	20.0	20.0	12.5	50.0	50.0	50.0	50.0	20.0
$\Delta$ (Clean run - Patched run)	79.7	56.2	13.8	7.0	25.2	5.4	0.2	0.5	1.3	14.6
$\Delta$ (Clean run - Controlled)	8.0	3.0	3.5	0.4	3.5	1.2	1.0	-1.0	1.2	8.5

Table 3: Results of causal intervention on Qwen3-VL 8B. Clean run shows VQA accuracy without intervention. Patched run shows accuracy when all patches exceeding the threshold in probing accuracy are replaced. Controlled refers to the controlled experiment setting described in section 6.1 .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect	Question
Node Color	What color is node A?
Node Shape	What shape is node A?
In-degree	What is the in-degree of node A?
Out-degree	What is the out-degree of node A?
Edge Color	What color is the edge between node A and node B?
Edge Style	What style is the edge between node A and node B?

Appendix

Table 4: Questions used to evaluate each aspect.

Parameter	Image Part	Text Part
Epochs	1000	1000
Batch size *	{1000, 2000, 4000, 8000, 16000}	{10, 20, 40, 80, 160}
Learning rate *	[0.0001, 0.01] (log scale)	[0.0001, 0.01] (log scale)
Optimizer	AdamW	AdamW
betas	[0.9, 0.999]	[0.9, 0.999]
eps	1e-08	1e-08

Appendix

Table 5: Hyperparameters used for probe training. Batch size and learning rate were subject to hyperparameter search using Optuna. Optimizer parameters use PyTorch’s AdamW default values. * Subject to hyperparameter search

Figure 9: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 10: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 11: Probing results for text tokens in Qwen3-VL 8B. The x-axis shows token position in the question, and the y-axis shows layer position.

Figure 12: Example diagrams with layouts distinct from those in Figure 2 .

Figure 13: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B vision encoder (node A placed on the left side). Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 14: Accuracy for each hidden state in the image input part of the Qwen3-VL 8B language model (node A placed on the left side). Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 15: Layer-wise maximum accuracy in the vision encoder of Qwen2.5-VL 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.

Figure 16: Accuracy for each hidden state in the image input part of the Qwen2.5-VL 7B vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 17: Layer-wise maximum accuracy in the language model of Qwen2.5-VL 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.

Figure 18: Accuracy for each hidden state in the image input part of the Qwen2.5-VL 7B language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 19: Probing results for text tokens in Qwen2.5-VL 7B. The x-axis shows token position in the question, and the y-axis shows layer position.

Figure 20: Layer-wise maximum accuracy in the vision encoder of LLaVA1.5 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.

Figure 21: Accuracy for each hidden state in the image input part of the LLaVA1.5 7B vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 22: Layer-wise maximum accuracy in the language model of LLaVA1.5 7B. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.

Figure 23: Accuracy for each hidden state in the image input part of the LLaVA1.5 7B language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 24: Probing results for text tokens in LLaVA1.5 7B. The x-axis shows token position in the question, and the y-axis shows layer position.

Figure 25: Layer-wise maximum accuracy in the vision encoder of Gemma3-4B-IT. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.

Figure 26: Accuracy for each hidden state in the image input part of the Gemma3-4B-IT vision encoder. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 27: Layer-wise maximum accuracy in the language model of Gemma3-4B-IT. The x-axis shows relative layer position (0 is the input layer, 1 is the final layer), and the y-axis shows accuracy. The black dotted line indicates the threshold.

Figure 28: Accuracy for each hidden state in the image input part of the Gemma3-4B-IT language model. Each heatmap shows the accuracy of individual hidden states for a specific layer and aspect.

Figure 29: Probing results for text tokens in Gemma3-4B-IT. The x-axis shows token position in the question, and the y-axis shows layer position.

	Node Color	Node Shape	In-degree Count	Out-degree Count	Edge Color	Edge Style	Edge Existence	Edge Direction	Multi-hop Path	Node Count	Edge Count
Clean run	91.9	78.0	37.0	28.8	54.9	71.0	50.0	46.0	65.0	83.6	22.2
Patched run	11.9	20.4	25.0	21.8	40.9	73.0	50.5	47.0	51.0	17.2	N/A
Controlled	93.4	78.0	37.4	26.8	56.6	69.0	50.0	45.0	69.5	21.0	N/A
Chance level	12.5	20.0	20.0	20.0	12.5	50.0	50.0	50.0	50.0	20.0	20.0
Patched ratio	28.5	22.3	37.9	31.6	16.0	8.20	3.50	1.56	28.1	93.0	100

Appendix

Table 6: Results of the causal intervention for Qwen2.5-VL 7B. “Patched” reports the results when all hidden states whose probing accuracy exceeded the threshold were replaced. “Controlled” reports the results when the same number of patches as those whose probing accuracy exceeded the threshold were randomly selected and replaced.

	Node Color	Node Shape	In-degree Count	Out-degree Count	Edge Color	Edge Style	Edge Existence	Edge Direction	Multi-hop Path	Node Count	Edge Count
Clean run	40.5	39.6	34.0	30.8	38.1	49.5	60.5	50.0	50.0	20.0	27.2
Patched run	49.6	28.8	71.2	52.8	16.0	50.5	59.5	50.0	50.0	21.8	N/A
Controlled	9.20	19.6	20.0	22.4	13.0	50.0	59.0	50.0	50.0	21.2	N/A
Chance level	12.5	20.0	20.0	20.0	12.5	50.0	50.0	50.0	50.0	20.0	20.0
Patched ratio	89.8	83.5	94.4	93.9	75.3	11.1	0.174	0	65.3	99.8	100

Appendix

Table 7: Causal intervention results for LLaVA1.5 7B. “Patched run” denotes the results when all patches whose probing accuracy exceeds the threshold are replaced. “Controlled” denotes the results when the same number of patches as those exceeding the threshold are randomly selected and replaced.

	Node Color	Node Shape	In-degree Count	Out-degree Count	Edge Color	Edge Style	Edge Existence	Edge Direction	Multi-hop Path	Node Count	Edge Count
Clean run	69.2	50.0	26.4	30.0	41.2	57.2	56.6	50.3	50.0	33.9	20.1
Patched run	35.8	33.6	25.1	27.0	41.0	56.9	57.0	50.1	50.0	9.40	20.5
Controlled	68.2	48.3	25.7	29.8	40.8	57.2	56.9	49.9	50.0	23.9	20.6
Chance level	12.5	20.0	20.0	20.0	12.5	50.0	50.0	50.0	50.0	20.0	20.0
Patched ratio	7.55	5.44	7.06	6.28	0.708	1.34	0.457	0.425	11.0	72.3	82.4

Appendix

Table 8: Causal intervention results for Gemma3-4B-IT. “Patched run” reports the results when all hidden states whose probing accuracy exceeded the threshold are replaced. “Controlled” reports the results when the same number of hidden states as those exceeding the probing-accuracy threshold is randomly selected and replaced.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 734 papers

Investigating Relational Reasoning in VLMs
Aug 24, 2026Adhithya Laxman Ravi Shankar Geetha, Aulia Kharis Rakhmasari, Haleema Ramzan +1Reasoning for VLMs Vision-Language Models
Department of Computer Science, ETH Zurich, Switzerland
Prior Bias in Vision Language Models on UML Diagram Interpretation
Jul 3, 2026Zaiyu Cheng, Khai-Nguyen Nguyen, Antonio MastropaoloVLM Evaluation Software Engineering
Dept. of Computer Science William & Mary, USA
Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
Apr 6, 2026Haruka Kawasaki, Ryota Tanaka, Kyosuke NishidaLarge Vision-Language Models Document Image Understanding
Human Informatics Labs., NTT, Inc.

Aug 24, 2026J/K move · Enter open · S save

Investigating Relational Reasoning in VLMs

Adhithya Laxman Ravi Shankar Geetha, Aulia Kharis Rakhmasari, Haleema Ramzan, Xander Yap

Department of Computer Science, ETH Zurich, Switzerland

Reasoning for VLMs Vision-Language Models VLM Reasoning VLM Interpretability

Open paper AskPDF arXiv

Vision-Language Models (VLMs) achieve strong performance in visual reasoning tasks, but it remains unclear whether they understand visual relations, or simply employ shortcuts such as language cues or priors. To investigate this, we use the Qwen3-VL-4B (Bai et al., 2025), a modern VLM, to decode how visual information is encoded across depths. For this, we propose a synthetic dataset of simple geometric shapes for controlled analysis, along with queries crafted to precisely test language cues. Furthermore, the dataset is modified to test causal reliance on visual evidence. Our results show that current VLMs combine genuine visual reasoning with shortcut strategies primarily rooted in language cues.

Abstract

Vision-Language Models

Explore similar work

Investigating Relational Reasoning in VLMs

Prior Bias in Vision Language Models on UML Diagram Interpretation

Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding