We study when and how multimodal input improves knowledge graph question answering (KGQA) in large vision--language models. Using a controlled evaluation over 1,500 WebQSP and CWQ questions and six VLMs within a fixed KGQA pipeline, we compare text-only prompting against text-plus-graph visualization at the per-question level and analyze performance changes as a function of reasoning complexity, model size, and image layout. Adding a graph image yields an average gain of 1.85 F1 points, driven by strong precision improvements on low-complexity questions (+4.65 F1). Mixed-effects and regression analyses show that these multimodal gains do not depend strongly on model scale, but are highly sensitive to graph layout: image-only settings sharply degrade performance, hierarchical Graphviz layouts outperform less organized renderings, and decomposing large subgraphs into separated paths yields the largest improvements. Taken together, the results indicate that visual prompting primarily aids KGQA by enhancing prediction precision in low-complexity contexts, provided that visual design respects cognitive and perceptual constraints. To optimize both accuracy and inference efficiency, we argue for adaptive visual prompting policies that selectively append diagrams to small, visually sparse subgraphs with well-structured layouts, avoiding unnecessary visual clutter and token overhead on complex queries. We conclude by relating these findings to a visual notation design theory, outlining how principled diagram encoding can guide future multimodal VLM prompting.
Figures & tables
Figure 1: High-level abstraction of KGQA pipelines. Steps 1–3 retrieve relevant KG evidence; Steps 4–5 use it to generate an answer. Our analysis is agnostic to 1–3. We focus on Step 4–5, comparing textual and multimodal evidence representations at inference time.
WebQSP
CWQ
All
Questions
750
750
1,500
Mean answers
16.52
2.12
9.32
Mean triples
18.47
20.54
19.51
Median triples
9
14
12
Low complexity ( ≤ 7)
354
282
636
Medium complexity (8–25)
159
184
343
Table 1: Evaluation dataset statistics.
Metric
Baseline
Mean Δ
95% CI
p -value
F1
75.14%
+1.85
[1.30, 2.42]
<10−4
Precision
77.58%
+2.89
[2.31, 3.51]
<10−4
Recall
79.87%
+0.63
[0.06, 1.21]
0.032
Table 2: Effect of adding images (multimodal vs text-only) on QA performance, macro-averaged across models at the per-question level. For the p-value we used a paired sign-flip permutation test.
F1 (%)
Precision (%)
Recall (%)
Complexity Tier
Base
Δ
Base
Δ
Base
Δ
Low ( n≤7 )
76.82
+4.65
79.10
+5.58
81.24
+2.86
Medium ( 7<n≤25 )
75.11
+0.47
77.62
+0.96
79.80
-0.22
High ( 25<n )
73.49
+0.28
76.02
+2.00
78.57
-0.86
Table 3: Baseline text-only performance and multimodal gain ( Δ percentage points) across graph complexity tiers defined by triple count n .
Variable / Term
Coefficient ( β )
95% CI
p -value
Intercept (Low, Text-Only)
0.8513
[0.839, 0.863]
<0.001
Main Effects
Medium Complexity
-0.0543
[-0.072, -0.037]
<0.001
High Complexity
-0.2504
[-0.268, -0.233]
<0.001
Multimodal (Text + Image)
+0.0465
[0.029, 0.063]
<0.001
Interaction Terms
Table 4: OLS regression results for F1 score, modeling graph complexity tiers, prompt modality, and their interaction terms. Reference level is Low Complexity × Text-Only.
Figure 2: Model-wise performance gains from multimodal input across all metrics. Significant at p < 0.05.
Condition
Δ F1
Δ Precision
Δ Recall
Image Only (Graphviz)
-9.42
-8.52
-9.76
Text + Image (Graphviz)
+2.40
+4.14
+1.13
Text + Image (Non-hierarchical)
+0.87
+2.59
-0.44
Text + Image (NetworkX)
-0.40
+1.08
-1.69
Text + Reasoning-Path Images
+3.03
+3.90
+2.19
Table 5: Ablation study of visual design choices and input modality conditions for Qwen 3.5 35B A3B. Values report absolute percentage point changes ( Δ ) relative to its text-only baseline ( F1=72.30% , Precision=74.90% , Recall=76.10% ).
Table 6: Matched Analysis for Case 1 (WebQTest-1319). Comparison showing how visual topology allows the model to prune majority-class textual frequency distractors.
Whom was the person (that was born in Garfield) that Judy Garland was married to?
Gold Answer
[’Mickey Deans’]
Reasoning Paths
1. Garfield → people_born_here → Mickey Deans
2. Garfield → people_born_here → Tippy Larkin
3. Garfield → people_born_here → Stephen Fabian
4. Marriage → type_of_union → m.0j4k8cq → spouse_s → James A. Garfield
Appendix
Table 7: Matched Analysis for Case 2 (WebQTrn-428). Example demonstrating visual prompting degradation due to hub-entity radial clutter, reasoning path noise, and visual entity-label distraction.
Table 8: Matched Analysis for Case 3 (WebQTrn-2496). Output progression for Qwen3.5-35B-A3B demonstrating how alternative spatial layouts ( NetworkX , non-hierarchical Graphviz ) and reasoning-path image decomposition resolve the visual bottleneck present in standard hierarchical Graphviz renderings.
Figure 3: The four visualizations for Case 1 ( WebQTest-1319 ): (a) Graphviz, (b) NetworkX, (c) Graphviz with a non-hierarchical layout, and (d) reasoning paths.
Figure 4: The four visualizations for case 2 ( WebQTest-428 ): (a) Graphviz, (b) NetworkX, (c) Graphviz with a non-hierarchical layout, and (d) excerpt of reasoning paths.
Figure 5: The four visualizations for case 3 ( WebQTest-2496 ): (a) Graphviz, (b) NetworkX, (c) Graphviz with a non-hierarchical layout, and (d) excerpt of reasoning paths.
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.