We study when and how multimodal input improves knowledge graph question answering (KGQA) in large vision--language models. Using a controlled evaluation over 1,500 WebQSP and CWQ questions and six VLMs within a fixed KGQA pipeline, we compare text-only prompting against text-plus-graph visualization at the per-question level and analyze performance changes as a function of reasoning complexity, model size, and image layout. Adding a graph image yields an average gain of 1.85 F1 points, driven by strong precision improvements on low-complexity questions (+4.65 F1). Mixed-effects and regression analyses show that these multimodal gains do not depend strongly on model scale, but are highly sensitive to graph layout: image-only settings sharply degrade performance, hierarchical Graphviz layouts outperform less organized renderings, and decomposing large subgraphs into separated paths yields the largest improvements. Taken together, the results indicate that visual prompting primarily aids KGQA by enhancing prediction precision in low-complexity contexts, provided that visual design respects cognitive and perceptual constraints. To optimize both accuracy and inference efficiency, we argue for adaptive visual prompting policies that selectively append diagrams to small, visually sparse subgraphs with well-structured layouts, avoiding unnecessary visual clutter and token overhead on complex queries. We conclude by relating these findings to a visual notation design theory, outlining how principled diagram encoding can guide future multimodal VLM prompting.
Figures & tables
Figure 1: High-level abstraction of KGQA pipelines. Steps 1–3 retrieve relevant KG evidence; Steps 4–5 use it to generate an answer. Our analysis is agnostic to 1–3. We focus on Step 4–5, comparing textual and multimodal evidence representations at inference time.
WebQSP
CWQ
All
Questions
750
750
1,500
Mean answers
16.52
2.12
9.32
Mean triples
18.47
20.54
19.51
Median triples
9
14
12
Low complexity ( ≤ 7)
354
282
636
Medium complexity (8–25)
159
184
343
Table 1: Evaluation dataset statistics.
Metric
Baseline
Mean Δ
95% CI
p -value
F1
75.14%
+1.85
[1.30, 2.42]
<10−4
Precision
77.58%
+2.89
[2.31, 3.51]
<10−4
Recall
79.87%
+0.63
[0.06, 1.21]
0.032
Table 2: Effect of adding images (multimodal vs text-only) on QA performance, macro-averaged across models at the per-question level. For the p-value we used a paired sign-flip permutation test.
F1 (%)
Precision (%)
Recall (%)
Complexity Tier
Base
Δ
Base
Δ
Base
Δ
Low ( n≤7 )
76.82
+4.65
79.10
+5.58
81.24
+2.86
Medium ( 7<n≤25 )
75.11
+0.47
77.62
+0.96
79.80
-0.22
High ( 25<n )
73.49
+0.28
76.02
+2.00
78.57
-0.86
Table 3: Baseline text-only performance and multimodal gain ( Δ percentage points) across graph complexity tiers defined by triple count n .
Variable / Term
Coefficient ( β )
95% CI
p -value
Intercept (Low, Text-Only)
0.8513
[0.839, 0.863]
<0.001
Main Effects
Medium Complexity
-0.0543
[-0.072, -0.037]
<0.001
High Complexity
-0.2504
[-0.268, -0.233]
<0.001
Multimodal (Text + Image)
+0.0465
[0.029, 0.063]
<0.001
Interaction Terms
Table 4: OLS regression results for F1 score, modeling graph complexity tiers, prompt modality, and their interaction terms. Reference level is Low Complexity × Text-Only.
Figure 2: Model-wise performance gains from multimodal input across all metrics. Significant at p < 0.05.
Condition
Δ F1
Δ Precision
Δ Recall
Image Only (Graphviz)
-9.42
-8.52
-9.76
Text + Image (Graphviz)
+2.40
+4.14
+1.13
Text + Image (Non-hierarchical)
+0.87
+2.59
-0.44
Text + Image (NetworkX)
-0.40
+1.08
-1.69
Text + Reasoning-Path Images
+3.03
+3.90
+2.19
Table 5: Ablation study of visual design choices and input modality conditions for Qwen 3.5 35B A3B. Values report absolute percentage point changes ( Δ ) relative to its text-only baseline ( F1=72.30% , Precision=74.90% , Recall=76.10% ).
Table 6: Matched Analysis for Case 1 (WebQTest-1319). Comparison showing how visual topology allows the model to prune majority-class textual frequency distractors.
Whom was the person (that was born in Garfield) that Judy Garland was married to?
Gold Answer
[’Mickey Deans’]
Reasoning Paths
1. Garfield → people_born_here → Mickey Deans
2. Garfield → people_born_here → Tippy Larkin
3. Garfield → people_born_here → Stephen Fabian
4. Marriage → type_of_union → m.0j4k8cq → spouse_s → James A. Garfield
Appendix
Table 7: Matched Analysis for Case 2 (WebQTrn-428). Example demonstrating visual prompting degradation due to hub-entity radial clutter, reasoning path noise, and visual entity-label distraction.
Table 8: Matched Analysis for Case 3 (WebQTrn-2496). Output progression for Qwen3.5-35B-A3B demonstrating how alternative spatial layouts ( NetworkX , non-hierarchical Graphviz ) and reasoning-path image decomposition resolve the visual bottleneck present in standard hierarchical Graphviz renderings.
Figure 3: The four visualizations for Case 1 ( WebQTest-1319 ): (a) Graphviz, (b) NetworkX, (c) Graphviz with a non-hierarchical layout, and (d) reasoning paths.
Figure 4: The four visualizations for case 2 ( WebQTest-428 ): (a) Graphviz, (b) NetworkX, (c) Graphviz with a non-hierarchical layout, and (d) excerpt of reasoning paths.
Figure 5: The four visualizations for case 3 ( WebQTest-2496 ): (a) Graphviz, (b) NetworkX, (c) Graphviz with a non-hierarchical layout, and (d) excerpt of reasoning paths.
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.
Recent work in visual question answering has shown that vision-language models can exhibit strong reasoning capabilities by translating visual inputs into textual representations. The effectiveness of this translation depends on how well visual details are retained; models need to surface and align both explicit and implicit knowledge sufficient to support reasoning, without introducing spurious assumptions. Existing methods that leverage detailed image captions introduce visual details unrelated to the reasoning task, inflating input token counts and increasing computational cost. To address these challenges, we propose VisKG, a reinforcement learning (RL) framework in which models learn to translate visual content into question-specific knowledge graph (KG) representations. This process filters out perceptual noise while preserving the entity-relation structure needed for chain-of-thought reasoning, following the principle of minimum sufficient information. To ensure stable RL post-training, VisKG adopts Group reward-Decoupled Normalization Policy Optimization (GDPO). In addition, we strengthen the supervision stage with negative rationale samples, exposing the model to incorrect reasoning paths before RL post-training. Experimental results across science, mathematics, and general visual understanding benchmarks show that VisKG achieves performance comparable to or better than baselines, while requiring fewer tokens than caption-based representations. Moreover, training VisKG with GDPO improves accuracy by 2% over its GRPO-trained counterpart on average. These results suggest that KG representations are a promising approach for supporting multi-step reasoning and open up future work on adaptively selecting the most suitable representation for a given task.
Multimodal large language models (MLLMs) can miss fine details in a full image that they recognize in a closer view. Recovering this evidence requires deciding where to look and how much surrounding context to retain. We present Q-CueGraph, a query-conditioned evidence acquisition method for frozen MLLMs. For text-rich images, it builds a reusable graph of OCR lines and layout relations. Each question activates anchors, expands them into contextual regions, and selects candidates for a single observation window. Query-conditioned object detections support natural-image search through the same region-selection and composition interface. A lightweight candidate scorer further learns which observations support correct answers from frozen-reader feedback and training answers, without evidence-box supervision. Across six benchmarks, we examine the roles of query conditioning, evidence composition, and learned answerability. With Qwen2.5-VL-7B, Q-CueGraph raises V*Bench accuracy from 0.696 to 0.832 using 19.1% of source-image area, and retains 92% of full-image ANLS on InfographicVQA using about half the image area. The analyses show that useful evidence depends on both its relevance to the question and the context available to the reader. Q-CueGraph makes these choices explicit before answer generation.
Graphs have been used to enhance large language models (LLMs) for structured reasoning, mostly as external knowledge sources are provided to models at test time. In this paper, we take a different view: the value of graphs for LLMs lie not only in supplying information, but also in organizing reasoning. Inspired by how humans use graph-structured mind maps to organize branching and converging thoughts, we ask whether graphs can serve as an internal form of reasoning assistance. We study this question on multi-hop question answering tasks, where teacher-provided reasoning traces are rewritten as graph mind maps and used to guide a student model. Our experiments reveal a clear modality gap. When graph structures are flattened into text, their benefits become limited once direct answer hints are removed. Under this abstract guidance setting, both reasoning efficiency and answer quality degrade substantially. In contrast, visual graph guidance remains effective without direct answer clues, and its advantage persists after supervised fine-tuning and KL-based distillation. The above findings support the claim that graphs should be studied not only as external knowledge structures for LLMs, but also as visual scaffolds for organizing reasoning.
Runlin Lei, Xiaokui Xiao, Zhewei Wei
Renmin University of China · National University of Singapore