GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG

Zafar Ali; Asad Khan; Aalia Malik; Pavlos Kefalas

cs.AIAug 15, 2026

GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG

Authors: Zafar Ali, Asad Khan, Aalia Malik, Pavlos Kefalas

Organizations: School of Computer Science and Engineering, Southeast University Nanjing, China · Dashub & Department of Informatics, Aristotle University Thessaloniki, Greece

Abstract

Multimodal retrieval-augmented generation (RAG) systems often rely on long unstructured contexts or aggressively expanded evidence graphs, which can introduce noisy evidence, weaken multi-hop reasoning, and increase unsupported generation. We present GraphLoom, a reliability-calibrated multimodal knowledge-graph RAG framework for compact and faithful evidence routing. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, extracted relational triples, and external commonsense knowledge. Instead of injecting all retrieved evidence into the generator, GraphLoom performs reliability-aware subgraph retrieval with bounded expansion and selectively routes high-utility evidence through hierarchical graph memory slots and joint graph-sequence attention in a frozen language model. To improve robustness in complex reasoning settings, GraphLoom further combines interleaved retrieval with budgeted corrective retrieval, enabling adaptive multi-hop evidence refinement under noisy retrieval conditions. We evaluate GraphLoom on ScienceQA, MultiModalQA, and OK-VQA, including large distractor evidence pools that approximate noisy external knowledge retrieval. Experimental results show consistent gains in answer quality and evidence faithfulness over strong multimodal RAG, graph-retrieval, and open-source vision-language baselines, with improved retrieval quality on MultiModalQA and stable performance under noisy evidence pools. Additional analyses using MiniCheck-based verification, human evaluation, and latency profiling show that reliability-calibrated graph evidence routing provides an effective alternative to long-context multimodal evidence injection.

Figures & tables

Figure 1. GraphLoom architecture. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph, retrieves a compact task-conditioned evidence subgraph, and organizes the retrieved evidence into multimodal prefix memories and hierarchical key–value graph slots. A reliability-calibrated router selects high-utility slots, which are injected into a frozen decoder through KG-JSA++ joint graph–sequence attention. Interleaved retrieval and budgeted corrective retrieval improve multi-hop evidence coverage, while verification–revision supports faithful answer generation.

Model	ScienceQA Acc $\uparrow$	MMQA EM $\uparrow$	MMQA F1 $\uparrow$
LLaVA-1.5 (7B) ( Liu et al., 2023 )	89.3 $\pm$ 0.2	44.8 $\pm$ 0.3	54.9 $\pm$ 0.4
Qwen2.5-VL ( Bai et al., 2025 )	91.2 $\pm$ 0.2	50.6 $\pm$ 0.4	61.9 $\pm$ 0.4
InternVL3 ( Zhu et al., 2025 )	91.8 $\pm$ 0.2	51.4 $\pm$ 0.4	62.8 $\pm$ 0.4
LLaVA-OneVision-1.5 ( An et al., 2025 )	91.5 $\pm$ 0.3	50.9 $\pm$ 0.5	62.3 $\pm$ 0.4
Qwen3-VL-Instruct (2B) ( Bai and others, 2025 )	89.5 $\pm$ 0.2	46.4 $\pm$ 0.4	55.9 $\pm$ 0.3
MuRAG ( Chen et al., 2022 )	87.9 $\pm$ 0.3	47.8 $\pm$ 0.5	57.8 $\pm$ 0.4

Table 1. QA results on ScienceQA and MMQA. Bold indicates the best controlled open-source result; closed-source models are reference-only comparisons.

Model	VQA Accuracy $\uparrow$
LLaVA-1.5 (7B) ( Liu et al., 2023 )	61.2 $\pm$ 0.4
Qwen2.5-VL ( Bai et al., 2025 )	66.2 $\pm$ 0.4
InternVL3 ( Zhu et al., 2025 )	67.4 $\pm$ 0.3
LLaVA-OneVision-1.5 ( An et al., 2025 )	66.8 $\pm$ 0.4
Qwen3-VL-Instruct (2B) ( Bai and others, 2025 )	63.8 $\pm$ 0.3
MuRAG ( Chen et al., 2022 )	59.5 $\pm$ 0.5

Table 2. Results on OK-VQA. GraphLoom achieves the strongest controlled open-source score and remains competitive with closed-source reference models.

Variant	EM $\uparrow$	F1 $\uparrow$	H-Rate $\downarrow$
Full GraphLoom	55.2 $\pm$ 0.4	66.5 $\pm$ 0.3	10.5 $\pm$ 0.4
w/o Interleaved Retrieval	53.6 $\pm$ 0.5	64.6 $\pm$ 0.4	11.1 $\pm$ 0.5
w/o Corrective Retrieval	54.1 $\pm$ 0.4	65.0 $\pm$ 0.5	11.8 $\pm$ 0.6
w/o Post-hoc Revision	54.7 $\pm$ 0.4	65.7 $\pm$ 0.4	13.9 $\pm$ 0.5
w/o Reliability-Calibrated Router	51.8 $\pm$ 0.6	61.6 $\pm$ 0.5	17.3 $\pm$ 0.7
w/o KG-JSA++	50.5 $\pm$ 0.7	60.3 $\pm$ 0.6	18.9 $\pm$ 0.8

Table 3. Ablation on MMQA. Removing key components lowers EM/F1 and increases H-Rate, highlighting the importance of controlled evidence routing and joint graph–sequence attention.

Panel of Figure 2 — Figure 2. Reliability diagram for slot routing. The calibrated router lies closer to the perfect-calibration diagonal, reducing ECE from 0.087 to 0.032.

Candidate pool	Recall@5 $\uparrow$	MRR $\uparrow$	MMQA F1 $\uparrow$	Online latency (ms) $\downarrow$
Instance graph only	78.4	0.742	66.5	119
+10K distractor triples	77.6	0.734	66.2	124
+100K distractor triples	75.9	0.713	65.8	132
+1M distractor triples	73.2	0.684	65.1	148

Table 4. Scalability under noisy candidate evidence pools on MMQA. GraphLoom degrades gradually as the retrieval space grows to one million distractor triples.

Setting	MMQA EM $\uparrow$	MMQA F1 $\uparrow$	H-Rate $\downarrow$
Default	55.2	66.5	10.5
$\tau_{\mathrm{edge}}-20\%$	54.8	66.1	11.2
$\tau_{\mathrm{edge}}+20\%$	54.5	65.8	10.9
$\tau_{\mathrm{KB}}-20\%$	54.7	66.0	11.3
$\tau_{\mathrm{KB}}+20\%$	54.4	65.7	11.0
$\tau_{\mathrm{eval}}-20\%$	54.6	65.9	11.8

Table 5. Robustness under

\pm 20\%

threshold perturbations on MMQA. Performance remains stable under moderate changes to the main evidence-selection and verification thresholds.

Model	Independent H-Rate (%) $\downarrow$	Human Hallucination-Free (%) $\uparrow$
LLaVA-1.5 ( Liu et al., 2023 )	25.2 $\pm$ 0.8	39.5 $\pm$ 1.2
Qwen2.5-VL ( Bai et al., 2025 )	17.6 $\pm$ 0.6	56.2 $\pm$ 1.1
InternVL3 ( Zhu et al., 2025 )	16.8 $\pm$ 0.5	57.9 $\pm$ 1.0
SKURG ( Yang et al., 2023 )	20.3 $\pm$ 0.7	49.8 $\pm$ 1.1
CRAG ( Yan et al., 2024 )	18.1 $\pm$ 0.6	54.7 $\pm$ 1.3
RARR ( Gao et al., 2023 )	15.2 $\pm$ 0.5	59.8 $\pm$ 1.0

Table 6. Hallucination assessment. Automatic H-Rate is computed with MiniCheck , independently of the revision-time verifier; human evaluation uses 300 balanced examples.

Panel of Figure 3 — Figure 3. Sensitivity analysis of active slots and edge budget. The selected defaults ( $k=4$ , $N_{\max}=50$ ) balance answer quality and computation.

Module	Latency / query (ms) $\downarrow$
Scene description generation	142
Triple extraction	44
Entity linking + KG enrichment	18
Initial retrieval	9
Interleaved retrieval	13
Corrective retrieval	7

Table 7. Latency breakdown of GraphLoom on one NVIDIA A100 80GB GPU. Conditional modules are reported as amortized per-query costs.

Multimodal RAG

Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.

See all 158 papers

TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning
Oct 7, 2026Ruochi Li, Jianzhe Lin, Haoxuan Zhang +4Graph Retrieval-Augmented Generation Multimodal RAG
Department of Computer Science, North Carolina State University, USA · Independent Researcher · Department of Information Science, University of North Texas, USA +2
HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document
Jul 26, 2026Xin He, Yili Wang, Wenqi Fan +4Multimodal RAG Retrieval-Augmented Generation
Jilin University Changchun, China · The Hong Kong Polytechnic University Hong Kong SAR, China
MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal Learning Multimodal RAG
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University

Oct 7, 2026J/K move · Enter open · S save

TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning

Ruochi Li, Jianzhe Lin, Haoxuan Zhang, Haihua Chen+3

Department of Computer Science, North Carolina State University, USA · Independent Researcher · Department of Information Science, University of North Texas, USA+2

Graph Retrieval-Augmented Generation Multimodal RAG RAG Evaluation

Open paper AskPDF arXiv

Real-world documents distribute evidence across text, tables, figures, and captions within complex page layouts. Answering complex questions over such documents therefore requires more than retrieving relevant passages: systems must recover the evidence topology that connects heterogeneous evidence units. Existing GraphRAG evaluations remain largely text-centered, while multimodal document RAG benchmarks assess cross-modal retrieval and generation without directly evaluating recovery of the intended evidence topology. We introduce TOPOGRAPHRAG-BENCH, a layout-grounded benchmark for multimodal evidence reasoning in GraphRAG, comprising 2,024 questions over 201 long, visually rich documents. Questions are constructed bottom-up from text, figure, and table evidence units under three controlled topologies: single-hop retrieval, bridge-chain reasoning, and multi-source synthesis. To ensure that questions preserve their intended structure, we apply counterfactual validation for shortcut resistance, modality necessity, and evidence necessity. We evaluate text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems using retrieval, generation, and topology-aware reasoning metrics. Multimodal GraphRAG systems achieve the strongest overall performance, but still fail when visual-textual evidence alignment or multi-unit composition is incomplete. Text-only GraphRAG struggles when key dependencies are grounded in figures or tables, while page-level visual retrieval lacks the fine-grained structure needed for topology recovery. These findings motivate GraphRAG systems that move beyond text-derived entity relation graphs to explicitly model document layouts, cross-modal evidence alignment, and the reasoning roles of evidence units. Code and data are available at https://richardlrc.github.io/TopoGraphRAG-Bench/.

Abstract

Multimodal RAG

Explore similar work

TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning

HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document

MG2^22-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation