Organizations: Hangzhou Dianzi University Hangzhou, China · Beijing University of Technology Beijing, China · Xi’an Jiaotong University Xi’an, China · Tsinghua University Beijing, China
Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data. Furthermore, existing refinement strategies often rely on exhaustive, full-page reconstruction to align cross-modal information, leading to prohibitive computational redundancy and the introduction of contextual noise in long-form document processing. In this paper, we propose Hyper-M2RAG, a novel framework that redefines multimodal document retrieval through High-order Hypergraph Representation Learning. We first formalize the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling. To mitigate semantic fragmentation caused by physical pagination, we introduce an Anchor-driven Incremental Refinement mechanism. Rather than performing a global sweep, our approach identifies boundary-crossing anchor nodes and reconstructs their local hyper-topology using one-hop neighborhood contexts. This targeted refinement effectively bridges cross-page knowledge gaps with minimal computational footprints. Extensive evaluations on multimodal benchmarking datasets demonstrate that Hyper-M2RAG significantly outperforms state-of-the-art methods in both retrieval precision and generation coherence. Our code is available at https://github.com/ShenAoChen2001/MMHRAG.
Figures & tables
Figure 1. Overview of the Hyper-M2RAG framework. The system progresses through three core stages: (1) Multi-modal Data Ingestion via PMUs; (2) Two-stage Hypergraph Construction featuring anchor-driven structural refinement; (3) Cognitive-inspired Hybrid Retrieval . The comparative case (bottom right) highlights our advantage in cross-page factual grounding over vanilla LLMs.
Metrics
Mix
Neurology
World History
Sustainable Report
GraphRAG
Ours
Tie
GraphRAG
Ours
Tie
GraphRAG
Ours
Tie
GraphRAG
Ours
Tie
Comprehensiveness
0.0
46.4
53.6
0.8
37.6
61.6
0.8
47.2
52.0
0.0
62.4
37.6
Diversity
11.2
60.8
28.0
9.6
56.0
34.4
4.8
69.6
25.6
3.2
91.2
5.6
Empowerment
4.0
56.0
40.0
3.2
61.6
35.2
5.6
66.4
28.0
1.6
77.6
20.8
Overall
2.4
51.2
46.4
2.4
52.0
45.6
4.8
61.6
33.6
0.8
77.6
21.6
HyperRAG
Ours
Tie
HyperRAG
Ours
Tie
HyperRAG
Ours
Tie
HyperRAG
Ours
Tie
Table 1. Win-rate analysis comparing Hyper-M2RAG against GraphRAG, HyperRAG, and MegaRAG across four distinct domains. Results are presented as percentages (%). Bold values indicate the winning method (excluding ties).
Method
TechReport
TechSlides
GraphRAG
30.0
57.0
HyperRAG
56.0
67.0
MegaRAG
68.0
69.0
Hyper-M2RAG (Ours)
70.0
78.0
Table 2. Local QA performance (Accuracy %) on RealMMBench.
Variant
TechReport
TechSlides
HyperRAG (Baseline)
56.0
67.0
+ MM Extraction
64.0
74.0
+ Anchor Refinement
60.0
69.0
Hyper-M2RAG (Full)
70.0
78.0
Table 3. Ablation study of Hyper-M2RAG components.
Variant
TechReport (1,674 pages)
TechSlides (1,963 pages)
Low
High
Total
Low
High
Total
HyperRAG (Baseline)
12,957
2,579
15,536
8,543
1,716
10,259
HyperRAG + MM
10,016
2,078
12,094
9,309
2,004
11,313
Hyper-M2RAG (Ours)
14,221
8,257
22,478
12,522
6,571
19,093
Table 4. Comparison of structural hyperedge counts across datasets.
Sustainable Report
World History
Evaluator
H-RAG
Ours
Tie
H-RAG
Ours
Tie
Kimi-K2.6
25.6
55.2
19.2
24.0
44.0
32.0
GLM-5.1
25.6
51.2
23.2
24.8
38.4
36.8
DeepSeek-V4 Pro
15.2
41.6
43.2
20.0
36.0
44.0
Table 5. Multi-Judge Audit (%). H-RAG = HyperRAG.
Metric
MegaRAG
HyperRAG + Page
Ours
Build & Refine Time (min)
16
18
12
Token Consumption ( 103 )
2,742.2
3,376.1
765.9
Table 6. Efficiency on World History (per 100 pages).
Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly ``translation-to-text'' pipelines that discard fine-grained visual information. To address these limitations, we propose \textbf{MG2-RAG}, a lightweight Multi-Granularity Graph RAG framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG2-RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual grounding, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this representation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) demonstrate that MG2-RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3× speedup and 23.9× cost reduction compared with advanced graph-based frameworks.
Sijun Dai, Qiang Huang, Xiaoxing You +1
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University
Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question--answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.
While Multimodal Retrieval-Augmented Generation (MM-RAG) has shown promising results, it still struggles with complex multi-hop reasoning tasks. Existing methods primarily focus on independent instance-level matching, which often fails to capture explicit relationships across modalities and documents. Although Graph-enhanced methods introduce structural modeling, they face a fundamental challenge in multimodal scenarios: incorporating fine-grained visual features leads to rapid graph expansion and retrieval noise, whereas coarse-grained representations cause the discarding of critical local evidence. To address this dilemma, we propose DualG-MRAG, a Dual-tier framework that introduces a decoupled architecture comprising Macro-reasoning and Micro-matching Graphs for Multimodal RAG. Specifically, to suppress retrieval noise by isolating global structural reasoning from fine-grained evidence matching, we construct a Macro Graph for global topological routing and a Micro Graph for precise local verification. Subsequently, to enable dynamic relevance propagation across heterogeneous evidence sources, we formulate retrieval as a query-driven message passing process via a GNN Retriever. Furthermore, to provide the generative model with coherent structural guidance, we introduce a dynamic programming decoding mechanism that extracts explicit reasoning paths directly from the GNN's forward pass, replacing the standard input of isolated document chunks. Extensive experiments demonstrate that DualG-MRAG outperforms baselines in both evidence recall and complex QA accuracy.
Jiacheng Tao, Qingyun Sun, Haonan Yuan +2
SKLCCSE, School of Computer Science and Engineering Beihang University Beijing, China