Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables Graph RAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fusion. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire passage. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on the selected vision-centric dataset validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability.
Figures & tables
Figure 1. An illustration of node-based Multimodal Knowledge Graph where image is also treated as a node. An illustration of node-based Multimodal Knowledge Graph.
Figure 2. An illustration of textual context for a selected visual element, where useful contextual information may be distributed across different parts of a document rather than being limited to the immediate surroundings of the visual element. Such contextual information may provide richer semantics and strengthen the connections between image and text, thereby facilitating the construction of higher-quality MMKGs. An illustration of textual context for a selected visual element, where useful context information may be distributed across different parts of a document rather than being limited to the immediate surroundings of the visual element. Such contextual information may provide richer semantics for visual elements and strengthen the connections between image and text, thereby facilitating the construction of higher-quality MMKGs.
Figure 3. An overview of the designed textual context in our work. Given a target image and texts in the document, we design different levels of contextual information, including local context with different granularities and global context. An overview of the designed textual context in our work. Given a target image and texts in the document, we design different levels of contextual information, including local context with different granularities and global context.
Figure 4. An overview of context utilization during MMKG construction. The designed textual contexts can be incorporated into both the Image2Graph and modality fusion stages, individually or jointly, to support visual knowledge extraction and cross-modal knowledge fusion. In practice, an image-based graph is constructed for each visual element and subsequently fused with the text-based graph. For clarity, only one image-based graph is illustrated in the figure. An overview of context utilization during MMKG construction. The designed textual contexts can be incorporated into the Image2Graph and modality fusion stages, individually or jointly, to support visual knowledge extraction and cross-modal knowledge fusion.
Method
Additional Local Context
Global Context
Figure
Table
Chart
Misc
Overall
Direct Inference
–
–
6.25 / 6.25
4.00 / 4.00
0.00 / 0.00
21.95 / 21.95
11.32 / 11.32
MMGraphRAG
–
–
6.25 / 6.25
20.00 / 22.46
31.25 / 31.25
34.15 / 34.15
23.58 / 24.17
Ours + MMGraphRAG
Ref Paragraph
Doc-level
12.50 / 12.50
16.00 / 16.00
31.25 / 31.25
26.83 / 26.83
22.64 / 22.64
Ref Paragraph Summary
Doc-level
15.63 / 15.63
24.00 / 26.46
25.00 / 25.00
43.90 / 46.76
31.13 / 32.82
Ref Sentence
Doc-level
21.88 / 24.70
24.00 / 24.00
31.25 / 31.25
51.22 / 54.01
34.91 / 36.84
Table 1. Main results of different textual context designs on the VisionHeavy subset. Results are reported as strict full-credit accuracy / official MMLongBench-Doc soft accuracy (%). We additionally report results by visual-content type for a more fine-grained comparison, where each question may be associated with multiple types (Figure, Chart, and Table). Misc contains the remaining questions without any of these three labels.
Method
Additional Local Context
Strict / Soft Acc. (%)
RAG-Anything
–
28.89 / 36.18
Ours + RAG-Anything
Ref Paragraph
28.89 / 34.49
Ours + RAG-Anything
Ref Paragraph Summary
31.11 / 33.13
Ours + RAG-Anything
Ref Sentence
35.56 / 41.12
Table 2. Performance of CEMMKG with RAG-Anything as the backbone under different context configurations. Results are reported as strict / soft accuracy (%).
Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly ``translation-to-text'' pipelines that discard fine-grained visual information. To address these limitations, we propose \textbf{MG2-RAG}, a lightweight Multi-Granularity Graph RAG framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG2-RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual grounding, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this representation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) demonstrate that MG2-RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3× speedup and 23.9× cost reduction compared with advanced graph-based frameworks.
Sijun Dai, Qiang Huang, Xiaoxing You +1
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University
Multimodal knowledge graphs (MMKGs) organize information across different modalities to support multimodal Graph Retrieval-Augmented Generation (RAG), which aims to improve the performance of multimodal large language models (MLLMs). The effectiveness of these multimodal Graph RAG methods largely depends on the quality of the constructed MMKGs. However, most existing methods construct MMKGs without explicit guidance on graph quality, while reinforcement learning (RL)-based methods that rely on final-answer correctness may provide only indirect feedback on MMKG quality. To address these limitations, we propose MMKG-R1, a novel MMKG-quality-driven RL framework for training an MLLM as a multimodal knowledge constructor. To be specific, MMKG-R1 introduces multiple quality-aware rewards to directly evaluate both the quality of information extracted from individual knowledge sources and the quality of the assembled MMKG. These rewards are then used to guide the RL training. Experiments across multiple datasets show that MMKG-R1 constructs MMKGs that improve downstream question-answering accuracy.
Hypergraph-based RAG systems surpass traditional graph-based approaches by organizing complex n-ary atomic facts among entities, rather than relying solely on binary relationships. Despite the advancements in multimodal large language models (MLLMs) with enhanced visual capabilities, current hypergraph-based RAG frameworks predominantly restrict knowledge retrieval and reconstruction to a unimodal, text-centric paradigm. This limitation prevents them from fully leveraging the powerful visual perception capabilities of modern MLLMs. To address this gap, we systematically explore the integration of hypergraph awareness in RAG systems through visual cues. By incorporating visual representations of hypergraphs into the RAG pipeline, we introduce VizRAG, the first RAG system to support visual hypergraph structure awareness. Experimental results demonstrate that VizRAG significantly outperforms strong baselines, validating the promising potential of hypergraph visualization as a novel approach for RAG systems.
Yanbin Wei, Yang Chen, Renling Gan +7
1Southern University of Science and Technology · 2Hong Kong University of Science and Technology · 3Huawei Research +1