MMGraph-Agent: Agentic Multimodal RAG via Cache-Inspired Multimodal Knowledge HyperGraphs
Organizations: Nanyang Technological University · Beijing University of Posts and Telecommunications · Monash University · Beijing Institute of Computer Technology and Application ·
Abstract
Driven by increasingly complex real-world applications, Retrieval-Augmented Generation (RAG) has evolved from textual pipelines to multimodal settings, comprising two complementary dimensions: knowledge organization and retrieval enhancement. However, existing methods still face challenges of high construction cost, context window limitations, and the lack of unified training across offline and online knowledge sources. To address these challenges, we propose MMGraph-Agent, a unified multimodal RAG framework based on cache-inspired multimodal knowledge hypergraphs. MMGraph-Agent enables extremely lightweight hypergraph construction with zero API cost and dynamic memory-based retrieval that alleviates long-context issues. Experiments across offline, online, and joint retrieval settings demonstrate consistent improvements in performance, efficiency, and architectural unification. Our software and data are publicly available at https://github.com/LHRLAB/MMGraph-Agent.
Figures & tables
| Method | Knowledge Organization | Retrieval Enhancement |
| StandardRAG ( Lewis et al., 2020 ) | ||
| StandardMMRAG ( Yu et al., 2025 ) | ||
| HyperGraphRAG ( Luo et al., 2025a ) | -ary relational entity hyperedges. | Staged entity-then-hyperedge retrieval. |
| RAG-Anything ( Guo et al., 2025b ) | Dual structural and textual graphs. | Graph navigation with semantic matching. |
| Graph-R1 ( Luo et al., 2026a ) | Lightweight higher-order hypergraph. | RL-driven multi-turn hyperedge search. |
| MMSearch-R1 ( Wu et al., 2026 ) | Dynamic multimodal text-image web. | RL-driven multi-turn web search agent. |
| You are a helpful assistant. Answer the given question based on the provided images; you can query the knowledge base as many times as you want. You must first conduct reasoning inside <reason> … </reason> . For image search, append <search><img></search> behind the reasoning, which runs Google Lens on the original image; for text search, output a concise query inside <text_search> … </text_search> , answered by KG/Memory retrieval (and Google Search when enabled). Retrieved evidence returns inside <information> … </information> ; when you have the final answer, output it inside <answer> … </answer> . Here is the image and the question: <image> question . Assistant: |
| Datasets | NaiveGen | StdMMRAG | SFT | GRPO | RAG-Any | MMS-R1 | MMGraph-Agent-3B | ||
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B | GPT-4.1 | Qwen2.5-3B | GPT-4.1 | Qwen2.5-3B | Qwen2.5-3B | GPT-4.1 | Qwen2.5-3B | (Ours) | |
| DocBench | 6.74 0.3 | 14.43 0.8 | 9.54 0.4 | 17.79 0.3 | 9.88 0.5 | 16.50 0.7 | 17.57 0.9 | 22.39 1.4 | 33.05 4.4 (+10.66) |
| MMLongBench-Doc | 4.14 0.2 | 4.78 0.3 | 5.63 0.2 | 7.33 1.8 | 4.95 0.3 | 9.91 1.1 | 9.29 1.5 | 11.08 1.1 | 20.64 2.3 (+9.56) |
| ViDoSeek † | 2.64 0.1 | 6.49 0.4 | 6.32 0.5 | 7.58 0.8 | 3.08 0.6 | 9.37 0.6 | 8.58 0.4 | 15.89 0.8 | 38.95 3.5 (+23.06) |
| 2Wiki | 4.11 0.3 | 15.89 2.0 | 12.51 1.3 | 15.94 1.5 | 12.42 0.7 | 14.42 0.7 | 15.30 1.1 | 15.36 1.1 | 17.68 5.3 (+1.74) |
| HotpotQA | 7.29 1.3 | 12.79 1.2 | 11.40 1.3 | 13.85 1.2 | 15.16 1.0 | 15.51 0.6 | 13.32 1.0 | 14.72 1.3 | 20.70 4.2 (+5.19) |
| Method | Trace |
|---|---|
| NaiveGen Qwen2.5-VL-3B no retrieval F 1 = 0.50 (avg. 0.028) | <reason> 71-y/o female, acute monocular vision loss; normal pupillary reflexes consistent with vascular occlusion. The fundus in Figure A shows a characteristic appearance of CRAO: a narrowed retinal artery with ‘beading’ and a pale optic disc. [hallucinated: absent from the actual fundus] </reason> <answer> (B) Central retinal artery occlusion </answer> wrong; F 1 = 0.50 (shared tokens “retinal”, “occlusion”). |
| MMSearch-R1 Qwen2.5-VL-3B RL-tuned web agent (text + Lens, no cache) F 1 = 0.50 (avg. 0.193) | <text_search> CRAO vs RVO elderly painless monocular vision loss </text_search> <information> “CRAO classically presents with sudden, painless monocular vision loss in the elderly …” </information> [anchors symptom-driven prior on CRAO] <search><img> </search> <information> “[Lens] The image likely depicts a retina with evidence of retinal vein occlusion (RVO) …” </information> [contradicts the text retrieval] <reason> Two retrievals conflict; the symptom triad is the canonical prior, weigh text more. </reason> <answer> (B) Central retinal artery occlusion </answer> wrong; without cache-grounded reward to trust visual evidence, the policy anchors on the textbook prior. |
| MMGraph-Agent (Ours) Qwen2.5-VL-3B hypergraph + L1/L2 cache + graph/web policy F 1 = 1.00 (avg. 0.372) | <reason> Multiple vascular differentials (CRAO, RVO, detachment) match the symptoms; the fundus is the discriminating signal. </reason> <search><img> </search> cache routing: L2 warm-up multimodal hypergraph web search cache write <information> “[Search Results] The images in the provided figure likely depict a retina with evidence of retinal vein occlusion (RVO) … Since the 71-year-old female presented with acute monocular vision loss in her right eye, combined with waking sensations without pain, the most probable diagnosis is (C) Retinal vein occlusion.” </information> [digest] <reason> Visual classification matches (C); CRAO prior never enters the chain. </reason> <answer> Retinal vein occlusion </answer> exact match; search_count = 1; one image-grounded round was both necessary and sufficient. |
Multimodal RAG
Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.
See all 167 papers