cs.AIOct 23, 2026

GEMS-RAG: Graph-grounded Engineering Multimodal Standards Retrieval-Augmented Generation

Authors: Abdul Hannan Azad, Irvin Wang, Jiwan Jiang, Chenchen Kuai, Bofeng Cao, Keshu Wu, Yuzhou Wang, Xiaopeng Li, 

Organizations: Texas A&M University, College Station, USA · A&M Consolidated High School College Station, USA · University of Wisconsin, Madison, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Engineering standards are challenging knowledge sources for large language models (LLMs) because they are hierarchical, multimodal, and densely connected through figures, tables, cross-references, and normative rules. Existing retrieval-augmented generation (RAG) methods often rely on unstructured text retrieval and struggle with questions requiring joint reasoning over textual provisions, visual specifications, and document structure. We propose GEMS-RAG (Graph-grounded Engineering Multimodal Standards Retrieval-Augmented Generation), a framework for grounded reasoning over engineering standards. GEMS-RAG combines hierarchical parsing, fine-grained visual entity extraction, multimodal knowledge graph construction, and graph-grounded hybrid retrieval. Instead of treating figures as monolithic images, it decomposes them into meaningful visual entities linked to textual provisions, engineering concepts, and regulatory rules. During inference, dense, sparse, and visual retrieval are combined with graph expansion to identify relevant evidence across text, figures, and document structure. We evaluate GEMS-RAG on the Manual on Uniform Traffic Control Devices (MUTCD), a large engineering standard containing egulations, technical drawings, and standardized traffic-control devices. Results show that graph-grounded multimodal retrieval improves retrieval effectiveness and question-answering performance over text-only, multimodal, and graph-based RAG baselines. Beyond MUTCD, GEMS-RAG offers a general framework for reasoning over engineering codes, technical manuals, regulatory documents, and scientific standards in safety-critical applications.

Figures & tables

Figure 1: Framework of GEMS-RAG
Figure 2: Knowledge graph for the R1-1 STOP sign example.
ComponentCount
Sections678
Text Chunks5705
Figures485
Tables68
Knowledge Graph Nodes8240
Knowledge Graph Edges18245
Table 1: MUTCD dataset statistics.
Figure 3: Retrieval and visual-evidence performance.
Figure 4: Question-answering performance.
Figure 5: Claude Fable 5 performance by evidence modality and by M-SDI difficulty and answerability.
Figure 6: Retrieval performance comparison
Figure 7: Question answering performance
ConditionRetr. (/25)Gen. (/60)Reliab. (/12)Overall (/100)
GEMS-RAG19.3555.1810.9088.08
Without VLM filter19.59 (+0.24)55.12 (-0.06)10.70 (-0.20)88.06 (-0.02)
Without hierarchy prior19.59 (+0.24)55.10 (-0.08)10.54 (-0.36)87.86 (-0.22)
Without reranker17.78 (-1.57)52.77 (-2.41)10.38 (-0.52)83.43 (-4.65)
Table 2: Ablation results. Best results are bolded and second-best results are underlined. Unit: percent.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
Text representationBAAI/bge-m3 , used for dense and learned sparse representations
Visual representationvidore/colqwen2-v0.1
Cross-encoder rerankermixedbread-ai/mxbai-rerank-large-v2
Page and figure rendering180 dpi for pages, 220 dpi for figure and table crops
Text retrieval depths30 dense, 30 sparse, 30 fused, 40 after graph expansion, 6 after reranking
Visual retrieval depths10 candidate entities, 6 direct visual candidates, at most 4 retained entities, 4 page images
Appendix
Table 1: Core indexing and retrieval hyperparameters.
SettingValue
Detailed-analysis and ablation modelClaude Fable 5, alias frontier_claude
Answer promptFew-shot, 3 examples
Visual-filter promptZero-shot
Claude output-token cap16,000 tokens, including adaptive-thinking tokens and visible answer tokens
Truncation handlingOne retry with a multiplier of 2.0, capped at 32,000 tokens for Claude
Temperature and top-pNot set explicitly in the adapter, so the provider defaults applied
Appendix
Table 2: Generation, benchmark-runner, and evaluation hyperparameters.
CheckVerified result
Strict full-credit flag compared with all four dimensions equal to 1.000 mismatches in 1,050 records
Descriptive mean compared with the mean of correctness, completeness, faithfulness, and citation0 mismatches in 1,050 records
Per-condition correctness, completeness, faithfulness, citation, and full-credit aggregates compared with the version 4 reportExact match for all seven conditions
Strict full-credit counts120, 114, 112, 114, 112, 109, and 103 for the baseline and A1 through A6
Rationale provenance895 structured reconstructions, 48 explicit adjudication notes, and 107 reused baseline notes
Reused-note criterion0 violations. Each reused note had the same question identifier and identical four official scores as the baseline
Appendix
Table 3: Programmatic checks of the per-response rationale package.
More in this topic

Multimodal RAG

Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.

See all 158 papers

Explore similar work

CardsList
Aug 2, 2026

BEST-KAG: Enhancing Question Answering of Building Engineering Standards with Multimodal Knowledge Graph Modeling and Large Language Model

Construction standards are critical for building safety and sustainability. Existing standard application workflows rely on keyword-based document retrieval and manual cross-clause interpretation, which cannot reliably support multi-clause reasoning, multimodal knowledge utilization, or traceable clause-level evidence linkage. To address these limitations, this study develops a multimodal knowledge-driven framework that supports question answering on standard knowledge named BEST-KAG (Knowledge-Augmented Generation for Building Engineering STandards). The framework introduces 1) a multimodal knowledge graph (MKG) for unified representation of document hierarchy and heterogeneous standard knowledge with various connections, 2) a rule-LLM hybrid knowledge construction pipeline for scalable multimodal knowledge extraction, creating a large MAG with 251 building engineering standards, 171,652 nodes and 310,914 edges, and 3) a graph-retrieval-based knowledge-augmented generation architecture for clause-grounded and traceable question answering. Experiments demonstrate that BEST-KAG consistently outperforms multiple mainstream LLMs in terms of Expert evaluation, and metrics including BLEU, and ROUGE, with the best improvement up to 74.01% compared to the baselines.

Jia-Rui Lin, Junxi Guo, Keyin Chen +1

Department of Civil Engineering, Tsinghua University, Beijing, China, 100084 · Key Laboratory of Digital Construction and Digital Twin, Ministry of Housing and Urban-Rural Development, Beijing, China, 100084

Document UnderstandingKG ConstructionGraph Retrieval-Augmented GenerationKnowledge Graph Question AnsweringMultimodal KGsKnowledge Graphs
AskPDF
Apr 4, 2026

MG2^2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly ``translation-to-text'' pipelines that discard fine-grained visual information. To address these limitations, we propose \textbf{MG2^2-RAG}, a lightweight Multi-Granularity Graph RAG framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG2^2-RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual grounding, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this representation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) demonstrate that MG2^2-RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3×\times speedup and 23.9×\times cost reduction compared with advanced graph-based frameworks.

Sijun Dai, Qiang Huang, Xiaoxing You +1

School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University

Cross-Modal LearningMultimodal RAGGraph Retrieval-Augmented GenerationMultimodal ReasoningCross-Modal RetrievalMultimodal KGs+2
AskPDF
Sep 16, 2026

Less Is More: Graph-free Multimodal RAG via Multi-signal Late Fusion

Graph-based retrieval-augmented generation (RAG) is widely used for multimodal, cross-document question answering. However, building corpus-level graphs is expensive, slow to query, and difficult to maintain. We present TrioRAG, a graph-free multimodal framework that integrates evidence from three complementary signals: the question, the anchor image, and a VLM-enhanced query generated from both. Each signal retrieves independently over a shared multi-vector index of page text and page images, and the results are combined through late fusion. Further, we introduce AutoQA, a multimodal automotive benchmark whose questions are grounded in noisy, web-sourced images rather than clean document-sourced figures. Its questions require reasoning across manuals. We position it as a model-curated testbed rather than a human-validated gold standard. Across three benchmarks, TrioRAG matches or outperforms graph-based systems while reducing total cost and accelerating per-query inference by 1.6-2.3 times. By construction, AutoQA grounds its questions in out-of-corpus web images. In this setting image retrieval reaches only 19.3% document-level recall, while text-derived signals, especially the VLM-enhanced query, keep retrieval robust.

Tithi Rakshit, Hongkuan Zhou, Lavdim Halilaj +1

University of Tübingen, Tübingen, Germany · Corporate Research, Robert Bosch GmbH, Renningen, Germany · University of Stuttgart, Stuttgart, Germany

Late FusionMultimodal RAGMultimodal IRQuestion AnsweringMultimodal QAMultimodal Fusion
AskPDF