PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

Nabaraj Subedi; Shuvo Dip Datta; Ahmed Abdelaty; Shivanand Venkanna Sheshappanavar

cs.IRAug 26, 2026

PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

Authors: Nabaraj Subedi, Shuvo Dip Datta, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar

Organizations: Department of Electrical Engineering & Computer Science, University of Wyoming · Department of Civil & Architectural Engineering & Construction Management, University of Wyoming, 1000 E. University Ave., Laramie, 82071, WY, USA

Abstract

Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the geometry and layout essential for interpreting these plans. We present a Visual-First Multimodal Retrieval-Augmented Generation (RAG) framework called PlanSightRAG. It indexes and reasons directly over plan imagery, integrates a ColNomic-3B multi-vector retrieval, an agentic Planner-Retriever-Auditor-Synthesizer, and MaxSim heatmaps as an evidence trail. We introduce a 4,056-pair benchmark from five state Departments of Transportation (DOT) standard plans (1,898 pages). PlanSightRAG achieves 91.47% Recall@5 on zero-shot retrieval, while on a held-out Michigan DOT corpus, it achieves 91.40%. On synthetic, parametrically-generated compliance drawings, our Qwen2.5-VL-72B pipeline reaches 100% verdict accuracy only when supplied a pre-resolved rule threshold, a controlled ceiling that a non-VLM OCR baseline already reaches at 76.4%. Finally, we demonstrate autonomous visual rule-grounding by extracting numeric limits directly from a specification corpus without any human-supplied rules.

Figures & tables

Panel of Figure 2 — Figure 2: Architecture of the Visual-First Multimodal RAG framework, organized into four phases: (1) document ingestion with high-resolution tiling and metadata enrichment; (2) ColNomic-3B visual indexing with multi-vector patch embeddings; (3) MaxSim late-interaction retrieval and grounding (optional VLM re-ranking and BM25 fusion); and (4) downstream tasks—(4a) multi-VLM visual question answering with heatmap grounding and (4b) agentic compliance auditing via the Planner–Retriever–Auditor–Synthesizer pipeline.

Panel of Figure 3 — Figure 3: Advanced Sliding Window Technique. Engineering sheets are rasterized at 400 DPI and decomposed into overlapping $1024\times 1024$ tiles with 256-pixel overlap, preserving fine-grained annotations and enabling sub-page retrieval granularity.

Category	$N$	Recall@5 (%)	Judge (%)
Dimensional	29	100.00 [100.0, 100.0]	79.31 [62.1, 93.1]
Visual	16	87.50 [68.8, 100.0]	68.75 [43.8, 87.5]
Logical	17	94.12 [82.4, 100.0]	94.12 [82.4, 100.0]
Hallucination	16	75.00 [50.0, 93.8]	81.25 [62.5, 100.0]
Overall	78	91.03 [84.6, 96.2]	80.77 [71.8, 89.7]

Table 1: Human-curated 78-pair WYDOT anchor set: per-category Recall@5 and end-to-end judge accuracy with

95\%

bootstrap CIs (

10,000

resamples, seed 42). Retrieval is over the 237-page WYDOT-only index; judge accuracy uses the local Qwen2.5-VL-7B answerer with a Qwen2.5-VL-72B judge (no proprietary API).

Category	WYDOT	Caltrans	AZDOT	CDOT	FDOT	Total
Visual Interpretation	201	416	149	85	549	1,400
Dimensional Accuracy	170	287	144	87	327	1,015
Logical Reasoning	172	331	104	104	446	1,157
Hallucination Rate	15	192	17	2	258	484
Total	558	1,226	414	278	1,580	4,056

Table 2: Distribution of the 4,056-pair five-DOT benchmark by agency and reasoning category. The held-out Michigan DOT zero-shot set (93 pairs) is not included in this table.

Model	Type	Params	Tier
Qwen2.5-VL-72B-Instruct	Open-source	72B	Primary
Qwen2.5-VL-7B-Instruct	Open-source	7B	Primary
InternVL-2.5-8B	Open-source	8B	Primary

Table 3: Vision–language models used in the framework.

Method	Modality	Mechanism	$N_{\text{idx}}$	Recall@5
Strong modern baselines (2024–2026)
Nemotron-ColEmbed-8B [ 43 ]	Visual (Patch)	Multi-vector late-int.	1898	95.28%
Nemotron-ColEmbed-4B [ 43 ]	Visual (Patch)	Multi-vector late-int.	1898	92.92%
ColNomic-7B [ 38 ]	Visual (Patch)	Multi-vector late-int.	1898	91.27%
ColQwen2.5-v0.2 [ 7 ]	Visual (Patch)	Multi-vector MaxSim	1898	87.26%
DSE-Qwen2-2B [ 44 ]	Visual (Global)	Single-vector dense	1898	22.17%

Table 4: Retrieval comparison on the 424-pair page-disjoint test split over the 1,898-page five-DOT index. Strong (modern) baselines on top, legacy/weak baselines below, and the predecessor ColPali backbone (zero-shot) at the bottom; the adopted retriever is ColNomic-3B (92.69%, bolded). All numbers measured under an identical evaluation protocol; per-agency and per-category bootstrap CIs are reported in Appendix C .

Configuration	Recall@5 (%)
ColPali (Stage 1 only)	76.89
ColPali + MonoQwen2-VL rerank	85.14
Top-20 ceiling	86.79

Table 5: Two-stage retrieve–rerank on the 424-pair test split. The predecessor ColPali backbone retrieves the top-20; MonoQwen2-VL-v0.1 (Apache-2.0 pointwise visual reranker) reorders them. The top-20 recall is the reranker’s ceiling.

Agency	$N$	Zero-shot Recall@5 (%)
WYDOT	558	93.37
FDOT	1,580	91.27
Caltrans	1,226	91.92
AZDOT	414	89.13
CDOT	278	90.29
Overall	4,056	91.47

Table 6: Zero-shot ColNomic-3B retrieval on the 4,056-pair five-DOT benchmark (1,898-page index), used off-the-shelf with no domain adaptation.

Category	$N$	Zero-shot Recall@5 (%)
Hallucination Rate	484	88.84
Dimensional Accuracy	1,015	95.37
Visual Interpretation	1,400	92.71
Logical Reasoning	1,157	87.64
Overall	4,056	91.47

Table 7: Zero-shot category-wise Recall@5 on the 4,056-pair five-DOT benchmark. Drafters: Qwen2.5-VL-72B for WYDOT, AZDOT, CDOT; Llama-3.2-90B-Vision for Caltrans, FDOT. Verifier: Qwen2.5-VL-7B (yes/no rubric).

Technique	Qwen-7B	Qwen-72B
Zero-shot	78.30	76.89
Chain-of-thought (CoT)	70.75	72.17
Self-consistency (5 samples)	69.58	69.10
Retrieval-augmented	62.50	70.75
Sliding-window tiling	64.39	72.17
OCR-hybrid	77.12	78.07

Table 8: End-to-end judge accuracy (%) for nine prompting and reasoning techniques on the 424-pair page-disjoint test split, with Qwen2.5-VL-7B and Qwen2.5-VL-72B as the answerer. Retrieval, judge, and decoding are held fixed; only the answering-stage prompting strategy varies.

Panel of Figure 6 — Figure 6: Visual evidence trail from the agentic compliance pipeline. Each audit step generates a sharpened MaxSim heatmap overlay on the retrieved standard plan page, creating a transparent chain of evidence.

Judge Configuration	Compliant	Non-compliant	Overall
Qwen2.5-VL-7B (plain)	5/5	1/5	60%
Qwen2.5-VL-7B (CoT)	0/5	5/5	50%
Qwen2.5-VL-72B (plain)	5/5	3/5	80%
Qwen2.5-VL-72B (CoT + thresholds)	5/5	5/5	100%
Multi-agent (Plan–Audit–Synth, 72B + thresholds)	5/5	5/5	100%
Dense multi-view drawings ( $n{=}4$ , best configuration):

Table 9: Verdict accuracy of five judge configurations on the

n{=}10

single-doc CAD set; the final block reports the best single-judge configuration on

n{=}4

dense schedule-table drawings. Interpretation is in the text below.

$N$ plans	Before d/2 fix	After d/2 fix
$N{=}2$	25/25	25/25
$N{=}3$	25/25	25/25
$N{=}4$	25/25	25/25
$N{=}5$	13/25	25/25
Total	88/100 (88%)	100/100 (100%)

Table 10: Multi-plan compliance verdict accuracy on the

n{=}100

scale set, before and after replacing the textual d/2 threshold in the rules registry with a per-drawing pre-resolved numeric value.

Set	Compliant	Non-compliant	Overall
$n{=}10$ single-doc (pilot)	5/5	5/5	100%
$n{=}4$ dense multi-view (pilot)	2/2	2/2	100%
$n{=}10$ multi-plan (pilot)	4/5	5/5	90%
$n{=}500$ single-doc (scale)	250/250	250/250	100%
$n{=}100$ multi-plan (scale, after $d/2$ fix)	48/48	52/52	100%
$n{=}50$ adversarial near-threshold (stress)	25/25	25/25	100%

Table 11: Verdict accuracy of the best judge recipe (Qwen2.5-VL-72B 4-bit + CoT + per-drawing pre-resolved thresholds) across six CAD-generated compliance test sets (674 drawings spanning small pilots, large-scale runs, dense multi-view schedules, multi-plan cross-document designs, and an adversarial near-threshold stress test).

Archetype	Gov. dim. (layout)	OCR found	OCR acc	VLM acc
Culvert	cover, horiz.	100.0	100.0	100.0
Inlet	grate, horiz.	98.0	99.0	100.0
Guardrail	footing, rotated	32.0	82.0	100.0
Sign post	anchor, rotated	3.0	51.0	100.0
Rebar	stirrup, distractor	0.0	50.0	100.0
Overall		46.6	76.4	100.0

Table 12: Non-VLM OCR

+

threshold-compare baseline on scale500, per archetype.

Retrieval pool	Cand.	R@1	R@5	Extract	Verdict
Spec only	15	100.0	100.0	100.0	100.0
$+$ 1,898 plan pages	1,913	80.0	100.0	100.0	100.0
Injected-threshold ref. (human rule)					100.0

Table 13: Autonomous rule-grounding on the 500 single-doc compliance drawings: the agent retrieves the governing specification sheet, extracts its numeric limit, and audits the design against the self-grounded threshold (no injected rule).

Capability	Synthetic	Real plans
Standard retrieval (rel.@5)	—	83.3%
Verdict acc. (supplied rule)	100%	100%
Threshold extraction (auto.)	100%	unreliable

Table 14: Synthetic vs. real-plan compliance. The framework transfers from generated drawings to two production WYDOT projects for retrieval and supplied-threshold auditing; autonomous threshold extraction from dense real standards is the remaining gap.

Retriever	Trainable params	5-DOT test R@5 (%)	Michigan R@5 (%)
Adopted backbone
ColNomic-3B (zero-shot)	0	92.69	93.55
ColNomic-3B + head-LoRA, gentle	332k (0.009%)	92.69 ( 0.00 )	93.55 ( 0.00 )
ColNomic-3B + head-LoRA, standard	332k (0.009%)	93.16 ( $+0.47$ )	92.47 ( $-1.08$ )
ColNomic-3B + full-LM LoRA	37.2M (0.98%)	55.66 ( $-37.03$ )	75.27 ( $-18.28$ )
Predecessor backbone (companion)

Table 15: Retrieval Recall@5: zero-shot vs. three LoRA fine-tuning configurations, on the page-disjoint 5-DOT test split (

N{=}424

, in-distribution) and the held-out Michigan DOT corpus (

N{=}93

, zero-shot 6th-agency transfer).

Granularity	R@5 (%)	Tiles/Page
Full-page (200 DPI)	76.89	1
Tile-level (400 DPI)	82.08	$\approx$ 15

Table 16: Full-page vs. tile-level retrieval on the 424-pair page-disjoint test split over the 1,898-page five-DOT visual index.

Retrieval granularity	Judge accuracy (%)
Full-page (200 DPI)	64.23 [59.4, 68.8]
Tile-level (400 DPI)	68.77 [64.2, 73.3]
Gain (tile $-$ full)	+4.53 pp

Table 17: Judge accuracy of full-page vs. tile-level retrieval on the 424-pair test split (

N=397

queries whose target page yields

\geq

15 400 DPI tiles (about half the

\approx

30-tile typical landscape sheet)—in practice nearly all plan pages, which are uniformly tile-dense).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Category	Hit (%)	Judge (%)
Qwen2.5-VL-72B	Dimensional	86.61 [80.4, 92.9]	62.16 [53.2, 71.2]
	Visual	86.21 [80.0, 91.7]	78.47 [71.5, 84.7]
	Logical	52.50 [43.3, 61.7]	69.17 [60.8, 77.5]
	Hallucination	87.23 [76.6, 95.7]	70.21 [57.4, 83.0]
	Overall	76.89 [72.9, 80.9]	70.62 [66.1, 74.9]
Qwen2.5-VL-7B	Dimensional	86.61 [80.4, 92.9]	60.36 [51.4, 69.4]

Appendix

Table A.1 : Per-category Recall@5 (Hit) and Judge Accuracy (Judge) on the full 424-pair page-disjoint test split over the 1,898-page five-DOT index, with 95% bootstrap CIs (10,000 resamples, seed 42). Recall@5 is a clean zero-shot ColPali and is generator-independent. Judge Accuracy is scored by a local Qwen2.5-VL-72B judge (no proprietary API). Per-category

N

: Dim

=

112, Vis

=

145, Log

=

120, Hal

=

47. Logical-Reasoning Recall@5 is markedly lower here (52.50%) than in Table 7 (87.64%) because this table uses the predecessor ColPali backbone, whose fixed

448{\times}448

encoder is weakest on note/table-dense logical queries; the adopted ColNomic-3B closes this gap.

Method	Overall (%)	WYDOT	Caltrans	AZDOT	CDOT	FDOT
ColQwen2.5-v0.2	87.26 [84.0, 90.3]	92.5 [84.9, 98.1]	87.3 [81.3, 92.5]	82.6 [71.7, 93.5]	87.1 [74.2, 96.8]	86.9 [81.2, 91.9]
DSE-Qwen2-2B	22.17 [18.2, 26.2]	30.2 [18.9, 43.4]	20.1 [13.4, 26.9]	26.1 [13.0, 39.1]	25.8 [9.7, 41.9]	19.4 [13.1, 25.6]
VisRAG-Ret	53.77 [49.1, 58.5]	58.5 [45.3, 71.7]	46.3 [38.1, 54.5]	67.4 [54.3, 80.4]	48.4 [32.3, 64.5]	55.6 [47.5, 63.1]
BGE-M3 + OCR	36.79 [32.3, 41.5]	45.3 [32.1, 58.5]	26.1 [18.7, 33.6]	45.7 [32.6, 60.9]	41.9 [25.8, 58.1]	39.4 [31.9, 46.9]
CLIP ViT-B/32	1.89 [0.7, 3.3]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	8.7 [2.2, 17.4]	0.0 [0.0, 0.0]	2.5 [0.6, 5.0]
LayoutLMv3	0.00 [0.0, 0.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]

Appendix

Table A.2 : Per-baseline Recall@5 with 95% bootstrap CIs on the 424-pair page-disjoint test split (10,000 resamples, seed 42). Per-agency

N

: WYDOT

=53

, Caltrans

=134

, AZDOT

=46

, CDOT

=31

, FDOT

=160

Category	$N$	R@1	R@5	MRR	nDCG@10
Dimensional	112	67.86	97.32	0.803	0.847
Visual	145	68.28	93.10	0.788	0.828
Logical	120	69.17	88.33	0.778	0.810
Hallucination	47	53.19	89.36	0.682	0.741
Overall	424	66.75	92.45	0.777	0.819

Appendix

Table A.3 : Rank-sensitive retrieval metrics for zero-shot ColNomic-3B on the 424-pair test split (1,898-page index).

Candidate pool	# Candidates	R@1	R@5
Joint ( $1,898{+}298$ )	2,196	68.82	91.40
Michigan-only	298	74.19	93.55

Appendix

Table A.4 : Zero-shot Michigan transfer, joint vs. Michigan-only candidate pool.

ID	Plans	GT	Pred	Match	Steps
comp_01	511-1A, 203-2A	PASS	PASS	✓	8
comp_02	606-7C, 606-2B	PASS	PASS	✓	7
comp_03	511-1A	PASS	PASS	✓	8
comp_04	202-1	PASS	PASS	✓	9
comp_05	511-1A	PASS	PASS	✓	10
comp_06	606-4B	PASS	PASS	✓	8

Appendix

Table A.5 : Agentic compliance confusion matrix over 8 cross-plan verification queries. GT and Pred are the ground-truth and predicted verdicts; Match indicates agreement; Steps is the number of verification steps the Planner produced. Expected minimum decomposition is 2 steps per query.

Stratum	$N$	PASS / FAIL
GT $=$ PASS (originals)	8	8 / 0
GT $=$ FAIL, M dim	8	1 / 7
GT $=$ FAIL, M note	8	1 / 7
GT $=$ FAIL, M sym	8	3 / 5
Overall	32	13 / 19

Appendix

Table A.6 : False-negative sensitivity on the

N{=}32

synthetic query corpus. Per-stratum verdict counts (PASS / FAIL); GT

=

FAIL rows quantify detection sensitivity, and the GT

=

PASS row repeats the Section 5.6 PASS accuracy for reference.

Percentile	Full-page (424 queries, 1,898-page index)
min	98.72
p25	100.24
p50	101.26
p75	101.74
p95	102.48
p99	103.71

Appendix

Table A.7 : Retrieval-only latency distribution (ms) for full-page ColNomic MaxSim retrieval on the 424-pair page-disjoint test split over the combined 1,898-page five-DOT visual index. Wall-clock time collected with time.perf_counter() on a single NVIDIA A100 80GB with PyTorch 2.x, one query at a time (batch size 1).

Stage	Cost	Notes
Indexing (1,898 pages)	$7.3$ min (one-time)	$4.34$ pages/s, offline
Retrieval / query	$\approx$ $0.10$ s (p50)	query enc. $+$ MaxSim
Re-rank (optional)	$+$ $\approx$ $0.3$ s/cand.	$\leq K$ passes; early-stop
VQA answer / query	$2$ – $5$ s	single-shot 72B
Agentic audit / query	$60.9$ s	$\thicksim$ $8.6$ sequential steps

Appendix

Table A.8 : Consolidated deployment latency on a single 80 GB GPU (batch 1). Indexing is a one-time offline cost; all other rows are per query. The full retrieval percentile distribution is in Table A.7 ; the agentic figure is the average over the compliance runs (Section 5.6 ).

Panel of Figure 29 — Figure A.1 : Representative dimensional-accuracy success cases on the WYDOT benchmark.

Panel of Figure 30 — Figure A.2 : Hard failures arising from component–dimension binding and semantic confusion.

Figure A.3 : A rendered Standard-Specifications reference sheet from the rule-grounding corpus. The seven governing limits for the compliance archetypes (e.g., min. cover $2.0^{\prime\prime}$ , max. stirrup spacing $d/2$ ) are embedded among distractor rows; the agent must retrieve the correct sheet among 1,913 candidates and extract the governing limit from the table.

Multimodal RAG

Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.

See all 153 papers

Training-Free Agentic Computer Vision for Structural Component Detection in 2D Structural Framing Plans
Aug 18, 2026Mohammad Talebi-Kalaleh, Qipei MeiStructural Engineering Document Image Understanding
Department of Civil and Environmental Engineering University of Alberta
RAGOCR: Optical Compression of Retrieval-Augmented Text via Visual Representation
Aug 1, 2026Jiayang Yu, Jialun Zhong, Lei ZouMultimodal RAG Retrieval-Augmented Generation
Wangxuan Institute of Computer Technology, Peking University Beijing, China
AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation
Feb 27, 2026Zhengren Wang, Dongsheng Ma, Huaping Zhong +4Multimodal RAG Visual Document Retrieval
Shanghai Artificial Intelligence Laboratory · Peking University · SenseTime

Aug 18, 2026J/K move · Enter open · S save

Training-Free Agentic Computer Vision for Structural Component Detection in 2D Structural Framing Plans

Mohammad Talebi-Kalaleh, Qipei Mei

Department of Civil and Environmental Engineering University of Alberta

Structural Engineering Document Image Understanding

Open paper AskPDF arXiv

Converting structural framing plans into editable finite-element model drafts is labor-intensive and susceptible to transcription errors. Existing building-component recognition systems generally depend on task-specific neural detectors, whereas language-model agents in structural engineering typically operate on text or model data rather than on drawings. To the authors' knowledge, this work is the first to apply an agentic vision-language layer to structural-component detection and model drafting from framing-plan PDFs without task-specific detector training or fine-tuning. A deterministic stage extracts geometric primitives, estimates scale by dimension-ratio consensus, recognizes five entity classes using an explicit drafting grammar, and assembles an editable layout. The agentic stage constrains typed corrections through deterministic candidates, operation-specific admission tests, change-level review, and fail-closed transactions. Evaluation used an author-generated benchmark of 100 plans, divided equally between a development half used for all rule revisions and a seed-disjoint held-out half generated after the rules were frozen and evaluated once. All scores are end-to-end results for the complete framework on the held-out half. Scale estimates were within 0.1% of the generator reference for every drawing. Recall and precision were 0.922/0.997 for columns, 0.886/0.990 for beams, 1.000/1.000 for walls, 1.000/1.000 for braces, and 1.000/0.964 for openings. A controlled study repeated two corruptions three times on three development drawings. Calibration passed all nine trials, whereas member repair satisfied every strict end-state criterion in five of nine trials. Because both benchmark halves share a generator, the evaluation does not address independently drafted plans, raster input, analytical connectivity, or solver validation.

Abstract

Multimodal RAG

Explore similar work

Training-Free Agentic Computer Vision for Structural Component Detection in 2D Structural Framing Plans

RAGOCR: Optical Compression of Retrieval-Augmented Text via Visual Representation

AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation