Grounded and Faithful P&ID Reasoning: Constraining Vision-Language Models with Recovered Evidence Graphs

Prathamesh Gadekar; Sagar Srinivas Sakhinana; Venkataramana Runkana

cs.LGSep 5, 2026

Grounded and Faithful P&ID Reasoning: Constraining Vision-Language Models with Recovered Evidence Graphs

Authors: Prathamesh Gadekar, Sagar Srinivas Sakhinana, Venkataramana Runkana

Organizations: Tata Research Development and Design Center, Pune, India 411057

Abstract

Piping and Instrumentation Diagrams (P&IDs) are the authoritative maps of process plants: isolation, maintenance, and HAZOP decisions depend on what connects to what. Vision-language models describe these sheets fluently, yet they often invent or miss process connections---and an invented or missed link can reverse an isolation or reachability call, so a plant decision cannot trust a fluent answer that was never checked against the linework. We instead recover an explicit graph of the drawing---its symbols, the process connections between them, and the tags that name them---and then require the model to answer only by querying that graph through seven read-only operators, so a topology claim is returned only when it cites the query results that support it. On TopoPID-VQA, a new suite of 3000 topology questions over these sheets, Graph-Grounded Harness (Ours) raises exact match accuracy from 36.7--41.3% under image-only prompting to 74.3--76.0% for Qwen3-VL-4B, Qwen3-VL-8B, and Gemma-4-E4B. It does so on an imperfect substrate: on Digitize-PID dataset the recovered graph scores F1 0.742 on exact process connections, and 0.801 once symbols and tags are pooled in. The residual errors track that gap---grounding pays off where the recovered graph is right, and perception error still breaks topology questions where it is not.

Figures & tables

Panel of Figure 1 — Figure 1: Grounding converts a plausible prediction into an auditable answer. An image-only VLM may invent a process connection without checking the underlying linework (left). A graph-grounded harness conditions the answer on recovered topology and returns inspectable evidence identifiers (right).

Panel of Figure 2 — Figure 2: Raster to evidence graph on one Digitize-PID sheet (notes and title block removed). Left: input drawing. Center: perception overlays for lines, symbols, and text. Right: process-connection graph recovered from those detections (nodes at symbol centers; edges from the first-symbol walk).

Panel of Figure 3 — Figure 3: Two-stage graph-grounded reasoning pipeline. Stage 1 converts a P&ID raster into the typed evidence graph $G_{\mathrm{auto}}$ (optional signal-edge layer on dashed instrument lines). Stage 2 combines $G_{\mathrm{auto}}$ with the marked sheet and question and produces a structured answer through a graph-tool harness. The reference graph $G_{\mathrm{oracle}}$ is used only for scoring and TopoPID-VQA supervision; it is never provided as agent input.

	Qwen3-VL-4B		Qwen3-VL-8B		Gemma-4-E4B
Level	Img.	Agent	Img.	Agent	Img.	Agent
Easy	$32.9\pm 2.5$	$85.7\pm 2.0$	$34.6\pm 2.4$	$83.1\pm 2.0$	$34.3\pm 2.6$	$82.6\pm 2.4$
	$\Delta^{\ast}=+52.8\pm 3.1\ \mathrm{pp}$		$\Delta^{\ast}=+48.5\pm 3.6\ \mathrm{pp}$		$\Delta^{\ast}=+48.3\pm 3.7\ \mathrm{pp}$
Medium	$44.5\pm 2.8$	$78.5\pm 2.8$	$45.8\pm 3.9$	$81.2\pm 2.0$	$38.2\pm 2.4$	$74.6\pm 2.7$
	$\Delta^{\ast}=+34.0\pm 4.6\ \mathrm{pp}$		$\Delta^{\ast}=+35.4\pm 4.6\ \mathrm{pp}$		$\Delta^{\ast}=+36.4\pm 3.7\ \mathrm{pp}$
Hard	$42.9\pm 3.7$	$63.9\pm 3.0$	$43.5\pm 3.7$	$60.9\pm 2.7$	$37.7\pm 2.5$	$65.8\pm 2.6$

Table 1: TopoPID-VQA exact-match mean accuracy (%) with sheet-clustered bootstrap CIs (

B{=}10{,}000

). Centered band:

\Delta=\mathrm{Agent}-\mathrm{Img.}

(paired half-width). Teal: light

\Delta{<}25

, mid

[25,40)

, strong

\geq 40

pp. ∗ = paired 95% CI excludes zero. Appendix A.5 .

Layer	$P$	$R$	$F_{1}$
Symbols (class-aware)	0.969	0.999	0.984
Exact edges	0.677	0.820	0.742
Conn. consistency	—	—	0.958
Symbol tags	0.792	0.954	0.866
Line tags	0.599	0.935	0.730
Combined tags	—	—	0.798

Table 2: Recovered process graph (

G_{\mathrm{auto}}

) as grounding-substrate fidelity against Digitize ground truth connectivity on 100 test sheets. Only solid process-connection edges are included. The symbol row scores graph nodes after class-aware matching inside the graph pipeline; the stitched detector evaluated under the standalone detection protocol scores

F_{1}

0.983 (Table 5 ).

Layer / rule	Exact $P$	Exact $R$	Exact $F_{1}$	Conn. cons.
Process connections	0.677	0.820	0.742	0.958
Signal (pred. fixed $j{=}10$ )	0.828	0.483	0.610	0.646
Signal (pred. adaptive)	0.808	0.840	0.824	0.869
Typed complete (adaptive signal)	0.686	0.822	0.748	—

Table 3: Process graph and optional signal-edge layer (100 sheets). Process:

G_{\mathrm{auto}}

vs. Digitize ground truth connectivity; signal: predicted vs. fixed geometric gold.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Unified perception-to-graph pipeline. Learned outputs stay in sheet coordinates and enter only at their grounding ports. The solid construction path is deterministic: no learned module can invent an edge alone.

Panel of Figure 5 — Figure 5: First-symbol linker walk. Traversal passes through empty scaffold junctions and emits an edge only when the first occupied junction on the selected continuation is reached. Spatial nearness or bounding-box overlap alone never creates a graph edge.

Panel of Figure 6 — Figure 6: Intersection noding inserts an exact shared coordinate at a true tee intersection. Crossings classified as fly-overs, and crossings without a nearby terminating tip, remain separate so that the recovered topology is not artificially connected.

Panel of Figure 7 — Figure 7: Adaptive snapping merges endpoints within the sheet-dependent tolerance $j_{\mathrm{eff}}=\min\!\bigl(\alpha_{\mathrm{cap}}\ell,\,\max(j_{\min},\,g+\alpha_{\mathrm{slack}}\ell)\bigr)$ . The capped tolerance limits both under-merging and over-merging.

Model	Mode	$P$	$R$	mAP50	mAP50–95
Ours (YOLO11s)	tiles	0.992	0.982	0.994	0.987
YOLOv10n	tiles	0.989	0.979	0.994	0.983
RT-DETR-L	letterbox	0.838	0.845	0.883	0.677
YOLO11s	letterbox	0.784	0.783	0.824	0.613
YOLOv10n	letterbox	0.291	0.488	0.281	0.180

Appendix

Table 4: Symbol detector zoo (Digitize-PID test set, 100 sheets).

NMS IoU	$P$	$R$	Det. $F_{1}$
0.50	0.810	1.000	0.895
0.20	0.925	0.999	0.961
0.10	0.969	0.998	0.983

Appendix

Table 5: Ours (YOLO11s): full-sheet stitch (tiles, 20% overlap, test set).

P

R

, and Det.

F_{1}

follow Eqs. ( 1 ) after greedy box matching at IoU

\geq 0.5

(Appendix A.2 ).

Backbone	Dice μ	IoU μ	Dice sol	Dice dash	IoU sol	IoU dash
Ours (EffNet-B0)	0.982	0.966	0.983	0.971	0.968	0.946
MobileNetV2	0.980	0.961	0.981	0.969	0.963	0.942

Appendix

Table 6: Line U-Net pixel scores (test set, 100 sheets).

Setting	$P$	$R$	Det. $F_{1}$	Exact	CER $\downarrow$
YOLO-text (det.)	0.712	0.999	0.831	—	—
+ pretrained TrOCR (GT crops)	—	—	—	0.857	0.031
Finetuned TrOCR (GT crops)	—	—	—	0.995	0.001
Ours (YOLO-text + finetuned TrOCR)	0.772	0.906	0.833	—	—

Appendix

Table 7: Text detection and OCR (test set, 100 sheets).

	Qwen3-VL-4B		Qwen3-VL-8B		Gemma-4-E4B
Level	Img.	Agent	Img.	Agent	Img.	Agent
Easy	49.0	10.0	21.0	9.7	33.3	9.0
	$\downarrow\,39.0\ \mathrm{pp}$		$\downarrow\,11.3\ \mathrm{pp}$		$\downarrow\,24.3\ \mathrm{pp}$
Medium	66.7	0.0	76.5	0.0	100.0	9.8
	$\downarrow\,66.7\ \mathrm{pp}$		$\downarrow\,76.5\ \mathrm{pp}$		$\downarrow\,90.2\ \mathrm{pp}$
Hard	5.1	34.1	70.6	40.8	52.2	42.0

Appendix

Table 8: Fabricated-true rate (%) on topology-related booleans with gold false . Lower is better. The centered band under each Img.–Agent pair is the absolute change:

\downarrow

is a beneficial reduction,

\uparrow

a deterioration. Grounding reduces pooled overclaim for all three models.

Setting	Exact edge $F_{1}$
P&ID Graph Recovery Stack (Ours)	0.742
Without intersection noding	0.729
Fixed $j{=}10$ snap on predicted solids (no adaptive)	0.193
Predicted boxes + GT lines	0.777
GT boxes + GT lines (topology)	1.000
Signal: pred. fixed $j{=}10$	0.610

Appendix

Table 9: Selected graph-construction ablations (existing measurements). Exact edge

F_{1}

on recovered process graphs unless noted.

Stage	Device	s/sheet	100-sheet total
Symbol stitch (YOLO11s)	2 $\times$ 16 GB GPU	1.2	$\sim$ 2 min
Line U-Net (EffNet-B0)	2 $\times$ 16 GB GPU	4.6	$\sim$ 8 min
Text det. + OCR	2 $\times$ 16 GB GPU	22.7	$\sim$ 38 min
Vectorize + graph + tags	CPU	$\sim$ 4	minutes
Oracle re-score	CPU	0.03	$\sim$ 3 s

Appendix

Table 10: P&ID Graph Recovery Stack (Ours): wall-clock on 100 test sheets.

Task	System	Primary score
OCR	Ours (YOLO-text + TrOCR)	Joint $F_{1}$ 0.810
OCR	Azure Read	Joint $F_{1}$ 0.779
OCR	AWS Textract	Joint $F_{1}$ 0.663
OCR	GCP Vision	Joint $F_{1}$ 0.151
Symbols	Ours (YOLO11s tiles)	Class-agnostic det. $F_{1}$ 0.961
Symbols	Azure / GCP / AWS	Nil / Nil / 0.001

Appendix

Table 11: Cloud and zero-shot controls on 100 test sheets. The table gives the primary metric for each task.

System	Det. $F_{1}$	Joint $F_{1}$	Exact	CER	Pred./sheet
Ours (YOLO-text + TrOCR)	0.833	0.810	0.971	0.008	183.3
Azure Read	0.818	0.779	0.954	0.021	163.1
AWS Textract	0.824	0.663	0.807	0.142	162.5
GCP Vision	0.174	0.151	0.872	0.055	385.2

Appendix

Table 12: OCR on diagram crop (100 sheets; mean 156.1 GT words per crop).

System	Det. $P$	Det. $R$	Det. $F_{1}$	Joint $P$	Joint $R$	Joint $F_{1}$
Ours (YOLO-text + TrOCR)	0.772	0.906	0.833	0.750	0.880	0.810
Azure Read	0.801	0.836	0.818	0.762	0.796	0.779
AWS Textract	0.808	0.841	0.824	0.650	0.677	0.663
GCP Vision	0.122	0.302	0.174	0.106	0.261	0.151

Appendix

Table 13: OCR precision and recall (diagram crop).

System	Det. $P$	Det. $R$	Det. $F_{1}$	Class-aware $F_{1}$	Pred./sheet
Ours (YOLO11s tiles)	0.925	0.999	0.961	0.961	127.76
Azure objects	Nil	Nil	Nil	Nil	Nil
GCP Object Localization	Nil	Nil	Nil	Nil	Nil
AWS Rekognition	0.115	0.001	0.001	Nil	0.61

Appendix

Table 14: Class-agnostic symbol boxes (diagram crop; mean 118.4 GT symbols per sheet).

System	Dice micro	IoU micro	Dice mean	$n$
Ours (U-Net EffNet-B0)	0.9823	0.9658	0.9823	100
SAM 2 AMG (union)	0.0121	0.0061	0.0098	100
SAM 2 AMG (elongated)	0.0005	0.0003	0.0006	100
DeepLabV3 (COCO)	Nil	Nil	Nil	100
DeepLabV3+ (VOC)	0.0009	0.0004	0.0003	100

Appendix

Table 15: Line-mask Dice vs. thickness-2 GT polylines (full sheets). Zero-shot rows: long side 1024 px then resize; ours: native-resolution U-Net tiles.

Panel of Figure 8 — Figure 8: Graph-grounded harness and planner loop. Question entities are grounded once on $G_{\mathrm{auto}}$ ; the planner composes operators over returned facts. Missing evidence starts another turn (cap six). Unsupported answers are pushed back by the answer contract. FINAL requires at least one operator call plus a passing evidence check.

Group	Tool	Arguments	Behavior
Nodes	node_info	{id}	Class name, OCR tag, degree, and sheet coordinates for one node.
	neighbors	{id}	One-hop neighbor ids with class, tag, degree, and coordinates.
Search	find_nodes	{attr, match, value}	All node ids whose class or tag matches value ( equals or case-insensitive prefix ), with count and a compact attribute table.
Paths	bfs	{id, ids?, exclude_ids?, max_hops?}	Hop distances from id ; optional ids restricts the returned map. Unreachable targets receive null .
	shortest_path	{a, b, exclude_ids?}	Shortest node path a..b (inclusive), or null if none. Optional exclude_ids are not traversed.
	edge_exists	{a, b}	Boolean: true iff an undirected process-connection edge is present in $G_{\mathrm{auto}}$ .

Appendix

Table 16: Graph-Grounded Harness (Ours) tool library: seven fact-returning primitives on

G_{\mathrm{auto}}

(arguments and return semantics).

Family	Ans.	Example
connected_process	bool	Are marks A and B directly connected by a process line?
connection_exists_tag	bool	Is symbol MN-46505 directly connected to symbol RO-10 181?
tag_connected_to_mark	bool	Is symbol CS-25 directly connected to mark A?
share_neighbor	bool	Do marks A and B share a common directly connected neighbor?
has_same_class_neighbor	bool	Does mark A connect to a same-type neighbor?
is_branch	bool	Is mark A a branch point ( $\geq 3$ direct connections)?

Appendix

Table 17: Easy TopoPID-VQA question families.

Family	Ans.	Example
reachable_within_k	bool	Is B reachable from A in $\leq 3$ connections?
same_network	bool	Are A and B on the same connected network?
same_type_neighbors_at_least_n	bool	Does A have $\geq 1$ same-type neighbor?
exists_type_within_k	bool	Within 2 hops of A, any symbol of B’s type?
shared_junction_within_k	bool	A symbol within 2 hops of both A and B?
route_via_tag_family	bool	Route A $\to$ B in $\leq 4$ hops through a tag family?

Appendix

Table 18: Medium TopoPID-VQA question families.

Family	Ans.	Example
more_options_near_a	bool	More named symbols within 3 hops of A than of B?
path_avoiding_type	bool	Can A reach B while bypassing C’s component class?
counterfactual_reconnect	bool	If A were removed, would B still connect to C?
route_comparison	bool	Is A closer to C than B is (hop count)?
multi_constraint_option	tag	Tag near A and same type as B (4 options)?
all_prefix_in_network	bool	All AB-* tags on same network as A?

Appendix

Table 19: Hard TopoPID-VQA question families.

Level	Boolean	Integer	String	Total
Easy	600	400	0	1000
Medium	699	100	201	1000
Hard	699	101	200	1000
Pooled	1998	601	401	3000

Appendix

Table 20: TopoPID-VQA answer-type counts (exact-match scoring).

Split	Majority (by answer type)	Random (by answer type)
Easy	64.8	35.0
Medium	42.4	37.2
Hard	39.7	37.2
Pooled	48.1	35.6

Appendix

Table 21: Chance baselines (% exact match). Majority always predicts the most frequent gold label within each answer type, then micro-averages. Random draws uniformly from the observed label set of that answer type (expected accuracy

1/|\mathcal{Y}_{\mathrm{type}}|

Model	System	Bool	Int	Str
Qwen3-VL-4B	Image-only	50.3	9.7	34.9
	Graph-Grounded Harness (Ours)	75.3	84.7	66.8
	Gold-query ceiling	89.9	92.0	79.1
Qwen3-VL-8B	Image-only	51.2	12.6	35.2
	Graph-Grounded Harness (Ours)	75.7	82.2	61.1
	Gold-query ceiling	89.9	92.0	79.1

Appendix

Table 22: Pooled TopoPID-VQA exact-match accuracy (%) by answer type. Image-only vs. Graph-Grounded Harness (Ours); gold-query ceiling uses gold entities and query structure on

G_{\mathrm{auto}}

Level	System	Bool	Int	Str
Easy	Image-only	52.0	4.2	—
	Graph-Grounded Harness (Ours)	87.0	83.8	—
	Gold-query ceiling	90.7	89.2	—
Medium	Image-only	48.8	25.0	39.3
	Graph-Grounded Harness (Ours)	77.1	91.0	77.1
	Gold-query ceiling	92.0	100.0	82.1

Appendix

Table 23: Per-level exact-match (%) by answer type for Qwen3-VL-4B-Instruct. Easy has no string items.

	Qwen3-VL-4B			Qwen3-VL-8B			Gemma-4-E4B
Level	$P$	$R$	$F_{1}$	$P$	$R$	$F_{1}$	$P$	$R$	$F_{1}$
Image-only
Easy	52.0	53.0	52.5	57.7	28.7	38.3	53.9	39.0	45.3
Medium	48.7	66.2	56.1	50.0	69.1	58.0	50.9	69.9	59.0
Hard	51.9	4.0	7.5	49.4	62.1	55.0	50.8	52.9	51.8
Pooled	50.1	40.4	44.7	50.8	54.4	52.6	51.5	54.6	53.0

Appendix

Table 24: TopoPID-VQA boolean subset for Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, and Gemma-4-E4B-it:

P

R

F_{1}

(%, positive class true ). Boolean counts: easy 600; medium 699; hard 699; pooled 1998.

Level	Category	Image-only	Graph-Grounded Harness (Ours)
Easy
	simple_counting	4.2	83.8
	spatial_connections	52.3	90.7
	structural	51.7	83.3
Medium
	reachability	50.5	91.1

Appendix

Table 25: TopoPID-VQA exact-match accuracy by category (%) for Qwen3-VL-4B-Instruct.

Level	Category	Image-only	Graph-Grounded Harness (Ours)
Easy
	simple_counting	5.8	75.0
	spatial_connections	54.7	91.3
	structural	53.0	85.7
Medium
	reachability	44.1	81.7

Appendix

Table 26: TopoPID-VQA exact-match accuracy by category (%) for Qwen3-VL-8B-Instruct.

Level	Category	Image-only	Graph-Grounded Harness (Ours)
Easy
	simple_counting	5.9	80.5
	spatial_connections	53.8	88.0
	structural	52.4	80.0
Medium
	reachability	49.5	95.0

Appendix

Table 27: TopoPID-VQA exact-match accuracy by category (%) for Gemma-4-E4B-it.

Panel of Figure 9 — Figure 9: Raw Digitize-PID sheets (downscaled for display).

Panel of Figure 10 — Figure 10: Perception stack: solid/dashed masks, vectorized segments, symbol boxes.

Panel of Figure 11 — Figure 11: Vectorized line geometry (solid and dashed polylines).

Panel of Figure 12 — Figure 12: Recovered process graph: process connections on $G_{\mathrm{auto}}$ .

Panel of Figure 13 — Figure 13: Dual-layer graph: process connections (solid) and signal edges (dashed).

Panel of Figure 14 — Figure 14: Dashed-ink zoom: masks and vectorized dashed segments.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDs
Mar 23, 2026Achmad Anggawirya Alimin, Artur M. SchweidtmannGraph Retrieval-Augmented Generation Agentic RAG
Process Intelligence Research Group, Department of Chemical Engineering, Delft University of Technology
SynthPID: P&ID digitization from Topology-Preserving Synthetic Data
Apr 15, 2026Suraj Prasad, Pinak MahapatraSynthetic Data Generation
Indian Institute Of Technology Bombay
LLMs in Process Diagram Engineering: From Optimal PFDs to Validated P&IDs
Jul 21, 2026Timur Zakarin, Sergei Voitov, Sergei Shumilin +1Large Language Model-Guided Optimization Workflow Automation
Skoltech, AI Center, Moscow, Russia · AIRI, Moscow, Russia

Mar 23, 2026J/K move · Enter open · S save

GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDs

Achmad Anggawirya Alimin, Artur M. Schweidtmann

Process Intelligence Research Group, Department of Chemical Engineering, Delft University of Technology

Graph Retrieval-Augmented Generation Agentic RAG Knowledge Graph Question Answering

Open paper AskPDF arXiv

Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG) and knowledge graphs offer new opportunities for interacting with engineering diagrams such as Piping and Instrumentation Diagrams (P&IDs). However, directly processing raw images or smart P&ID files with LLMs is often costly, inefficient, and prone to hallucinations. This work introduces ChatP&ID, an agentic framework that enables grounded and cost-effective natural-language interaction with P&IDs using Graph Retrieval-Augmented Generation (GraphRAG), a paradigm we refer to as GraphRAG for engineering diagrams. Smart P&IDs encoded in the DEXPI standard are transformed into structured knowledge graphs, which serve as the basis for graph-based retrieval and reasoning by LLM agents. This approach enables reliable querying of engineering diagrams while significantly reducing computational cost. Benchmarking across commercial LLM APIs (OpenAI, Anthropic) demonstrates that graph-based representations improve accuracy by 18% over raw image inputs and reduce token costs by 85% compared to directly ingesting smart P&ID files. While small open-source models still struggle to interpret knowledge graph formats and structured engineering data, integrating them with VectorRAG and PathRAG improves response accuracy by up to 40%. Notably, GPT-5-mini combined with ContextRAG achieves 91% accuracy at a cost of only $0.004 per task. The resulting ChatP&ID interface enables intuitive natural-language interaction with complex engineering diagrams and lays the groundwork for AI-assisted process engineering tasks such as Hazard and Operability Studies (HAZOP) and multi-agent analysis.

Abstract

VLM Reasoning

Explore similar work

GraphRAG for Engineering Diagrams: ChatP&ID Enables LLM Interaction with P&IDs

SynthPID: P&ID digitization from Topology-Preserving Synthetic Data

LLMs in Process Diagram Engineering: From Optimal PFDs to Validated P&IDs