cs.CVAug 18, 2026

Training-Free Agentic Computer Vision for Structural Component Detection in 2D Structural Framing Plans

Authors: Mohammad Talebi-Kalaleh, Qipei Mei

Organizations: Department of Civil and Environmental Engineering University of Alberta

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Converting structural framing plans into editable finite-element model drafts is labor-intensive and susceptible to transcription errors. Existing building-component recognition systems generally depend on task-specific neural detectors, whereas language-model agents in structural engineering typically operate on text or model data rather than on drawings. To the authors' knowledge, this work is the first to apply an agentic vision-language layer to structural-component detection and model drafting from framing-plan PDFs without task-specific detector training or fine-tuning. A deterministic stage extracts geometric primitives, estimates scale by dimension-ratio consensus, recognizes five entity classes using an explicit drafting grammar, and assembles an editable layout. The agentic stage constrains typed corrections through deterministic candidates, operation-specific admission tests, change-level review, and fail-closed transactions. Evaluation used an author-generated benchmark of 100 plans, divided equally between a development half used for all rule revisions and a seed-disjoint held-out half generated after the rules were frozen and evaluated once. All scores are end-to-end results for the complete framework on the held-out half. Scale estimates were within 0.1% of the generator reference for every drawing. Recall and precision were 0.922/0.997 for columns, 0.886/0.990 for beams, 1.000/1.000 for walls, 1.000/1.000 for braces, and 1.000/0.964 for openings. A controlled study repeated two corruptions three times on three development drawings. Calibration passed all nine trials, whereas member repair satisfied every strict end-state criterion in five of nine trials. Because both benchmark halves share a generator, the evaluation does not address independently drafted plans, raster input, analytical connectivity, or solver validation.

Figures & tables

Figure 1: System architecture with the deterministic detection layer, the guarded agentic refinement layer, and the model builder.
RuleStatement
Hatched-band wallThe rule classifies a qualifying diagonal-stroke corridor between adjacent columns as a shear wall; thickness is twice d(⌊0.9n⌋+1)d_{(\lfloor 0.9n\rfloor+1)} , and the raw value must reach 100 mm before capping and rounding.
X-in-rectangle openingThe rule classifies two long diagonals with coincident midpoints whose endpoints form an inked rectangle as a framed floor opening.
X-mark eraserUnder the supported grammar, two short diagonals (4–22 mm on paper) crossing at a shared midpoint are treated as a symbol and removed before member chaining.
Angle-gated dedupThe duplicate test for centerlines measures offset to an infinite line and therefore requires parallelism within 12∘12^{\circ} ; without the gate, joists whose midpoints lie on a perpendicular girder’s line are deleted.
Dimension-line exclusionSegments riding a dimension line already matched by the scale resolver are treated as annotation rather than members.
Void ruleCandidates whose midpoints fall strictly inside a detected opening region are removed as opening-symbol remnants.
Table 1: Eleven detection rules in the supported drafting grammar.
ConstantValueRole
Scale resolution
Text-to-line offset, angle12 mm paper, 6∘6^{\circ}Pairing a dimension string with its line
Text parameter window[0.05,0.95][0.05,0.95]Text must project into the line’s middle
Terminatortick [0.8,6][0.8,6] mm at >20∘>20^{\circ} , or arrowhead within 1.6 mmBoth ends required
Outlier bandmax⁡(0.01s~, 4 MAD)\max(0.01\tilde{s},\ 4\,\mathrm{MAD})Minimum band for near-zero MAD
Reference-scale snap≤1.5%\leq 1.5\% relative15 configured scales
Table 2: Deterministic-layer configuration. Paper millimeters govern tests of printed symbols, whereas model-space millimeters and meters govern structural tests.
Figure 2: Illustrative scanned-plan layouts with assumed lattice framing in panels (a)–(c) and an independent six-story model draft in panel (d).
ConstantValueRole
Overlay rendering
Composition width1600 px, ≤8\leq 8 px/mmDrawing legible without oversized payloads
Halo width, opacity2.8×2.8\times , α=0.55\alpha=0.55Drawn ink stays black inside a colored halo
Content crop margin42 mm paperKeeps grid bubbles and two dimension chains
Model invocation
Provider modelgemini-3.5-flash [ 44 ]Pinned pretrained model used in the reported runs
Table 3: Agentic-layer configuration in paper millimeters and model-space meters.
Figure 3: Four PD-50 development drawings from the core-l, skew-wing, braced, and mixed families.
FamilyDesigned to test
core-lL-, T-, and C-shaped shear-wall cores and perimeter panels on regular grids: wall symbols, walls replacing beams, and variable-thickness wall geometry
core-uTwin U-, box-, and C-cores with variable bay counts: multi-core layouts and core-adjacent framing
skew-wingA wing rotated 1414 – 28∘28^{\circ} with sloped connector beams and rotated dimension chains: rotated lattices and skew-aware scale
chamferChamfered corners producing diagonal edge beams and trimmed slabs
atriumA multi-bay central void with corner diagonals: large-opening semantics and building-scale X symbols
bracedUp to six X-braced perimeter bays: brace signs against beam marks on shared edges
Table 4: Ten PD-50 families and the capabilities they test.
Figure 4: Coverage-based beam scoring. Black denotes a ground-truth span, blue denotes supporting normalized detections, and red denotes an off-axis false detection.
Figure 5: Per-class recall and precision of the complete framework on the held-out PD-50-T half.
ClassRecallPrecisionRecall hits / ground truthPrecision hits / detections
Columns0.9220.997998 / 1,082998 / 1,001
Beams0.8860.9902,412 / 2,7212,390 / 2,413
Walls1.0001.000194 / 194194 / 194
Braces1.0001.00047 / 4747 / 47
Openings1.0000.96480 / 8080 / 83
Table 5: End-to-end entity recovery on the held-out PD-50-T half, with separate beam hit counts for recall and precision.
Figure 6: Representative end-to-end overlays for four held-out PD-50-T drawings.
 ColumnsBeamsWallsBracesOpenings
FamilyRPRPRPRPRP
atrium0.9691.0000.9390.9931.0001.000––1.0000.769
braced1.0001.0000.8930.9951.0001.0001.0001.0001.0001.000
chamfer1.0001.0000.8731.0001.0001.000––1.0001.000
core-l1.0001.0000.9821.0001.0001.000––1.0001.000
core-u1.0001.0000.9711.0001.0001.000––1.0001.000
Table 6: Per-family end-to-end recall and precision on the held-out half; “–” denotes an undefined metric because its denominator is zero.
  ColumnsBeamsWallsBracesOpenings
CutnnRPRPRPRPRP
US250.9330.9940.9030.9871.0001.0001.0001.0001.0001.000
CA250.9121.0000.8690.9941.0001.0001.0001.0001.0000.930
steel170.9261.0000.8830.9861.0001.0001.0001.0001.0001.000
concrete170.9171.0000.8800.9951.0001.0001.0001.0001.0001.000
timber160.9240.9910.8970.9911.0001.0001.0001.0001.0000.897
Table 7: Held-out end-to-end results by notation variant and material; nn is the number of drawings and “–” denotes an undefined metric.
DrawingMember repairComplete entity recoveryCalibration
st-us-010/30/33/3
co-us-023/33/33/3
co-ca-032/33/33/3
Aggregate5/96/99/9
Table 8: Strict outcomes for three repeated controlled-corruption trials per drawing.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

CaseFamilyNotationMaterialScaleColsBeamsWallsBracesOpen.
atrium-06atriumUStimber3/16 ′′24/2554/562/20/02/2
atrium-07atriumCAsteel1:7524/2553/562/20/02/2
atrium-08atriumUSconcrete3/16 ′′24/2553/562/20/02/2
atrium-09atriumCAtimber1:7524/2549/562/20/02/2
atrium-10atriumUSsteel3/16 ′′30/3066/692/20/02/2
braced-06bracedCAconcrete1:7520/2047/534/46/61/1
Appendix
Table 9: Held-out PD-50-T inventory and per-case end-to-end recall counts (matched/ground truth).
Figure 7: Detection overlays, held-out cases 1–25 of PD-50-T.
Figure 8: Detection overlays, held-out cases 26–50 of PD-50-T.
More in this topic

Structural Engineering

See all 21 papers

Explore similar work

CardsList
Aug 26, 2026

PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the geometry and layout essential for interpreting these plans. We present a Visual-First Multimodal Retrieval-Augmented Generation (RAG) framework called PlanSightRAG. It indexes and reasons directly over plan imagery, integrates a ColNomic-3B multi-vector retrieval, an agentic Planner-Retriever-Auditor-Synthesizer, and MaxSim heatmaps as an evidence trail. We introduce a 4,056-pair benchmark from five state Departments of Transportation (DOT) standard plans (1,898 pages). PlanSightRAG achieves 91.47% Recall@5 on zero-shot retrieval, while on a held-out Michigan DOT corpus, it achieves 91.40%. On synthetic, parametrically-generated compliance drawings, our Qwen2.5-VL-72B pipeline reaches 100% verdict accuracy only when supplied a pre-resolved rule threshold, a controlled ceiling that a non-VLM OCR baseline already reaches at 76.4%. Finally, we demonstrate autonomous visual rule-grounding by extracting numeric limits directly from a specification corpus without any human-supplied rules.

Nabaraj Subedi, Shuvo Dip Datta, Ahmed Abdelaty +1

Department of Electrical Engineering & Computer Science, University of Wyoming · Department of Civil & Architectural Engineering & Construction Management, University of Wyoming, 1000 E. University Ave., Laramie, 82071, WY, USA

Multimodal RAGDocument UnderstandingAgentic RAGVisual Document RetrievalDocument Image Understanding
AskPDF
Jun 2, 2026

Agentic Large Language Models for Automated Structural Analysis of 3D Frame Systems

Large language models (LLMs) have emerged as powerful foundation models with strong reasoning capabilities across domains. Beyond reactive text generation, agentic LLMs enable autonomous workflow execution through modular task decomposition and coordinated tool use. In structural engineering, recent efforts have developed agentic LLMs for automated analysis of plane frames. However, their extension to 3D frames remains underexplored due to challenges in irregular geometric representation, topological consistency, and long-horizon reasoning. This paper proposes an agentic LLM framework for automated structural analysis of 3D frames from natural language inputs. Irregular 3D frames are represented by projection onto a 2D plan, where orthogonal gridlines define spatial coordinates and a matrix of number of stories encodes vertical extrusion of each grid cell. Building on this representation, the framework establishes a multi-agent pipeline: a problem analysis agent parses input into structured JSON; a floor decomposition agent derives the spatial layout of each floor; the 3D geometry is assembled by node, girder, slab, and column agents; support and load agents assign boundary and loading conditions, and code translation agents generate executable SAP2000 script. Evaluated on ten representative 3D frames, the proposed framework achieves an average accuracy of 90% across repeated trials, demonstrating consistent and reliable performance.

Ziheng Geng, Ian Franklin, Santiago Martinez +3

Department of Civil and Architectural Engineering, University of Miami, Coral Gables, FL 33146, USA · School of Architecture, University of Miami, Coral Gables, FL 33146, USA · HBC Engineering Company, Doral, FL 33178, USA +1

Structural MechanicsStructural EngineeringMulti-Agent LLMs
AskPDF
Sep 7, 2026

BlueprintAgent: Constraint-Triggered Targeted Revisits for Simulation-Ready Generation from Scanned Structural Blueprints

Converting in-service reinforced-concrete (RC) building blueprints into simulation-ready models---structured frame representations that support deterministic FEM export and qualified-engineer review---underpins safety assessment and seismic retrofit, but the process remains manual. Direct prompting of a multimodal large language model (MLLM) over a scanned sheet is unreliable: outputs often violate engineering constraints on beam--column support, span count, or 3D continuity. We present BlueprintAgent (BPA), a constraint-triggered multimodal agent for simulation-ready frame extraction from scanned blueprints. BPA treats the MLLM as the primary reader and decision maker, with OCR and computer vision supplying localized evidence. Its central mechanism realizes engineering constraints as callable validators whose entity-level conflict reports trigger targeted MLLM revisits over the local region---an inference-time control distinct from fixed pipelines and free-form self-reflection. We evaluate BPA on 300 real scanned blueprint sheets from 20 anonymized RC frame projects, against five baselines and six ablations. BPA reaches a macro-averaged Beam F1 of 0.994, against 0.301 for single-MLLM zero-shot and 0.820 for a fixed pipeline; removing MLLM-led axis adjudication collapses Beam and Column F1 on complex multi-sheet projects. For dense technical drawings, engineering constraints are best deployed as triggers for entity-level targeted revisits rather than as post-hoc output filters.

Zhouyuan Xu, Chen Yang, Linhao Wang +2

Tsinghua University, Beijing 100084, China

Document UnderstandingConstrained Generative ModelingMultimodal AgentsStructural Engineering
AskPDF