Converting a set of architectural blueprints into a complete material quantity takeoff requires visual perception across drawing sheets, dimensional and multi-hop reasoning, and grounding in construction conventions that the drawings never state. We present Handoff-H1, a takeoff system built from three layers: purpose-built computer-vision models that extract primitives; tool-using agents equipped with image operations and in-house visual-task tools, including CV-model-backed counting, detection and plan decomposition; and a persistent, hierarchically structured project foundation, grounded in a curated construction knowledge base. We evaluate on the Construction Blueprint Takeoff Benchmark: 10 real residential blueprint sets paired with consensus-validated expert takeoffs - 2,009 verified line items, restricted for scoring to the 1,348 primary-tier materials that drive an estimate - scored per trade by an LLM judge on material coverage and quantity Precision@25% ([email protected]) and combined into a weighted composite. Under identical scoring from the raw PDF, seven frontier and open-weight models span composites of 35-61, and independent professional estimators - scored against the same reconciled gold standard - post 77.6% (65.5% coverage, 87.9% [email protected]). Handoff-H1, working end-to-end from the raw PDF, reaches 81.6% (86.1% coverage, 78.8% [email protected]): roughly 20 points above the strongest frontier agent, and above the independent estimators by pairing near-human quantity precision with coverage they do not reach. The evaluation harness is public for the open harbor framework; the blueprint sets and ground truth are available upon request for research use.
Figures & tables
Figure 1: The problem in one sheet. A dimension floor plan from a residential drawing set (title block removed), the three demands it makes of its reader, and four of the line items it must yield (quantities from Handoff-H1’s takeoff of this set). A set like this one produces on the order of a hundred primary line items across nine trades, and none of the quantities on the right is written anywhere on the left.
Figure 2: A typical foundation detail—a spread-base sonotube footing, drawn at its own scale ( \nicefrac34′′=1′ - 0′′ ). A single callout block fans out into quantities across three scopes, and none of them appears as a number anywhere on the sheet. Reading the words is not enough; a takeoff must attach each one to the right trade and assembly.
Figure 3: The Handoff-H1 architecture at the level this report discusses it. Model identities, prompts, and orchestration specifics are withheld.
Figure 4: Reading a floor plan. The perception layer recovers typed primitives from each sheet: rooms classified by function (living, kitchen, garage, porch, bathroom), and walls, doors, windows, and fixtures detected and labeled with provenance. Discrete-count elements like these sit at the easy end of the difficulty spectrum; continuous-measurement trades (Section 6 ) are substantially harder.
Table 1: Main comparison on TakeoffBench-v1 (all 10 sets). Every system works from the raw blueprint PDF; baselines additionally receive the visual-analysis equalizer where needed, and their scores are averaged over repeated runs. The independent estimators’ takeoffs are scored by the same judge against the same approved ground truth. Composite is defined by Equation 1 ; values are percentages, higher is better.
Figure 5: Top: composite score by system (Equation 1 ), Handoff-H1 highlighted. Bottom: the mean coverage and precision beneath each composite, systems in alphabetical order.
Trade
H1
Estimators
fable-5
gpt-5.6
opus-4-8
Kimi-K2.6
sonnet-4-6
GLM-5.2
DeepSeek-V4
Concrete
72
72
54
51
45
43
46
30
25
Framing
82
71
54
49
38
53
39
31
29
Roofing
79
69
64
60
62
55
56
34
32
Siding
65
82
51
48
42
50
49
38
26
Insulation
76
62
68
65
53
51
46
44
33
Drywall
85
61
52
54
56
57
45
33
29
Table 2: Per-trade composite score on TakeoffBench-v1 , pooled over all 10 sets (baselines also over repeated runs). Values are percentages; bold marks the per-trade leader. Handoff-H1 leads five of the nine trades; the independent estimators lead on Siding decisively and on Electrical and Windows & Doors by small margins; Concrete is tied; no frontier agent leads any trade.
Figure 6: Per-trade composite as a radar: Handoff-H1 (solid), the independent estimators (dashed), and the strongest frontier agent, claude-fable-5 (dotted). The estimator profile collapses where scopes went unquantified (Drywall, Insulation) and dominates on Siding; the frontier agent sits inside both profiles on seven of the nine trades.
Figure 7: A roof plan with surfaces broken out by plane and pitch. Roofing is sold in squares, and the true area depends on a slope the plan view hides: the billed quantity must be recovered from pitch and framing information rather than read off the flat footprint. Continuous-measurement trades like this are where estimators legitimately disagree, and where both Handoff-H1 and the human ceiling fall furthest from a single gold number.
Converting in-service reinforced-concrete (RC) building blueprints into simulation-ready models---structured frame representations that support deterministic FEM export and qualified-engineer review---underpins safety assessment and seismic retrofit, but the process remains manual. Direct prompting of a multimodal large language model (MLLM) over a scanned sheet is unreliable: outputs often violate engineering constraints on beam--column support, span count, or 3D continuity. We present BlueprintAgent (BPA), a constraint-triggered multimodal agent for simulation-ready frame extraction from scanned blueprints. BPA treats the MLLM as the primary reader and decision maker, with OCR and computer vision supplying localized evidence. Its central mechanism realizes engineering constraints as callable validators whose entity-level conflict reports trigger targeted MLLM revisits over the local region---an inference-time control distinct from fixed pipelines and free-form self-reflection. We evaluate BPA on 300 real scanned blueprint sheets from 20 anonymized RC frame projects, against five baselines and six ablations. BPA reaches a macro-averaged Beam F1 of 0.994, against 0.301 for single-MLLM zero-shot and 0.820 for a fixed pipeline; removing MLLM-led axis adjudication collapses Beam and Column F1 on complex multi-sheet projects. For dense technical drawings, engineering constraints are best deployed as triggers for entity-level targeted revisits rather than as post-hoc output filters.
Converting structural framing plans into editable finite-element model drafts is labor-intensive and susceptible to transcription errors. Existing building-component recognition systems generally depend on task-specific neural detectors, whereas language-model agents in structural engineering typically operate on text or model data rather than on drawings. To the authors' knowledge, this work is the first to apply an agentic vision-language layer to structural-component detection and model drafting from framing-plan PDFs without task-specific detector training or fine-tuning. A deterministic stage extracts geometric primitives, estimates scale by dimension-ratio consensus, recognizes five entity classes using an explicit drafting grammar, and assembles an editable layout. The agentic stage constrains typed corrections through deterministic candidates, operation-specific admission tests, change-level review, and fail-closed transactions. Evaluation used an author-generated benchmark of 100 plans, divided equally between a development half used for all rule revisions and a seed-disjoint held-out half generated after the rules were frozen and evaluated once. All scores are end-to-end results for the complete framework on the held-out half. Scale estimates were within 0.1% of the generator reference for every drawing. Recall and precision were 0.922/0.997 for columns, 0.886/0.990 for beams, 1.000/1.000 for walls, 1.000/1.000 for braces, and 1.000/0.964 for openings. A controlled study repeated two corruptions three times on three development drawings. Calibration passed all nine trials, whereas member repair satisfied every strict end-state criterion in five of nine trials. Because both benchmark halves share a generator, the evaluation does not address independently drafted plans, raster input, analytical connectivity, or solver validation.
Mohammad Talebi-Kalaleh, Qipei Mei
Department of Civil and Environmental Engineering University of Alberta
The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license.