Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprints

Bruno Chicelli; Henrique Alves; Rodrigo Anselmo; Joshua Weinberg; Felipe Lemos; Jan Baryla

cs.CLAug 15, 2026

Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprints

Authors: Bruno Chicelli, Henrique Alves, Rodrigo Anselmo, Joshua Weinberg, Felipe Lemos, Jan Baryla

Organizations: Handoff AI Research

Abstract

Converting a set of architectural blueprints into a complete material quantity takeoff requires visual perception across drawing sheets, dimensional and multi-hop reasoning, and grounding in construction conventions that the drawings never state. We present Handoff-H1, a takeoff system built from three layers: purpose-built computer-vision models that extract primitives; tool-using agents equipped with image operations and in-house visual-task tools, including CV-model-backed counting, detection and plan decomposition; and a persistent, hierarchically structured project foundation, grounded in a curated construction knowledge base. We evaluate on the Construction Blueprint Takeoff Benchmark: 10 real residential blueprint sets paired with consensus-validated expert takeoffs - 2,009 verified line items, restricted for scoring to the 1,348 primary-tier materials that drive an estimate - scored per trade by an LLM judge on material coverage and quantity Precision@25% ([email protected]) and combined into a weighted composite. Under identical scoring from the raw PDF, seven frontier and open-weight models span composites of 35-61, and independent professional estimators - scored against the same reconciled gold standard - post 77.6% (65.5% coverage, 87.9% [email protected]). Handoff-H1, working end-to-end from the raw PDF, reaches 81.6% (86.1% coverage, 78.8% [email protected]): roughly 20 points above the strongest frontier agent, and above the independent estimators by pairing near-human quantity precision with coverage they do not reach. The evaluation harness is public for the open harbor framework; the blueprint sets and ground truth are available upon request for research use.

Figures & tables

Figure 1: The problem in one sheet. A dimension floor plan from a residential drawing set (title block removed), the three demands it makes of its reader, and four of the line items it must yield (quantities from Handoff-H1’s takeoff of this set). A set like this one produces on the order of a hundred primary line items across nine trades, and none of the quantities on the right is written anywhere on the left.

Figure 2: A typical foundation detail—a spread-base sonotube footing, drawn at its own scale ( $\nicefrac{{3}}{{4}}^{\prime\prime}=1^{\prime}$ - $0^{\prime\prime}$ ). A single callout block fans out into quantities across three scopes, and none of them appears as a number anywhere on the sheet. Reading the words is not enough; a takeoff must attach each one to the right trade and assembly.

Panel of Figure 3 — Figure 3: The Handoff-H1 architecture at the level this report discusses it. Model identities, prompts, and orchestration specifics are withheld.

Figure 4: Reading a floor plan. The perception layer recovers typed primitives from each sheet: rooms classified by function (living, kitchen, garage, porch, bathroom), and walls, doors, windows, and fixtures detected and labeled with provenance. Discrete-count elements like these sit at the easy end of the difficulty spectrum; continuous-measurement trades (Section 6 ) are substantially harder.

System	Composite	Coverage	[email protected]
Handoff-H1 (end-to-end)	81.6	86.1	78.8
Human estimators (independent)	77.6	65.5	87.9
claude-fable-5 (closed)	61.4	77.3	52.8
gpt-5.6 (closed)	58.6	72.3	51.3
claude-opus-4-8 (closed)	54.6	67.4	48.0
Kimi-K2.6 (open)	54.5	64.6	50.2

Table 1: Main comparison on TakeoffBench-v1 (all 10 sets). Every system works from the raw blueprint PDF; baselines additionally receive the visual-analysis equalizer where needed, and their scores are averaged over repeated runs. The independent estimators’ takeoffs are scored by the same judge against the same approved ground truth. Composite is defined by Equation 1 ; values are percentages, higher is better.

Figure 5: Top: composite score by system (Equation 1 ), Handoff-H1 highlighted. Bottom: the mean coverage and precision beneath each composite, systems in alphabetical order.

Trade	H1	Estimators	fable-5	gpt-5.6	opus-4-8	Kimi-K2.6	sonnet-4-6	GLM-5.2	DeepSeek-V4
Concrete	72	72	54	51	45	43	46	30	25
Framing	82	71	54	49	38	53	39	31	29
Roofing	79	69	64	60	62	55	56	34	32
Siding	65	82	51	48	42	50	49	38	26
Insulation	76	62	68	65	53	51	46	44	33
Drywall	85	61	52	54	56	57	45	33	29

Table 2: Per-trade composite score on TakeoffBench-v1 , pooled over all 10 sets (baselines also over repeated runs). Values are percentages; bold marks the per-trade leader. Handoff-H1 leads five of the nine trades; the independent estimators lead on Siding decisively and on Electrical and Windows & Doors by small margins; Concrete is tied; no frontier agent leads any trade.

Panel of Figure 6 — Figure 6: Per-trade composite as a radar: Handoff-H1 (solid), the independent estimators (dashed), and the strongest frontier agent, claude-fable-5 (dotted). The estimator profile collapses where scopes went unquantified (Drywall, Insulation) and dominates on Siding; the frontier agent sits inside both profiles on seven of the nine trades.

Figure 7: A roof plan with surfaces broken out by plane and pitch. Roofing is sold in squares, and the true area depends on a slope the plan view hides: the billed quantity must be recovered from pitch and framing information rather than read off the flat footprint. Continuous-measurement trades like this are where estimators legitimately disagree, and where both Handoff-H1 and the human ceiling fall furthest from a single gold number.

Tool-Using Vision-Language Agents

Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.

See all 139 papers

BlueprintAgent: Constraint-Triggered Targeted Revisits for Simulation-Ready Generation from Scanned Structural Blueprints
Sep 7, 2026Zhouyuan Xu, Chen Yang, Linhao Wang +2Document Understanding Constrained Generative Modeling
Tsinghua University, Beijing 100084, China
Training-Free Agentic Computer Vision for Structural Component Detection in 2D Structural Framing Plans
Aug 18, 2026Mohammad Talebi-Kalaleh, Qipei MeiStructural Engineering Document Image Understanding
Department of Civil and Environmental Engineering University of Alberta
AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction
Mar 31, 2026Harsh Mankodiya, Chase Gallik, Theodoros Galanos +1AI Agent Benchmarks Multimodal Agents

Sep 7, 2026J/K move · Enter open · S save

BlueprintAgent: Constraint-Triggered Targeted Revisits for Simulation-Ready Generation from Scanned Structural Blueprints

Zhouyuan Xu, Chen Yang, Linhao Wang, Jiansheng Fan+1

Tsinghua University, Beijing 100084, China

Document Understanding Constrained Generative Modeling Multimodal Agents Structural Engineering

Open paper AskPDF arXiv

Converting in-service reinforced-concrete (RC) building blueprints into simulation-ready models---structured frame representations that support deterministic FEM export and qualified-engineer review---underpins safety assessment and seismic retrofit, but the process remains manual. Direct prompting of a multimodal large language model (MLLM) over a scanned sheet is unreliable: outputs often violate engineering constraints on beam--column support, span count, or 3D continuity. We present BlueprintAgent (BPA), a constraint-triggered multimodal agent for simulation-ready frame extraction from scanned blueprints. BPA treats the MLLM as the primary reader and decision maker, with OCR and computer vision supplying localized evidence. Its central mechanism realizes engineering constraints as callable validators whose entity-level conflict reports trigger targeted MLLM revisits over the local region---an inference-time control distinct from fixed pipelines and free-form self-reflection. We evaluate BPA on 300 real scanned blueprint sheets from 20 anonymized RC frame projects, against five baselines and six ablations. BPA reaches a macro-averaged Beam F1 of 0.994, against 0.301 for single-MLLM zero-shot and 0.820 for a fixed pipeline; removing MLLM-led axis adjudication collapses Beam and Column F1 on complex multi-sheet projects. For dense technical drawings, engineering constraints are best deployed as triggers for entity-level targeted revisits rather than as post-hoc output filters.

Abstract

Tool-Using Vision-Language Agents

Explore similar work

BlueprintAgent: Constraint-Triggered Targeted Revisits for Simulation-Ready Generation from Scanned Structural Blueprints

Training-Free Agentic Computer Vision for Structural Component Detection in 2D Structural Framing Plans

AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction