MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft

Sewoong Lee; Risham Sidhu; Julia Hockenmaier; Yoonhwa Jung

cs.AIAug 28, 2026

MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft

Authors: Sewoong Lee, Risham Sidhu, Julia Hockenmaier, Yoonhwa Jung

Organizations: Siebel School of Computing and Data Science The Grainger College of Engineering University of Illinois Urbana-Champaign · Department of Civil and Coastal Engineering University of Florida

Abstract

We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to systematically evaluate the reliability and limitations of LLMs for construction tasks in Minecraft. The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, spanning 17 distinct task categories, providing a safe and controllable experimental environment for assessing LLMs' ability to perform realistic construction engineering tasks. With this benchmark, we conduct an in-depth evaluation of state-of-the-art LLMs and perform a detailed error analysis, revealing key failure modes and practical challenges in applying LLMs to construction engineering tasks.

Figures & tables

Figure 1: LLM-based agents fail to reliably execute even a simple instruction to build an arched bridge. We decompose the task into rule-based evaluation components. This benchmark reveals the limitations that prevent LLM-based agents from being reliably used in real-world construction scenarios.

Figure 2: Examples of LLM-generated constructions in MineCEraft tasks. In the embodied Minecraft environment, the LLM frequently produces structurally flawed outputs even for relatively simple instructions. All examples shown here were generated using claude-3-7-sonnet-20250219 with the agent configuration of White* et al. (2025) (Appendix E.2 ). Examples created by humans are provided in Appendix H .

Benchmark	Embodied Environment	Natural Lang. Requirements	Long-horizon Evaluation	Physics-aware	Construction Engineering
Voyager ( Wang et al., 2024 )	✓	✗	✓	✗	✗
EmbodiedBench ( Yang et al., 2025 )	✓	✓	✓	✓	✗
MineCollab ( White* et al., 2025 )	✓	✗	✓	✗	✗
CEQuest Wu et al. (2025)	✗	✓	✗	✗	✓
BAPv2 Jayannavar et al. (2026)	✓	✓	✓	✗	✗
MineCEraft (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of benchmarks.

Panel of Figure 3 — Figure 3: Evaluating arch-like structures beyond simple geometry. Prompt: “Build an arch bridge.” Physical plausibility is verified via breadth-first search, penalizing disconnected floating blocks (e.g., Figure 3(a) , generated by claude-3-7-sonnet-20250219 ). Structural stability is measured via von Mises stress. We use a standard flat bridge (Figure 3(b) , $\approx 7.8$ kPa) as the max threshold ( $\sigma_{\mathrm{max}}$ ). In structural engineering, a true arch distributes loads more efficiently (Figure 3(c) , $\approx 5.3$ kPa), yielding a lower maximum stress than a flat bridge of the same span. Any suboptimal arch exceeding the flat baseline stress ( $\sigma_{\max}>\sigma_{\mathrm{ref}}$ ) receives a proportionally penalized stability score.

Panel of Figure 4 — Figure 4: MineCEraft shows that the LLM-generated structures in (a) and (b) are structurally unstable compared to the alternative in (c). When the center column is attached to another column, the load is not distributed uniformly, leading to a biased stress field and a higher peak von Mises stress ( $\approx 140\,\mathrm{MPa}$ ) compared to the balanced five-column configuration ( $\approx 60\,\mathrm{MPa}$ ). The construction was generated using claude-3-7-sonnet-20250219 with the prompt: “Build a 10 $\times$ 10 flat roof supported by exactly five 2 $\times$ 2 $\times$ 2 columns. Arrange the columns so that the roof is as structurally stable as possible.”

Category	Subcategory	Models				Human
Category	Subcategory	llama-4 -17b-16e	claude-4-5 -sonnet	gpt-5 -mini	gemini-3 -pro
Accuracy	Material	27.0%	84.5%	92.7%	70.4%	97.8%
	Shape	22.2%	55.8%	53.4%	50.3%	99.2%
	Size	3.2%	68.1%	77.0%	52.8%	91.7%
Safety	Physical Plausibility	77.8%	85.0%	68.3%	92.0%	98.3%
Safety	Structural Stability	25.0%	57.1%	50.7%	42.7%	93.9%

Table 2: Performance on MineCEraft across different model families and sizes. Human results are reported on MineCEraft-Lite for reference (see Appendix E.1 ). Overall, LLMs underperform human participants across most evaluation categories. For Lite-only comparisons across all models, see Table 10 in Appendix F . For evaluation results on more models, see Table 11 in Appendix F .

	llama-4-17b-16e		claude-4-5-sonnet		gpt5-mini		gemini-3-pro
#Problems	#Correct	Overall Acc.	#Correct	Overall Acc.	#Correct	Overall Acc.	#Correct	Overall Acc.
723	3	0.4%	192	26.6%	286	39.6%	277	38.3%

Table 3: Overall problem-level accuracy across the 723 MineCEraft prompts. Scores are computed by requiring that all success/failure evaluation criteria for a problem be satisfied simultaneously. A more detailed instruction-level aggregation method is explained in Appendix D.1 . Even the strongest models achieve less than 40% overall accuracy. For comparison, human participants achieve 90% accuracy on MineCEraft-Lite.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Qualitative examples of building arch bridges in 2D . While LLMs can generate arch bridges when prompted with “Draw an arch bridge as an image.” , they often produce irregular or structurally incorrect shapes when asked in a text-based format, such as “Let’s build an arch bridge viewed from the side using stones (#) and empty spaces (-).” . By comparison, Gemini 3 can generate a relatively arch-like structure even in the text-based setting.

d(C_{i},C_{j})=\min_{\begin{subarray}{c}p\in C_{i}\\ q\in C_{j}\end{subarray}}\lVert p-q\rVert_{2}

Appendix

Algorithm 6 Minimum Inter-Cluster Span Check

		llama-4-17b-16e		claude-4-5-sonnet		gpt5-mini		gemini-3-pro
Metric	#Problems	#Correct	Accuracy	#Correct	Accuracy	#Correct	Accuracy	#Correct	Accuracy
Overall Accuracy ( $\mathrm{acc}\cap\mathrm{pp}\cap\mathrm{dep}$ )	723	3	0.4%	192	26.6%	286	39.6%	277	38.3%
Accuracy ( $\mathrm{material}\cap\mathrm{shape}\cap\mathrm{size}$ )	699	11	1.6%	197	28.2%	317	45.4%	266	38.1%
Material	455	129	28.4%	314	69.0%	410	90.1%	296	65.1%
Shape	563	11	2.0%	115	20.4%	213	37.8%	178	31.6%
Size	369	7	1.9%	145	39.3%	284	77.0%	195	52.8%

Appendix

Table 4: Problem-level aggregation of binary success criteria. A construction is counted as accurate only if it satisfies all applicable accuracy requirements for that problem simultaneously, i.e., material

\cap

shape

\cap

size. The stricter overall criterion additionally requires physical plausibility and dependency satisfaction, i.e., accuracy

\cap

physical plausibility

\cap

dependency. This evaluation is stricter than the instruction-level results in Table 2 : a single problem may contain multiple instructions (for multi-turn tasks) and multiple checks within the same category (e.g., several shape constraints), so success is counted only when all required conditions within the problem are satisfied together. The performance drop compared to Table 2 more clearly reveals that current LLMs still struggle to reliably execute construction tasks.

		llama-4-17b-16e			claude-4-5-sonnet			gpt5-mini			gemini-3-pro
Metric	#All	All	#Acc.	Acc.-only	All	#Acc.	Acc.-only	All	#Acc.	Acc.-only	All	#Acc.	Acc.-only
Eff. (L1)	112	47.1%	0	–	56.2%	94	54.9%	55.6%	110	55.1%	59.7%	100	54.8%
Stab. (stress)	21	25.0%	0	–	57.1%	5	54.1%	50.7%	7	59.7%	42.7%	10	80.7%

Appendix

Table 5: Graded metrics evaluated at the problem level. ‘#All’ denotes the number of applicable problems for each metric and is shared across models. For each model, ‘All’ reports the mean score across all applicable problems, ‘#Acc.’ is the number of problems whose binary accuracy constraints are fully satisfied, and ‘Acc.-only’ reports the mean score restricted to that accurate subset. Accuracy does not consistently improve efficiency, but accurate constructions sometimes exhibit modest gains in structural stability.

Prompt Type	llama -4 -scout	claude -4-5 -sonnet	gpt-5 -mini	gemini -3-pro
Baseline	54.8 %	58.3%	58.3%	58.3%
Step-by-step	52.4%	58.3%	58.3%	58.3%
$\Delta$ (vs. Baseline)	-2.4%	0.0%	0.0%	0.0%
Order-aware	52.4%	79.2%	79.2%	100.0%
$\Delta$ (vs. Baseline)	-2.4%	+20.9%	+20.9%	+41.7%

Appendix

Table 6: Construction efficiency (%, higher is better) for the foundation-building task under different prompting strategies. The baseline prompt contains only the task instruction. Step-by-step adds the generic reasoning cue “Let’s think step by step.” , whereas Order-aware adds the task-specific cue “Let’s think about the order in which we should place the blocks to optimize the movement path.” Rows labeled

\Delta

show the change in efficiency relative to the baseline. The generic reasoning cue does not improve efficiency, while the order-aware cue yields substantial gains for several models.

	Professional abb.	Plain words
Material	66.7	50.0
Shape	66.7	66.7
Physical Plausibility	100.0	100.0
Structural Stability	66.7	16.7
Dependency	66.7	100.0

Appendix

Table 7: Sensitivity to instruction wording for claude-4-5-sonnet evaluated on 72 construction tasks of roofs and columns in Section 3.2.1 . Top: comparison between a field-specific abbreviation (SOMD) and its plain-language equivalent (“slab on metal deck”) shows little difference in performance, suggesting that construction-specific terminology is reasonably well captured. Bottom: prompts differing only in the verb (“build”, “construct”, “create”) show measurable differences.

Model	System Prompt	Material	Shape	Size	P.P.	S.S.	Eff.	Dep.
llama-4-scout-17b-16e	Original	20.6	23.8	24.6	85.7	16.7	37.8	0.0
	w/o brevity constraint	33.3	29.5	16.7	83.3	0.0	84.2	20.0
	$\Delta$	+12.7	+5.7	-7.9	-2.4	-16.7	+46.4	+20.0
claude-4-5-sonnet	Original	84.1	60.5	83.3	83.3	55.6	54.9	60.0
	w/o brevity constraint	90.5	47.5	91.7	66.7	91.9	53.1	100.0
	$\Delta$	+6.4	-13.0	+8.4	-16.6	+36.3	-1.8	+40.0

Appendix

Table 8: Sensitivity to the brevity constraint in the system prompt. We compare the original MINDcraft-based system prompt with a variant that removes the instruction “Be very brief in your responses.” Values are percentages, and

\Delta

denotes the change relative to the original prompt.

Model	Material	Shape	Size	P.P.	S.S.	Eff.	Dep.
GPT-4	42.9%	50.8%	50.0%	70.0%	0.0%	53.1%	40.0%
GPT-4o	85.7%	55.9%	61.1%	84.4%	67.8%	62.0%	60.0%

Appendix

Table 9: Additional GPT-4 vs. GPT-4o comparison.

Category	Subcategory	Models				Human
Category	Subcategory	llama-4 -scout-17b-16e	claude-4-5 -sonnet	gpt-5-mini	gemini-3 -pro
Accuracy	Material	20.6% ${}_{\pm 7.2}^{(\approx 27.0)}$	84.1% ${}_{\pm 2.4}^{(\approx 84.5)}$	87.1% ${}_{\pm 7.1}^{(\approx 92.7)}$	71.4% ${}_{\pm 4.8}^{(\approx 70.4)}$	97.8% ±2.1
	Shape	23.8% ${}_{\pm 7.1}^{(\approx 22.2)}$	60.5% ${}_{\pm 3.6}^{(\approx 55.8)}$	52.5% ${}_{\pm 4.1}^{(\approx 53.4)}$	47.5% ${}_{\pm 4.0}^{(\approx 50.3)}$	99.2% ±0.8
	Size	24.6% ${}_{\pm 12.0}^{(\approx 3.2)}$	83.3% ${}_{\pm 16.7}^{(\approx 68.1)}$	83.3% ${}_{\pm 8.4}^{(\approx 77.0)}$	38.0% ${}_{\pm 19.5}^{(\approx 52.8)}$	91.7% ±0.0
Safety	Physical Plausibility	85.7% ${}_{\pm 3.2}^{(\approx 77.8)}$	83.3% ${}_{\pm 0.0}^{(\approx 85.0)}$	59.2% ${}_{\pm 13.4}^{(\approx 68.3)}$	97.8% ${}_{\pm 3.4}^{(\approx 92.0)}$	98.3% ±1.7
Safety	Structural Stability	16.7% ${}_{\pm 25.0}^{(\approx 25.0)}$	55.6% ${}_{\pm 17.4}^{(\approx 57.1)}$	67.1% ${}_{\pm 22.3}^{(\approx 50.7)}$	49.4% ${}_{\pm 19.7}^{(\approx 42.7)}$	93.9% ±6.1

Appendix

Table 10: Performance on MineCEraft-Lite compared with the full benchmark. . Reported values are means on Lite; subscripts ( ± ) indicate half of the min–max range across three random seeds (or, for humans, across different participants). Superscripts ( ≈ ) denote the corresponding mean on the full benchmark dataset. Although the full benchmark provides more stable estimates due to its larger task set, the values mostly remain within the Lite min–max range.

Model	Accuracy			Safety		Planning
	Material	Shape	Size	Physical Plausibility	Structural Stability	Efficiency	Dependency
Qwen3-32B	34.9% ±2.4	45.2% ±4.1	41.7% ±16.7	70.0% ±3.3	18.3% ±13.4	54.2% ±1.4	20.0% ±20.0
llama-4-scout-17b-16e-Instruct	20.6% ±7.2	23.8% ±7.1	24.6% ±12.0	85.7% ±3.2	16.7% ±25.0	37.8% ±17.8	0.0% ±0.0
llama-3.3-70b-Instruct	66.7% ±7.2	54.2% ±5.9	77.8% ±12.5	85.6% ±5.0	63.5% ±23.4	54.1% ±1.3	60.0% ±0.0
claude-haiku-4-5-20251001	79.4% ±4.8	59.6% ±10.7	75.0% ±8.3	85.6% ±5.0	65.7% ±6.6	54.2% ±1.4	60.0% ±0.0
claude-opus-4-5-20251101	95.2% ±0.0	64.0% ±5.0	88.9% ±4.2	80.0% ±10.0	67.6% ±17.4	54.2% ±1.4	73.3% ±10.0

Appendix

Table 11: Performance on MineCEraft-Lite across evaluation categories for multiple LLMs.

Figure 6: Example of von Mises stress distribution in a bridge-like structure. The peak stress appears near the folded (bending) region of the arch rather than around the ground supports where the dead load is concentrated, to evaluate the stress associated with deformation and structural failure.

Panel of Figure 7 — Figure 7: Qualitative Analysis of LLM vs Human Behavior

Panel of Figure 8 — Figure 8: Qualitative Analysis of LLM vs Human Behavior (cont.)

Panel of Figure 9 — Figure 9: Qualitative Analysis of LLM vs Human Behavior (cont.)

Panel of Figure 10 — Figure 10: Qualitative Analysis of LLM vs Human Behavior (cont.)

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Oct 18, 2025

BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction

Engineering construction automation aims to transform natural language specifications into physically viable structures, requiring complex integrated reasoning under strict physical constraints. While modern LLMs possess broad knowledge and strong reasoning capabilities that make them promising candidates for this domain, their construction competencies remain largely unevaluated. To address this gap, we introduce BuildArena, the first physics-aligned interactive benchmark designed for language-driven engineering construction. Technically, it contributes to the community in two aspects: (1) an extendable task design strategy spanning static and dynamic mechanics across multiple difficulty tiers; (2) a 3D Spatial Geometric Computation Library for supporting construction based on language instructions. On nine frontier LLMs and three additional open-weight models, BuildArena comprehensively evaluates their capabilities for language-driven and physics-grounded construction automation. We release the code at https://github.com/AI4Science-WestlakeU/BuildArena to benefit construction automation in engineering applications.

Tian Xia, Tianrun Gao, Wenhao Deng +4

Westlake University, China · Uniforce AI, China

LLM Evaluation LLM Grounding Benchmark Construction

Ask PDF

May 29, 2026

MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft

Multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and action generation. However, their ability to sustain exploration in dynamic open worlds remains unclear. Existing embodied and game-based benchmarks often compress interaction into short-horizon tasks or entangle success with domain-specific game mechanics. In this paper, we introduce MineExplorer benchmark for evaluating open-world exploration capabilities of MLLM agents in Minecraft. We first filter atomic tasks whose solutions rely heavily on Minecraft-specific knowledge to better reflect general open-world reasoning. Then we organize the benchmark around a ReAct-style capability formulation and compose atomic tasks into implicit multi-hop tasks. To further construct reliable instances, MineExplorer uses a multi-agent synthesis workflow that jointly designs task graphs, sandbox scenes, and rule-based milestone evaluators. Human evaluation shows that the multi-agent synthesis workflow produces significantly more reliable instances than a single-agent baseline. Experiments with advanced MLLM agents show that open-world exploration remains challenging, as strong models can handle many single-hop tasks but degrade sharply when hidden prerequisites must be coordinated over longer trajectories. Further analysis finds that task difficulty tracks agent completion, and larger models or thinking modes do not consistently translate into better performance. Code and dataset are available at https://github.com/Jometeorie/MineExplorer.

Tianjie Ju, Yueqing Sun, Zheng Wu +7

School of Computer Science, Shanghai Jiao Tong University · 2Meituan

Long-Horizon Agent Evaluation AI Agent Benchmarks Multimodal Agents

Ask PDF

Aug 10, 2026

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its

1{,}212

short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating

18

open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about

30\%

and the best proprietary system

52\%

, while human experts reach

95\%

, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.

Chenxu Du, Kang An, Tengyue Wang +8

Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology +7

Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation

Ask PDF

Abstract

LLM Evaluation

Explore similar work

BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction

MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence