Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling

Xiaojun Wu; Cehao Yang; Xuhui Jiang; Chengjin Xu; Jia Li; Jian Guo

cs.CLNov 5, 2026

Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling

Authors: Xiaojun Wu, Cehao Yang, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo

Organizations: IDEA Research · The Hong Kong University of Science and Technology (Guangzhou) · DataArcTech Ltd.

Abstract

Mathematical modeling turns an open-ended real-world problem into assumptions, a model, a solution, and a defended recommendation, and its answers matter only if they are both close to a credible reference and physically feasible; existing benchmarks either score solver fragments or grade reports with an LLM judge, leaving those numbers unchecked. We introduce TB-MathModel (the Terminal-style Mathematical Modeling Bench), the first benchmark to make complete contest-style modeling problems agentic, verifiable, and terminal at once, over 18 real CUMCM and MCM problems (2023--2025). Each task ships the verbatim statement, official attachments, and figures in a containerized terminal; the agent returns one JSON artifact, scored on the numeric core of the final question against a reproduction of the Outstanding-award paper. Two design choices distinguish it: direction-aware, one-sided scoring, which never penalises beating the anchor, and two-layer feasibility verification---a programmatic gate over six decidable invariant forms plus 624 Lean~4 obligations---with a hard gate that zeroes infeasible answers. Each task is scored against a panel of human outstanding papers rather than one, so a model must be close to many accepted answers; across seven multimodal entries (three seeds each) the strongest reach a multi-reference O-Eval of 64.58%, while the $2.54 open model holds 59.81% at $1/100$ the cost. TB-MathModel turns the numeric core of modeling into a reproducible, machine-checkable signal for measuring long-horizon modeling competence and for collecting agent trajectories. The source code is available at https://github.com/wxj630/terminal-bench-math-modeling.

Figures & tables

Benchmark	#Tasks	#Fig.	Evaluation signal	Agentic	Physical	Lean	Multimodal
Mamo Huang et al. (2025b)	1209	–	Solver check	✗	✗	✗	✗
ModelingBench Qian et al. (2025)	68	–	LLM judge	✗	✗	✗	✗
MM-Bench Liu et al. (2025)	111	–	LLM judge	✗	✗	✗	✗
TB-MathModel (ours)	18	21	Executable verifier	✓	✓	✓	✓

Table 1: The closest mathematical-modeling benchmarks. Only TB-MathModel combines a complete contest problem, an agentic terminal run, an executable verifier, physical-feasibility checking, and a machine-checkable proof, on a multimodal task (“Judge”: LLM grades a report; “Solver”: solver checks one formal artifact).

Figure 1: Why modeling is hard to evaluate, and what TB-MathModel changes. (a) In the status quo an LLM judge grades a report on organisation while the numbers are never checked. (b) TB-MathModel runs the problem, attachments, and figures in a terminal container, requires one structured artifact, scores it one-sidedly against an O-award reproduction, and verifies feasibility with a gate and Lean certificates—so a plausible-but-infeasible answer is caught.

Figure 2: The TB-MathModel construction and evaluation pipeline. Build : official statements, attachments, and PDF figures are compiled into a task directory. Run : the agent works inside a network-isolated terminal container and writes one JSON artifact. Evaluate : the artifact is scored by a direction-aware, one-sided panel against an O-award reproduction, and independently checked by a programmatic feasibility gate plus a Lean 4 certificate layer; a failed gate zeroes the task.

Model + harness	Art.	O-Eval (%)	Multi-BO	Cost (USD)
GPT-6 Astra + codex	18	$64.58\pm 2.93$	$1.132\pm 0.02$	246.41
Fable 5.1 + claude-code	18	$63.11\pm 1.33$	$1.081\pm 0.05$	272.80
DeepSeek-Flash + opencode	18	$59.81\pm 1.25$	$1.000$	2.54
GPT-5.5 xhigh + codex	18	$59.75\pm 1.55$	$0.998\pm 0.01$	99.37
Kimi K3 + opencode	18	$58.23\pm 1.12$	$0.953\pm 0.04$	23.26
GLM-5.3 + opencode	18	$57.78\pm 1.30$	$0.941\pm 0.05$	28.15

Table 2: Main results: the seven multimodal entries, 18 tasks, sorted by O-Eval . “Art.”: tasks with a scoreable artifact; cost: USD per 18-task pass. O-Eval is the multi-reference panel score (%) and Multi-BO the same score relative to the DeepSeek V4.1 Flash baseline (

1.000

= level with Flash), both three-seed means (

\pm

std).

Figure 3: Left: multi-reference O-Eval versus cost (log scale)—the Pareto frontier is compressed, with near-frontier scores at two orders of magnitude lower cost. Centre: O-Eval per entry. Right: O-Eval by suite, showing the CUMCM–MCM gap.

Harness	O-Eval (%)	Multi-BO
opencode	$59.81\pm 1.25$	$1.000$
claude-code	$59.83\pm 1.13$	$1.081\pm 0.05$
codex	$59.84\pm 1.21$	$1.085\pm 0.03$
DeepSeek (native)	$59.88\pm 1.20$	$1.092\pm 0.03$

Table 3: Harness sensitivity, holding the model fixed (DeepSeek-Flash, 18 tasks, three seeds). Multi-BO is

1.000

for the opencode reference harness by definition.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Slug	Contest	Topic	Final Q
cumcm-2023-a-heliostat	CUMCM 2023 A	Heliostat field design	Q3
cumcm-2023-b-multibeam	CUMCM 2023 B	Multibeam survey lines	Q4
cumcm-2023-c-vegetable	CUMCM 2023 C	Vegetable pricing and replenishment	Q4
cumcm-2024-a-dragon	CUMCM 2024 A	Bench-dragon kinematics	Q5
cumcm-2024-b-production	CUMCM 2024 B	Production decision	Q4
cumcm-2024-c-crop	CUMCM 2024 C	Crop-planting strategy	Q1

Appendix

Table 4: The 18 TB-MathModel tasks and their O-award anchors. “Final Q” is the contest question whose numeric core is scored; oracle coverage is 18/18.

Slug	Final Q	Source	#Fig.	#Metric
cumcm-2023-a-heliostat	Q3	CUMCM 2023 A	2	5
cumcm-2023-b-multibeam	Q4	CUMCM 2023 B	8	4
cumcm-2023-c-vegetable	Q4	CUMCM 2023 C	0	2
cumcm-2024-a-dragon	Q5	CUMCM 2024 A	5	2
cumcm-2024-b-production	Q4	CUMCM 2024 B	1	6
cumcm-2024-c-crop	Q1	CUMCM 2024 C	0	2

Appendix

Table 5: Per-task summary. “Final Q” is the sanctioned final question.

Task	GPT-6 Astra	Fable 5.1	GPT-5.5 xhigh	DeepSeek-Flash
cumcm-2023-a-heliostat	86.2	80.7	80.5	80.2
cumcm-2023-b-multibeam	94.0	80.5	91.7	59.3
cumcm-2023-c-vegetable	64.3	64.3	64.3	64.3
cumcm-2024-a-dragon	83.4	83.4	83.4	83.4
cumcm-2024-b-production	95.7	80.5	87.2	87.2
cumcm-2024-c-crop	88.2	83.5	99.7	63.8

Appendix

Table 6: Per-task multi-reference O-Eval (%) for four representative entries. Each column’s mean equals that entry’s main-table O-Eval , so the per-task values reconcile with the board.

Figure 4: An official statement figure, cropped from the CUMCM 2023 A PDF and delivered inline in the prompt. 21 such figures across 6 tasks are load-bearing: the final answer depends on reading them.

Model	Harness / tier	Art.	O-Eval (%)	Multi-BO
GPT-6 Astra	codex CLI / max	18	$64.58\pm 2.93$	$1.132\pm 0.02$
Fable 5.1	claude-code CLI / max	18	$63.11\pm 1.33$	$1.081\pm 0.05$
GPT-5.5 xhigh	codex CLI / xhigh	18	$59.75\pm 1.55$	$0.998\pm 0.01$
DeepSeek-Flash	opencode CLI / high	18	$59.81\pm 1.25$	$1.000$
Kimi K3	opencode CLI / high	18	$58.23\pm 1.12$	$0.953\pm 0.04$
GLM-5.3	opencode CLI / high	18	$57.78\pm 1.30$	$0.941\pm 0.05$

Appendix

Table 7: Multimodal board, 18 tasks, sorted by O-Eval . “Art.” is the number of tasks with a scoreable artifact; O-Eval is the multi-reference panel score and Multi-BO the same score relative to the DeepSeek V4.1 Flash baseline, both three-seed means (

\pm

std).

Slug	Paper id	Slug	Paper id
cumcm-2023-a-heliostat	A175	mcm-2023-a-plant	2309229
cumcm-2023-b-multibeam	B226	mcm-2023-b-maasai	2315379
cumcm-2023-c-vegetable	C050	mcm-2023-c-wordle	2307946
cumcm-2024-a-dragon	A016	mcm-2024-a-lamprey	2407093
cumcm-2024-b-production	B159	mcm-2024-b-submersible	2419984
cumcm-2024-c-crop	C038	mcm-2024-c-tennis	2401298

Appendix

Table 8: Outstanding-award anchor per task.

Quantity	Oracle (anchor)	GPT-5.5	DeepSeek V4.1
max_head_speed_mps	1.246266	1.24626636	1.24626636
max_speed_ratio	1.604793	1.60479338	1.60479338

Appendix

Table 9: Dragon-dance (CUMCM 2024 A, Q5). All three roles agree; the paper value is the outlier and is infeasible.

Quantity (yuan, 7-year)	Anchor (exact MILP)	GPT-5.5	DeepSeek V4.1
waste-surplus profit	40,418,877	40,320,587	41,559,515
50%-discount profit	58,718,967	66,485,662	70,378,276

Appendix

Table 10: Crop-planting (CUMCM 2024 C, Q1). Special case: the anchor is a reproducible exact optimum under the paper’s per-plot cap.

Quantity	Anchor	GPT-5.5	DeepSeek V4.1	Fable 5.1
daily cap	11,000	20,000	16,000	14,000
visitor fee (USD)	55	20	30	30
annual visitors (M)	1.492	1.453	1.434	1.728
resident acceptance	1.000	0.743	0.938	0.696
sustainability score	1.007	0.900	0.706	79.1
total revenue ($M)	431.6	369.6	366.7	458.5

Appendix

Table 11: Juneau tourism (MCM 2025 B, final question). The sustainability score is reported on different unit scales by different agents.

Form	Example obligation
domain / range	probability $\in[0,1]$ ; thickness $>0$
integer / binary	decision bits $\in\{0,1\}$ ; counts integer
ordering	$q_{95}\geq$ mean; peak $\geq$ average
exact identity	area $=$ count $\times$ w $\times$ h; union $\leq\sum$
finite domain	decision-bit count $\in\{16,17\}$
partition / sum	land-use counts sum to the grid

Appendix

Table 12: The six decidable invariant forms covering the hard-error branches of all 18 tasks.

Level	Content	Layer
A — unit / range	probability $\in[0,1]$ , thickness $>0$	gate + Lean
B — consistency	area $=$ count $\times$ w $\times$ h; partitions sum	gate + Lean
C — physical truth	does this field really produce 60 MW?	neither

Appendix

Table 13: Three levels of feasibility. Both layers cover A and B; C needs formalised physics.

Task	Oblig.	Skipped	Task	Oblig.
cumcm-2023-a-heliostat	12	0	mcm-2023-a-plant	66
cumcm-2023-b-multibeam	3	1	mcm-2023-b-maasai	17
cumcm-2023-c-vegetable	7	0	mcm-2023-c-wordle	3
cumcm-2024-a-dragon	3	0	mcm-2024-a-lamprey	302
cumcm-2024-b-production	61	0	mcm-2024-b-submersible	8
cumcm-2024-c-crop	5	0	mcm-2024-c-tennis	7

Appendix

Table 14: Obligations per task (624 total across 18 tasks). “Skipped” obligations were reported as missing evidence rather than silently satisfied.

Mathematical Reasoning Benchmarks

Mathematical reasoning benchmarks test whether AI systems can solve, explain, and verify problems across mathematical levels, formats, and languages.

See all 116 papers

MathDuels: Evaluating LLMs as Problem Posers and Solvers
Apr 23, 2026Zhiqiu Xu, Shibo Jin, Shreya Arya +1Mathematical Reasoning Benchmarks LLM Evaluation
Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania
How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling
Apr 6, 2026Yuhang Liu, Heyan Huang, Yizhe Yang +3LLM Evaluation LLM Mathematical Reasoning
Beijing Institute of Technology, Beijing, China · Southeast Academy of Information Technology, Putian, China
MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs
May 8, 2026Viresh Pati, Zhengyu Li, Piyush Jha +3Mathematical Reasoning Benchmarks Synthetic Benchmark Generation
Georgia Institute of Technology, USA · KNOWIDEA Technologies

Apr 23, 2026J/K move · Enter open · S save

MathDuels: Evaluating LLMs as Problem Posers and Solvers

Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik

Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania

Mathematical Reasoning Benchmarks LLM Evaluation Mathematical Problem Generation LLM Mathematical Reasoning

Open paper AskPDF arXiv

As frontier language models attain near-ceiling performance on static mathematical benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets. We introduce MathDuels, a self-play benchmark in which models occupy dual roles: each authors math problems under adversarial prompting and solves problems authored by every other participant. Problems are produced through a three-stage generation pipeline (meta-prompting, problem generation, and difficulty amplification), and validated by an independent verifier that excludes ill-posed questions. A Rasch model (Rasch, 1993) jointly estimates solver abilities and problem difficulties; author quality is derived from the difficulties of each model's authored problems. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation reveals capability separations invisible in single-role benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the benchmark's difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. We host a public leaderboard that updates as new models are released.

Abstract

Mathematical Reasoning Benchmarks

Explore similar work

MathDuels: Evaluating LLMs as Problem Posers and Solvers

How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling

MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs