\$OneMillion-Bench: How Far are Language Agents from Human Experts?

Yang Liu; Jiaqi Li; Jun Bai; Qianyu Yang; Xiaobo Hu; Tao Peng; Zaiyuan Wang; Ran Tian

cs.LGMar 9, 2026

$OneMillion-Bench: How Far are Language Agents from Human Experts?

Authors: Yang Liu, Jiaqi Li, Jun Bai, Qianyu Yang, Xiaobo Hu, Tao Peng, Zaiyuan Wang, Ran Tian, +15 more

Organizations: BIGAI · Humanlaya · xbench · M-A-P

Abstract

As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce $OneMillion-Bench ($OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, $OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.

Figures & tables

Figure 1 : Leaderboard performance on $OneMillion-Bench.

Figure 2 : Data curation pipeline of $OneMillion-Bench.

Figure 3 : $OneMillion-Bench consists of 5 macro domains, 37 sub-domains and 92 third-level categories covering a wide variety of real applications and professional scenarios ( cf. Appendix C ).

Subset	Domain	# Questions	Question Len.	# Rubrics	# Negative Rubrics
CN	Finance	40	554.8	18.8	4.2
	Law	40	709.1	16.3	3.9
	Healthcare	40	584.2	16.9	4.3
	Natural Science	40	555.8	16.8	4.3
	Industry	40	965.2	15.9	4.0
Global	Finance	40	1810.9	18.5	4.1

Table 1: Data Statistics of $OneMillion-Bench

Panel of Table 2 — Table 2: Comparison of Economic Value ($USD), Expert Score (%), and Pass Rate(%) on Global set . The sign of denotes enabling the usage of web search tool, and the same holds for Table 3 and 4 .

Panel of Table 3 — Table 3: Comparison of Economic Value (¥CNY), Expert Score (%), and Pass Rate (%) on CN set .

Panel of Table 4 — Table 4 : Expert Score aggregated by rubric type, averaged over Global and CN subsets. FI: Factual Information. AR: Analytical Reasoning. IF: Instructions Following. SF: Structure and Formatting.

Figure 4 : Comparison of web search scaffolds.

Figure 5 : Test-time scalibility on Finance of the Global subset for consistency validation.

Figure 6 : Pareto frontiers of various agents and models on $OneMillion-Bench.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Task UUID	Domain	CNY	USD	Subset
f4cdab18-7636-4416-bcc0-9f40705687a5	Economics and Finance	5,793	–	cn
9709dc9b-ea0a-4e14-af06-3c636798b242	Economics and Finance	9,837	–	cn
50cea244-cd7b-4213-94f0-eb199e056eb5	Economics and Finance	6,664	–	cn
5e9d346b-e174-4f7c-bc0e-c0ddd3c0bb64	Economics and Finance	7,298	–	cn
0781ab70-3513-4da6-8032-a327aba51d26	Economics and Finance	3,862	–	cn
a4d2c2e8-9143-414a-86df-b76d565283c9	Economics and Finance	9,202	–	cn

Appendix

Table 5: Task economic value of $OneMillion-Bench

Figure 7: Sample cases of different domains with varying rubric weights and tags.

Figure 8: Diachronical analysis ( i.e . , time-sensitivity analysis) on $OneMillion-Bench.

Figure 9: Correlation of Expert Score between model judge and human judge.

Model / Agentic System	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
Claude-Opus-4.6	1	1	1	1	1
GPT-5.4-High	2	2	2	2	3 (+1)
Qwen3.5-Plus	3	3	3	4 (+1)	2 (-1)
GPT-5.2-High	4	4	4	3 (+1)	4
Kimi-K2.5	5	5	6 (+1)	7 (+2)	6 (+1)
GPT-5.3-Codex	6	6	5 (-1)	5 (-1)	5 (-1)

Appendix

Table 6: Model ranking results (

\Delta

) using various cliff thresholds.

Panel of Figure 10 — Figure 10: Prompt template rubric-based evaluation.

Model	Search Provider	Thinking Effort	Misc. Parameters
DeepSeek-V3.2	OpenRouter	[On by default]	Temperature = 0
			Max new tokens = 65,536
DeepSeek-V3.2-Speciale	OpenRouter	[On by default]	Temperature = 0
			Max new tokens = 65,536
MiniMax-M2.1	OpenRouter	[On by default]	Temperature = 0
			Max new tokens = 65,536

Appendix

Table 7: Sampling parameters adopted in the models evaluation against the $OneMillion-Bench.

Model / Agentic System	Finance		Healthcare		Industry		Law		Natural Science
Model / Agentic System	Vanilla	Search	Vanilla	Search	Vanilla	Search	Vanilla	Search	Vanilla	Search
GPT-5.4-High	42.0	68.8 ↑26.8	63.6	53.9 ↓9.7	54.7	58.2 ↑3.5	55.2	55.6 ↑0.5	59.4	59.4 ↓0.0
Claude-Opus-4.6	42.1	55.5 ↑13.4	68.4	73.7 ↑5.3	58.9	67.5 ↑8.7	53.0	59.5 ↑6.5	52.4	58.9 ↑6.5
GPT-5.3-Codex	36.2	63.2 ↑27.1	53.2	48.8 ↓4.4	50.4	48.2 ↓2.1	46.7	50.8 ↑4.0	53.8	42.0 ↓11.9
Qwen3.5-Plus	30.7	31.3 ↑0.6	59.3	60.9 ↑1.6	50.8	52.8 ↑2.0	49.7	48.6 ↓1.1	47.5	51.1 ↑3.6
GPT-5.2-High	29.3	60.7 ↑31.4	59.5	59.3 ↓0.2	50.1	52.5 ↑2.4	48.8	57.0 ↑8.2	50.0	54.7 ↑4.7

Appendix

Table 8: Comparison of Expert Score across different domains on the Global evaluation set .

Model / Agentic System	Finance		Healthcare		Industry		Law		Natural Science
Model / Agentic System	Vanilla	Search	Vanilla	Search	Vanilla	Search	Vanilla	Search	Vanilla	Search
GPT-5.4-High	53.6	63.3 ↑9.7	67.3	57.1 ↓10.1	55.8	57.5 ↑1.7	43.9	53.6 ↑9.7	60.6	62.1 ↑1.5
Claude-Opus-4.6	46.9	60.9 ↑14.0	68.4	70.6 ↑2.2	59.0	69.7 ↑10.7	49.4	60.9 ↑11.4	55.5	60.7 ↑5.2
Qwen3.5-Plus	48.4	46.9 ↓1.4	60.1	61.7 ↑1.5	50.2	49.0 ↓1.2	48.9	56.1 ↑7.2	49.6	46.7 ↓2.9
GPT-5.2-High	41.4	56.1 ↑14.8	57.0	60.8 ↑3.8	46.4	50.3 ↑4.0	35.0	42.5 ↑7.4	55.8	53.3 ↓2.5
GPT-5.3-Codex	45.6	54.7 ↑9.1	57.1	54.9 ↓2.2	45.6	54.6 ↑9.0	37.0	41.2 ↑4.3	49.1	49.2 ↑0.1

Appendix

Table 9: Comparison of Expert Score across different domains on the CN evaluation set .

Domain	Count	Subset	Avg. Time (h)	Avg. Wage (/h)	Avg. Value/Q	Total Value
Finance	80	CN (¥)	23.5	312.6	7,410.8	296,432
Finance	80	Global ($)	26.1	175.7	4,593.2	183,726
Law	80	CN (¥)	26.2	180.4	4,722.5	188,901
Law	80	Global ($)	31.4	243.0	7,666.1	306,646
Healthcare	80	CN (¥)	22.9	189.9	4,379.9	175,197
Healthcare	80	Global ($)	22.9	354.9	8,188.9	327,557

Appendix

Table 10: Economic value (

USD) of

OneMillion-Bench by domain and regional subsets.

Explore similar work

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
Jun 9, 2026Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin +12Customer Support Automation LLM Agent Evaluation
AI Foundations, Capital One
OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
Jul 16, 2026Chengyu Shen, Yujie Fu, Gangtao Xin +13Computer-Use Agent Benchmarks AI Agent Evaluation
Huawei Cloud Post-Training Team
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University

Jun 9, 2026, cs.CLJ/K move · Enter open · S save

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains

Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin, Swasthi P Rao+11

AI Foundations, Capital One

Customer Support Automation LLM Agent Evaluation AI Agent Benchmarks Tool-Using Agents

Open paper AskPDF arXiv

Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems. However, existing benchmarks remain limited in task complexity, realism, and domain diversity, and often fail to capture interactions that span multiple domains, limiting their ability to evaluate agents in realistic multi-step settings that require sustained reasoning and coordination. To address these limitations, we introduce T1-Bench, a high-fidelity, comprehensive benchmark for evaluating agentic systems in realistic customer-facing, multi-domain environments, featuring interleaved scenarios that require structured reasoning across multi-turn user-assistant interactions and substantially increasing both compositional complexity and evaluative rigor across 25 domains of varying difficulty. We evaluate T1-Bench using 12 proprietary and open-weight models, providing a reproducible and standardized framework for assessing agent behavior, tool utilization, and conversational quality in complex, multi-step environments. We further complement automatic evaluation with human judgments to strengthen the assessment of qualitative performance. Overall, T1-Bench substantially advances prior benchmarks by increasing task complexity, interaction depth, and domain coverage in simulated multi-domain environments. To facilitate future research on agentic systems, we will publicly release data and evaluation code as open source.

Abstract

Explore similar work

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

WorldBench: Culturally Grounded Benchmark for Multilingual Agents