General Agent Evaluation

Elron Bandel; Asaf Yehudai; Lilach Eden; Yehoshua Sagron; Yotam Perlitz; Elad Venezian; Natalia Razinkov; Natan Ergas

cs.AIFeb 26, 2026

General Agent Evaluation

Authors: Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, +7 more

Organizations: IBM Research · MIT

Abstract

General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse unfamiliar environments. This is the first systematic study, comparing tool-calling, MCP, code-generation, and CLI agents on the same benchmarks with the same models. Two gaps blocked such a study: existing harnesses require per-benchmark wiring or fixed protocol classes (web for BrowserGym, CLI for Harbor), and benchmarks themselves expect human-authored prompts, context, and integration glue. To enable this study, we contribute (1) a unifying protocol that bridges existing benchmark and agent protocols; (2) an evaluation harness that surfaces any benchmark to any general-purpose agent and backbone model; and (3) the first Open General Agent Leaderboard of agent configurations, a full factorial over 5 agent architectures x 5 backbone LLMs (three closed-source, two open-weight) x 6 benchmarks spanning software engineering, customer service, deep research, and personal assistance. We find that (i) general agents adapt to every tested domain without per-domain customization; (ii) agent architecture choice swings results by up to 12pp within a single model, yet backbone model choice dominates overall performance; (iii) on 4 of 6 tested benchmarks, top general agents are indistinguishable from the leading heavily-customized domain-specific agents; (iv) open-weight models tested exhibit "generality sinks" absent from frontier closed-source models: they consistently collapse on specific agent architectures or benchmarks; (v) a behavioral failure analysis reveals architecture-distinctive error signatures that aggregate scoring cannot discriminate. Code, harness, leaderboard, and traces are at https://www.exgentic.ai.

Figures & tables

System	Multi-Protocol Benchmark	Multi-Protocol Agent	Agent Plug-and-Play	General-Purpose Tasks	Systematic Factorial Study
agent-eval [ 10 ]	✓	✓	✗	✗	$\overset{\times}{\checkmark}$
HAL [ 19 ]	✓	✓	✗	✓	$\overset{\times}{\checkmark}$
Inspect [ 1 ]	✓	✓	✗	✓	✗
BrowserGym [ 12 ]	✗	✗	✓	$\overset{\times}{\checkmark}$	$\overset{\times}{\checkmark}$
Harbor [ 35 ]	✗	✗	✓	$\overset{\times}{\checkmark}$	✗
AgentBeats	✗	✗	✓	✓	✗

Table 1: Positioning of this work relative to prior agent-evaluation studies and frameworks along five axes. ✓ indicates the property holds; ✗ indicates it does not;

\overset{\times}{\checkmark}

indicates it holds within a constrained class. Multi-Protocol Benchmark : benchmarks are supported in their native form across heterogeneous protocols, without forcing a fixed transport. Multi-Protocol Agent : agents are supported in their native form across heterogeneous protocols. Agent Plug-and-Play : an agent integrates without per-benchmark adaptation. Systematic Factorial Study : The work perform a comparative study that decomposes performance across the full agent-architecture

\times

backbone-model

\times

benchmark factorial. General-Purpose Tasks : evaluated tasks span domains rather than a single application class. Per-cell justifications are provided in App. A .

#	Agent	Model	App	BC+	SWE	T-Air	T-Ret	T-Tel	Avg	Cost
1	OpenAI Solo	Claude Opus 4.5	0.68	0.61	0.81	0.74	0.85	0.84	0.73	$8.5
2	Claude Code	Claude Opus 4.5	0.66	0.53	0.74	0.66	0.83	0.76	0.67	$8.0
3	Smolagent	Claude Opus 4.5	0.70	0.61	0.65	0.72	0.78	0.58	0.66	$4.4
4	ReAct Short	Gemini 3	0.55	0.48	0.71	0.70	0.82	0.73	0.62	$0.7
5	ReAct Short	Claude Opus 4.5	0.64	0.49	0.61	0.66	0.78	0.76	0.62	$3.8
6	ReAct	Gemini 3	0.50	0.48	0.71	0.70	0.82	0.73	0.61	$0.8

Table 2: The first Open General Agent Leaderboard, ranking all 25 (agent architecture

\times

backbone model) configurations on standardized benchmarks. Each cell reports binary success rate; Avg is the benchmark-weighted average; Cost is average USD per task. Per-cell Wilson 95% CIs span

\pm 7

–

13

pp; aggregate significance tests in App. F . Zero scores are discussed in § 4.7 .

Figure 1: Cost-performance tradeoffs across all 25 agent-model configurations (cost on log scale). The Pareto frontier (dashed line) shows optimal tradeoffs: DeepSeek-V3.2- and GPT 5.2-backed configurations occupy the cost-efficient end, while Claude Opus 4.5-backed configurations (notably OpenAI Solo+Claude Opus 4.5) achieve the highest performance at 1–2 orders of magnitude higher cost. Each point aggregates

n=550

tasks; model-axis contrasts aggregate

n=2{,}750

per backbone model, giving tight CIs of

\sim\pm 1.8

pp (App. F ).

Benchmark	Best Configuration	General Agent Score	Domain-Specific Agent Score
SWE-Bench Verified	OpenAI Solo + Claude Opus 4.5	0.81	0.79
BrowseComp+	Smolagent + Claude Opus 4.5	0.61	0.80
$\tau^{2}$ -Bench-Airline	OpenAI Solo + Claude Opus 4.5	0.74	0.73
$\tau^{2}$ -Bench-Retail	OpenAI Solo + Claude Opus 4.5	0.85	0.86
$\tau^{2}$ -Bench-Telecom	Smolagent + Gemini 3	0.88	0.98
AppWorld	Smolagent + Claude Opus 4.5	0.70	0.73

Table 3: Best-performing general-agent configuration per benchmark compared against the top reported domain-specific agent performance on the original benchmark leaderboards (links: App. E.2 ). Scores are success rates on 100 randomly sampled instances per benchmark; original leaderboards use full benchmarks.

Figure 2: Per-architecture failure-category shares (top-15 categories, column-normalized). The architecture axis produces structured spread on this dependent variable, in contrast to its $0.5\%$ contribution to success-rate variance (§ 4.3 ); per-architecture findings are discussed in § 4.10 .

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

System	Multi-Protocol Benchmark	Multi-Protocol Agent	Agent Plug-and-Play	General-Purpose Tasks	Systematic Factorial Study
agent-eval [ 10 ]	✓ (Inspect harness)	✓ (InspectAI solvers)	✗ (solver implementation)	✗ (scientific research only)	$\overset{\times}{\checkmark}$ (no model axis; sci only)
HAL [ 19 ]	✓	✓	✗ (per-benchmark adaptation)	✓	$\overset{\times}{\checkmark}$ (architecture per-benchmark wired)
Inspect [ 1 ]	✓	✓	✗ (per-task solver setup)	✓	✗ (framework only)
BrowserGym [ 12 ]	✗ (web only)	✗ (custom API)	✓	$\overset{\times}{\checkmark}$ (web-mediated)	$\overset{\times}{\checkmark}$ (single architecture across LLMs)
Harbor [ 35 ]	✗ (CLI only)	✗ (CLI only)	✓	$\overset{\times}{\checkmark}$ (CLI-mediated)	✗ (framework only)
AgentBeats	✗ (A2A+MCP)	✗ (A2A+MCP)	✓	✓	✗ (framework only)

Appendix

Table 4: Positioning of this work relative to prior agent-evaluation studies and frameworks along five axes. ✓ indicates the property holds; ✗ indicates it does not;

\overset{\times}{\checkmark}

indicates it holds within a constrained class. Multi-Protocol Benchmark : benchmarks are supported in their native form across heterogeneous protocols, without forcing a fixed transport. Multi-Protocol Agent : agents are supported in their native form across heterogeneous protocols. Agent Plug-and-Play : an agent integrates without per-benchmark adaptation. General-Purpose Tasks : evaluated tasks span domains rather than a single application class. Systematic Factorial Study : the work performs a comparative study that decomposes performance across the full agent-architecture

\times

backbone-model

\times

benchmark factorial. This is the extended version of Table 1 in the main paper.

Figure 3: Evolution of Agentic Evaluation. (A) Collection of separate benchmarks, each requiring a custom agent or an agent with specific adaptation per benchmark (HAL). (B) Multiple benchmarks consolidated through a single protocol, such as CLI or Web. (C) Multiple benchmarks consolidated through a common protocol that can be adapted to any agent’s protocol (Exgentic).

Benchmark	Reference impl.	Actions exposed	Task / context fields
SWE-Bench Verified	mini-SWE agent	bash , finish	Task: bash-protocol prompt + issue body. Context: empty.
AppWorld	native interpreter	one per AppWorld API ( $\sim$ 468); finish	Task: supervisor instruction. Context: policy , supervisor , datetime .
BrowseComp+	authors’ retriever	search , get_document , submit	Task: scoring-rubric prompt + question. Context: empty.
$\tau^{2}$ -Airline	$\tau^{2}$ reference ReAct	message + benchmark tools	Task: “customer service agent” line. Context: policy (airline).
$\tau^{2}$ -Retail	$\tau^{2}$ reference ReAct	message + benchmark tools	As above; policy (retail).
$\tau^{2}$ -Telecom	$\tau^{2}$ reference ReAct	message + benchmark tools	As above; policy (telecom).

Appendix

Table 5: Per-benchmark adaptation summary. Released adaptor code: src/exgentic/benchmarks/{appworld, browsecompplus, swebench, tau2}/ .

Benchmark	Reference agent	Source
SWE-Bench Verified	mini-SWE agent	github.com/SWE-agent/mini-swe-agent
AppWorld	minimal_agent.ipynb (ReAct)	StonyBrookNLP/appworld/notebooks/minimal_agent.ipynb
BrowseComp+	OpenAI search-agent baseline	texttron/BrowseComp-Plus/search_agent/openai_client.py
$\tau^{2}$ -Bench	LLMAgent (text default)	sierra-research/tau2-bench/src/tau2/agent/llm_agent.py

Appendix

Table 6: Reference agent implementations mirrored when defining each benchmark’s Unified Protocol adaptor. Each row links to the canonical implementation in the benchmark’s own repository.

Figure 4: Exgentic architecture. Exgentic defines a unified protocol between agents and benchmarks. The Exgentic Orchestrator connects the agent and the benchmark, first passing the task definition and then mediating the observations and actions that are passed between the benchmark and the agent. Exgentic provides adaptors that convert the Unified Protocol into the specific protocols required by the agents and benchmarks. Finally, the benchmark provides the quality result metrics while the agent provides the agent runtime cost.

Agent	Version	Source
ReAct	exgentic_0.1.0	first-party (LiteLLM tool-calling)
ReAct Short	exgentic_0.1.0 + litellm_1.79.1	first-party + tool shortlisting
Smolagent	smolagents_1.24.0	HuggingFace smolagents
OpenAI Solo	openai_sdk_0.7.0	openai-agents (Solo + MCP)
Claude Code	claude_code_2.1.7	@anthropic-ai/claude-code (Docker)

Appendix

Table 7: Agent versions used in every leaderboard cell. Versions for ReAct/ReAct Short reflect the first-party adaptor’s release; the ReAct Short variant additionally pins the LiteLLM tool-calling library it uses for shortlisting. Claude Code runs unmodified inside a Docker container that pins the CLI version via @anthropic-ai/[email protected] .

Agent	Execution Runtime	Tool Shortlisting	Tool Schema Guard	Communication Protocol	Memory	Planning
ReAct	✗	✗	✗	Tool-calling	$\overset{\times}{\checkmark}$	$\overset{\times}{\checkmark}$
ReAct Short	✗	✓	✗	Tool-calling	$\overset{\times}{\checkmark}$	$\overset{\times}{\checkmark}$
Smolagent	✓	✗	✓	Python-Functions	$\overset{\times}{\checkmark}$	$\overset{\times}{\checkmark}$
OpenAI Solo	✗	✗	✓	MCP	$\overset{\times}{\checkmark}$	$\overset{\times}{\checkmark}$
Claude Code	✓	✗	✓	MCP	✓	✓

Appendix

Table 8: Architectural components of evaluated agents. ✓ denotes an explicit, modular component;

\overset{\times}{\checkmark}

denotes an implicit or non-modular capability; ✗ denotes absence.

Figure 5: The six benchmarks evaluated in the Open General Agent Leaderboard, spanning software engineering, customer service, deep research, and personal assistance.

Agent	Model	App	BC+	SWE	T-Air	T-Ret	T-Tel	Mean
Solo	Opus	0.68	0.61	0.81	0.74	0.85	0.84	0.73
CC	Opus	0.66	0.53	0.74	0.66	0.83	0.76	0.67
Smol	Opus	0.70	0.61	0.65	0.72	0.78	0.58	0.66
React+Short	Gemini	0.55	0.48	0.71	0.70	0.82	0.73	0.62
React+Short	Opus	0.64	0.49	0.61	0.66	0.78	0.76	0.62
React	Gemini	0.50	0.48	0.71	0.70	0.82	0.73	0.61

Appendix

Table 9: Full agent-model configuration leaderboard (binary success rates per benchmark, deduped from runs.csv). All 25 (5 agents × 5 models) configurations across 6 benchmarks.

Configuration	Score	Cost/Task	Efficiency
ReAct + GPT 5.2	0.41	$0.17	2.43
OpenAI Solo + DeepSeek-V3.2	0.32	$0.13	2.43
OpenAI Solo + GPT 5.2	0.39	$0.19	2.03
ReAct Short + DeepSeek-V3.2	0.45	$0.23	1.95
Claude Code + DeepSeek-V3.2	0.42	$0.21	1.94
ReAct Short + GPT 5.2	0.46	$0.26	1.79

Appendix

Table 10: Cost-efficiency for all 25 (architecture, model) configurations: bench-weighted success divided by bench-weighted cost per task (score/$). Sorted by efficiency, descending.

Benchmark	Best Generalist	$n$	Gen	Spec	Gap (pp)	HW gen/spec (pp)	Verdict
SWE-Bench Verified	OpenAI Solo+Claude Opus 4.5	100	0.81	0.79	$-2$	7.5 / 8.0	Indistinguishable
BrowseComp+	Smolagent+Claude Opus 4.5	100	0.61	0.80	$+19$	9.5 / 8.0	Specialist leads
$\tau^{2}$ -Bench-Airline	OpenAI Solo+Claude Opus 4.5	50	0.74	0.73	$-1$	12.0 / 12.5	Indistinguishable
$\tau^{2}$ -Bench-Retail	OpenAI Solo+Claude Opus 4.5	100	0.85	0.86	$+1$	7.0 / 7.0	Indistinguishable
$\tau^{2}$ -Bench-Telecom	Smolagent+Gemini 3	100	0.88	0.98	$+10$	6.5 / 3.5	Specialist leads
AppWorld	Smolagent+Claude Opus 4.5	100	0.70	0.73	$+3$	9.0 / 8.5	Indistinguishable

Appendix

Table 11: Per-benchmark Wilson 95% half-widths and generalist–specialist gaps.

n

is the number of evaluated instances. “HW gen/spec” are Wilson half-widths in percentage points at the respective scores. “Gap” is specialist minus generalist (positive favors specialist). The four “Indistinguishable” rows back the body claim of parity on 4 of 6 benchmarks.

Model	AppWorld $\Delta$ (pp)	Aggregate $\Delta$ (pp)	Cost $\Delta$ /task
Claude Opus 4.5	$+3$	$+0.7$	$-\$ 1.97$
Gemini 3	$+5$	$+1.2$	$-\$ 0.15$
GPT 5.2	$+22$	$+5.5$	$+\$ 0.09$
DeepSeek-V3.2	$-5$	$-1.3$	$-\$ 0.15$
Kimi-K2.5	$+1$	$+0.3$	$-\$ 0.07$

Appendix

Table 12: Tool-shortlisting ablation: success-rate delta from adding shortlisting to a vanilla ReAct agent (ReAct+ReAct Short minus ReAct), in percentage points. Aggregate is bench-weighted across all six benchmarks.

Benchmark	Claude Code Succ	Claude Code Fail	OpenAI Solo Succ	OpenAI Solo Fail	Smolagent Succ	Smolagent Fail	ReAct Succ	ReAct Fail	ReAct Short Succ	ReAct Short Fail
AppWorld	23.67	38.56	26.41	39.39	25.69	34.17	13.24	27.91	12.34	21.41
BrowseComp+	13.90	23.64	15.23	17.96	15.89	23.83	9.28	15.53	9.28	15.53
SWE-Bench Verified	27.69	32.21	27.76	29.30	29.28	32.03	28.38	34.22	28.38	34.22
airline	10.43	13.02	10.19	13.65	10.39	14.06	9.31	12.18	9.31	12.18
retail	11.73	11.54	11.36	9.81	11.48	11.31	10.81	11.52	10.81	11.52
telecom	12.98	12.25	13.06	13.26	11.99	12.75	13.23	15.84	13.23	15.84

Appendix

Table 13: Average steps per benchmark and architecture, split by successful vs. failed sessions. The three closed-source backbones (Claude Opus 4.5, Gemini 3, GPT 5.2) are aggregated within each (benchmark, architecture) cell. Zero-step sessions are excluded (4.0% of closed-source runs); step counts are capped at 50 to limit outlier influence (7.1% of remaining runs). SWE-Bench Verified success is patch-pass ( score

\geq 1

); other benchmarks use binary session success. Per-model raw values for all five backbones are in App. E.6 .

Benchmark	Claude Code	OpenAI Solo	Smolagent	ReAct	ReAct Short
AppWorld	63%	49%	33%	111%	74%
BrowseComp+	70%	18%	50%	67%	67%
SWE-Bench Verified	16%	6%	9%	21%	21%
$\tau^{2}$ -Bench-Airline	25%	34%	35%	31%	31%
$\tau^{2}$ -Bench-Retail	-2%	-14%	-2%	7%	7%
$\tau^{2}$ -Bench-Telecom	-6%	2%	6%	20%	20%

Appendix

Table 14: Percentage difference in interactions between failed and successful runs (from the absolute counts above). Positive values mean failures take more interactions; negative values mean they take fewer. The Average row is the bench-weighted simple mean of per-benchmark percentages (1/4 each non-tau, 1/12 each tau subdomain).

Model	Agent	App	BC+	SWE	T-Air	T-Ret	T-Tel	Bench-wgt
Claude Opus 4.5	ReAct	$+127\%$	$+115\%$	$+41\%$	$+33\%$	$+10\%$	$+63\%$	$+79.8\%$
Claude Opus 4.5	ReAct Short	$+163\%$	$+115\%$	$+41\%$	$+33\%$	$+10\%$	$+63\%$	$+88.7\%$
Claude Opus 4.5	Smolagent	$+40\%$	$+97\%$	$+26\%$	$+32\%$	$-1\%$	$-12\%$	$+42.4\%$
Claude Opus 4.5	OpenAI Solo	$+71\%$	$+86\%$	$+23\%$	$+32\%$	$-9\%$	$+38\%$	$+50.1\%$
Claude Opus 4.5	Claude Code	$+93\%$	$+56\%$	$+29\%$	$+28\%$	$+8\%$	$+43\%$	$+51.1\%$
Gemini 3	ReAct	$+98\%$	$+38\%$	$+4\%$	$+26\%$	$+4\%$	$+22\%$	$+39.3\%$

Appendix

Table 15: Per-(model, agent) bench-weighted percentage difference in steps between failed and successful runs. Positive values mean failures take more steps; negative values mean failures terminate earlier than success. AppWorld, BrowseComp+, and SWE-Bench Verified contribute weight 1/4 each; the three

\tau^{2}

-Bench subdomains contribute weight 1/12 each. “–” indicates a cell with insufficient succ/fail data after filtering.

Model	Best arch.	Mean	Worst arch.	Mean	$\Delta$ (pp)	$n$	$p$
Opus	Solo	0.727	React	0.610	+11.7	105	0.100
Gemini	React+Short	0.622	Smol	0.557	+6.6	5	0.317
GPT	React+Short	0.463	Smol	0.380	+8.3	105	0.005
DeepSeek	React	0.459	Solo	0.322	+13.7	550	$<$ 0.001
Kimi	React+Short	0.428	Solo	0.250	+17.7	547	$<$ 0.001

Appendix

Table 16: Within-model agent-architecture sensitivity. For each model we report the best and worst agent architecture by bench-weighted mean success, plus a paired

t

-test on shared (benchmark, task) pairs comparing the two. Closed-source models swing 7–12pp; open-weight models swing 14–18pp with highly significant best-vs-worst tests.

	React	React+Short	Smol	Solo	CC
React	—	+0.0	-20.0	-7.6	-20.0
React+Short	$<$ 0.001***	—	+0.0	-20.0	-16.7
Smol	0.317	1.000	—	+0.0	-16.7
Solo	0.100	0.317	$<$ 0.001***	—	+0.0
CC	0.317	0.317	0.317	$<$ 0.001***	—

Appendix

Table 17: Pairwise agent-architecture paired-

t

-test

p

-values for Opus. Lower triangle:

p

; upper triangle:

\Delta

(row

-

column, in pp).

***\,p<0.001

**\,p<0.01

*\,p<0.05

	React	React+Short	Smol	Solo	CC
React	—	-4.8	+20.0	-5.7	-20.0
React+Short	0.224	—	+20.0	-1.0	-20.0
Smol	0.317	0.317	—	+0.0	-33.3
Solo	0.256	0.836	1.000	—	-40.0
CC	0.317	0.317	0.114	0.102	—

Appendix

Table 18: Pairwise agent-architecture paired-

t

-test

p

-values for Gemini. Lower triangle:

p

; upper triangle:

\Delta

(row

-

column, in pp).

***\,p<0.001

**\,p<0.01

*\,p<0.05

	React	React+Short	Smol	Solo	CC
React	—	-21.0	-7.6	+0.0	-1.9
React+Short	$<$ 0.001***	—	+13.3	+21.0	+19.0
Smol	0.003**	0.005**	—	+7.6	+5.7
Solo	1.000	$<$ 0.001***	0.003**	—	-1.9
CC	0.155	$<$ 0.001***	0.031*	0.155	—

Appendix

Table 19: Pairwise agent-architecture paired-

t

-test

p

-values for GPT. Lower triangle:

p

; upper triangle:

\Delta

(row

-

column, in pp).

***\,p<0.001

**\,p<0.01

*\,p<0.05

	React	React+Short	Smol	Solo	CC
React	—	+0.9	-0.5	+36.0	+7.1
React+Short	0.025*	—	-1.5	+35.1	+6.2
Smol	0.768	0.442	—	+36.5	+7.6
Solo	$<$ 0.001***	$<$ 0.001***	$<$ 0.001***	—	-28.9
CC	$<$ 0.001***	0.003**	$<$ 0.001***	$<$ 0.001***	—

Appendix

Table 20: Pairwise agent-architecture paired-

t

-test

p

-values for DeepSeek. Lower triangle:

p

; upper triangle:

\Delta

(row

-

column, in pp).

***\,p<0.001

**\,p<0.01

*\,p<0.05

	React	React+Short	Smol	Solo	CC
React	—	-0.2	+1.5	+34.2	+29.5
React+Short	0.564	—	+1.7	+34.4	+29.7
Smol	0.442	0.389	—	+33.0	+28.2
Solo	$<$ 0.001***	$<$ 0.001***	$<$ 0.001***	—	-4.6
CC	$<$ 0.001***	$<$ 0.001***	$<$ 0.001***	0.004**	—

Appendix

Table 21: Pairwise agent-architecture paired-

t

-test

p

-values for Kimi. Lower triangle:

p

; upper triangle:

\Delta

(row

-

column, in pp).

***\,p<0.001

**\,p<0.01

*\,p<0.05

Component	Axis decomp. (%)	Additive on cells (%)
Model	27.8	93.9
Agent	0.5	0.7
Benchmark	40.7	–
Interaction	–	5.4
Residual (within-cell + interactions)	31.0	–
Model / Agent ratio	57.8 $\times$	128.6 $\times$

Appendix

Table 22: Variance decomposition (closed-only, 90 deduped (agent, model, bench) cells). Both methodologies match the Exgentic analyse model-agent CLI exactly. Axis decomposition (model 27.8% vs. agent 0.5%) is the headline ratio cited in § 4.3 ; additive decomposition over (model, architecture) cell means yields a 5.4% interaction term.

Benchmark	Capability gap (pp)	Arch.-sens. gap (pp)	Sink type
AppWorld	$+57.0$ $[+50,+66]$	$-18.5$ $[-25,-11]$	Benchmark sink
SWE-Bench Verified	$+7.0$ $[-2,+16]$	$+0.2$ $[-9,+9]$	(none)
BrowseComp+	$+5.0$ $[-3,+19]$	$+5.8$ $[-6,+16]$	(none)
$\tau^{2}$ -Bench-Airline	$+12.0$ $[+3,+20]$	$+41.7$ $[+30,+47]$	Architecture sink
$\tau^{2}$ -Bench-Retail	$+3.0$ $[-5,+10]$	$+54.6$ $[+44,+60]$	Architecture sink
$\tau^{2}$ -Bench-Telecom	$+4.0$ $[-5,+10]$	$+54.2$ $[+44,+61]$	Architecture sink

Appendix

Table 28

	App	BC+	SWE	T-Air	T-Ret	T-Tel
App	—	+0.67	+0.60	+0.73	+0.81	+0.57
BC+	+0.67	—	+0.62	+0.75	+0.60	+0.44
SWE	+0.60	+0.62	—	+0.69	+0.73	+0.80
T-Air	+0.73	+0.75	+0.69	—	+0.73	+0.56
T-Ret	+0.81	+0.60	+0.73	+0.73	—	+0.66
T-Tel	+0.57	+0.44	+0.80	+0.56	+0.66	—

Appendix

Table 23: Cross-benchmark Spearman rank correlations across the 15 closed-model configurations. Predominantly positive correlations indicate that model identity drives consistency across benchmarks.

Agent	Model	App	BC+	SWE	T-Air	T-Ret	T-Tel	Mean
React	Opus	0.61	0.49	0.61	0.66	0.78	0.76	0.610
React+Short	Opus	0.64	0.49	0.61	0.66	0.78	0.76	0.617
Smol	Opus	0.70	0.61	0.65	0.72	0.78	0.58	0.663
Solo	Opus	0.68	0.61	0.81	0.74	0.85	0.84	0.727
CC	Opus	0.66	0.53	0.74	0.66	0.83	0.76	0.670
React	Gemini	0.50	0.48	0.71	0.70	0.82	0.73	0.610

Appendix

Table 24: Full agent-configuration leaderboard: per-(agent architecture, backbone model, benchmark) success rates (binary, deduped from runs.csv via the same path the Exgentic CLI uses) for all 25 agent configurations. Rightmost column is the bench-weighted mean (1/4 each non-tau2, 1/12 each tau2 subdomain).

Agent	Model	Benchmark	Completed	Planned	#runs
React	Opus	App	100	100	1
React	Opus	BC+	0	100	1
React	Opus	SWE	99	99	1
React	Opus	T-Air	50	50	1
React	Opus	T-Ret	100	100	1
React	Opus	T-Tel	100	100	1

Appendix

Table 25: Data quality per cell. “Completed/Planned” is the canonical (most-completed) run; “#runs” is the number of run rows for that cell in raw runs.csv (a value above 1 indicates re-runs or partial-run artifacts that the dedup step collapses to a single row).

Parameter	Value
LLMs	GPT 5.2, Claude Opus 4.5, Gemini 3 Pro, DeepSeek-V3.2, Kimi-K2.5
LLM sampling	Provider API defaults (temperature, top- $p$ )
Reasoning mode	Provider default for each model (not manually toggled)
Max turns per task	100
Agent architectures	ReAct, ReAct Short, Smolagent, OpenAI Solo, Claude Code
Benchmarks	AppWorld, BrowseComp+, SWE-Bench Verified, $\tau^{2}$ -Bench (3 subdomains)

Appendix

Table 26: Configuration shared across all runs. Per-benchmark prompts and tool description templates follow the protocol in Appendix C .

Model	Endpoint	temp.	top- $p$	top- $k$	thinking
Claude Opus 4.5	AWS Bedrock	$1.0$	undisclosed	undisclosed	off
GPT 5.2	Azure OpenAI	unsupported	unsupported	unsupported	off
Gemini 3 Pro	GCP Vertex AI	$1.0$	$0.95$	$64$	high
DeepSeek-V3.2	Azure / Foundry	$1.0$	$1.0$	unsupported	enabled
Kimi-K2.5	Azure / Foundry	$1.0^{*}$	$1.0$	unsupported	enabled

Appendix

Table 27: Provider-documented default inference settings, used verbatim in every cell. “undisclosed” = the provider’s documentation does not state a default; “unsupported” = the parameter is not exposed by the endpoint. * For Kimi-K2.5,

1.0

is the documented default for the Thinking mode that Azure serves; the generic request-schema default across modes is

0.6

Script	Reproduces	Invocation
audit_failure_steps.py	§ 4.9 , App. F	python3 tools/audit_failure_steps.py
audit_variance.py	§ 4.3 , Tab. 22	python3 tools/audit_variance.py
audit_within_model_spread.py	§ 4.4 , Tab. 16	python3 tools/audit_within_model_spread.py
audit_pairwise_significance.py	App. F : pairwise tables	python3 tools/audit_pairwise_significance.py
audit_spearman.py	§ 4.7 , Tab. 23	python3 tools/audit_spearman.py
audit_shortlisting.py	§ 4.4 : shortlisting deltas	python3 tools/audit_shortlisting.py

Appendix

Table 28: Audit scripts released with the paper. Cross-references in the “Reproduces” column point to the corresponding section, claim cluster, or table in this manuscript; some references resolve to the closest available label if no granular label exists. generate_appendix_tables.py regenerates every tables/auto_*.tex file ( auto_full_leaderboard , auto_within_model , auto_pairwise_<m> , auto_variance , auto_spearman , auto_data_quality ) in a single call.

Figure 6: Top-10 categories by share of categorized records ( $n=2{,}868$ ).

Figure 7: Per-model failure-category shares (top-15 categories, column-normalized).

Figure 8: Per-architecture failure-category shares (top-15 categories, column-normalized).

Figure 9: Per-benchmark failure-category shares (top-15 categories, column-normalized).

#	Category	n	%
1	Premature Termination	232	8.1
2	Evidence Retrieval Omission	211	7.4
3	Search Recovery & Adaptation	183	6.4
4	Unsupported Fabrication	167	5.8
5	Tool Invocation Error	167	5.8
6	Constraint Interpretation Error	157	5.5

Appendix

Table 29: Top-level agent failure categories with

\geq

1% prevalence (

n=2{,}868

categorized records).

Category	Definition
Action Execution Error	Agent has authorization and correct intent but carries out required mutations, payments, bookings, patches, cleanup, batching, or edits incorrectly or incompletely.
Authentication Handling Error	Agent omits, misreads, invents, corrupts, extracts incorrectly, or misuses credentials, tokens, login state, authentication order, or authenticated context.
Candidate Verification Failure	Agent commits to or rejects a candidate without checking discriminating constraints, alternatives, contradictions, disqualifiers, or uncertainty against available evidence.
Confirmation Handling Error	Agent omits, mistimes, misreads, repeats, or malforms required consent, clarification, preference check, disclosure, approval, or post-action confirmation communication.
Constraint Interpretation Error	Agent misreads explicit user or task constraints, including scope, dates, filters, preferences, privacy requirements, answer form, variants, or replacement boundaries.
Coverage Enumeration Omission	Agent fails to exhaust required pages, lists, histories, variants, batches, statuses, transactions, recommendations, options, contacts, or libraries for complete coverage.

Appendix

Table 30: Definitions of the 27 top-level failure categories produced by the recursive categorization stage. Definitions are the LLM judge’s own outputs (saved at category-construction time); the analysis records released alongside the paper include the raw responses.

Explore similar work

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
Jul 16, 2026Chengyu Shen, Yujie Fu, Gangtao Xin +13Computer-Use Agent Benchmarks AI Agent Evaluation
Huawei Cloud Post-Training Team
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents
Horizon Research, Frontis.AI
ResearchGym: Evaluating Language Model Agents on Real-World AI Research
Feb 16, 2026Aniketh Garikaparthi, Manasi Patwardhan, Arman CohanAI Agent Benchmarks Autonomous Research Agents
TCS Research · Yale University

Jul 16, 2026J/K move · Enter open · S save

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou+12

Huawei Cloud Post-Training Team

Computer-Use Agent Benchmarks AI Agent Evaluation Agent Evaluation AI Agent Benchmarks Tool-Using Agents

Open paper AskPDF arXiv

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.

Abstract

Explore similar work

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

ResearchGym: Evaluating Language Model Agents on Real-World AI Research