OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity

Yu Liu; Zhilin Liu; Zhiwei Yang; Shaojie Zhang; Zheyuan Deng; Tingwei Huang; Zhenbo Luo; Lei Jiang

cs.AISep 5, 2026

OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity

Authors: Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang,

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · MiLM Plus, Xiaomi Inc. · Department of Computer Science, Brown University

Abstract

As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to assess multimodal perception, multi-step execution, tool use, and artifact delivery. However, existing benchmarks are often tied to specific task types, execution environments, or scoring protocols, limiting their comparability, interpretability, and reliability for deployment decisions. We introduce OmniStepBench, a benchmark for deployment-aware agent evaluation across modalities and task complexity. Built on a shared OpenClaw execution environment, OmniStepBench organizes 233 tasks selected and adapted from 22 source benchmarks into a $2\times3$ workload matrix defined by input modality and execution form, and evaluates them under a unified contract-based protocol with evidence-based score auditing. We evaluate 30 commercial API models and 12 locally deployed open-weight models (42 models in total) over 9{,}218 applicable model--task runs, reporting accuracy and token consumption alongside reference costs for API models. Results show workload-dependent rankings, distinct accuracy--cost trade-offs between text and multimodal tasks, and competitive but weaker overall performance from local models. These findings support deployment decisions based on workload profiles, deployment mode, and accuracy--cost trade-offs rather than a single aggregate score.

Figures & tables

Figure 1: Accuracy–cost trade-offs for API models on (a) text and (b) multimodal tasks. Bracketed labels indicate average token consumption per task (in thousands); cost is reported in USD per task on a logarithmic scale. Dashed lines connect the Pareto-optimal models among those shown: MiMo-V2.5, MiMo-V2.6-Pro, Qwen3.8-Max, and GPT-6.1-Sol in (a); and MiMo-V2.5, MiMo-V2.6-Pro, GPT-6.1-Sol, and GPT-6-Astra in (b).

Figure 2: Overview of OmniStepBench. The benchmark comprises four stages: (1) constructing 233 tasks from 22 source benchmarks and organizing them by modality and execution form; (2) executing all models in a shared agent environment; (3) applying automated, LLM-based, or hybrid scoring with evidence-based auditing; and (4) reporting overall and workload-level accuracy, token usage, reference cost, and accuracy–cost frontiers.

	Text				Multi-Modal
Model	SS	MS	MT	Avg	SS	MS	MT	Avg
API Deployment
GPT-6-Astra ( OpenAI, 2026d )	96.0	82.8	70.6	80.3	70.0	94.2	67.6	78.0
GPT-6.1-Sol ( OpenAI, 2026e )	96.1	84.0	71.3	81.2	70.0	92.3	62.9	75.7
Claude-Opus-5.5 ( Anthropic, 2026c )	99.2	84.4	65.0	79.6	75.0	92.3	53.2	73.7
Claude-Opus-4.8 ( Anthropic, 2026b )	98.8	78.1	65.3	76.3	85.0	69.2	61.8	71.1

Table 1: Per-model accuracy (%) across the six OmniStepBench task groups. Averages are task-count-weighted (Text: 20/87/55; Multi-Modal: 20/26/25). SS, MS, and MT denote Single-Step, Multi-Step, and Multi-Step+Tools. Bold , underlined , and italicized values denote first, second, and third place within each deployment mode (competition ranking); “—” marks text-only models not evaluated on multimodal tasks.

Pattern	Local	API	Total
Deadlock loop	86	0	86
Rubric leakage	12	14	26
Pseudo-tool-call credulity	20	0	20
Output instability	5	2	7
Timeout-with-credit	1	3	4
Total	124	19	143

Table 3: Audit type of LLM Judge Hallucination Patterns

Figure 3: Multi-Step Visual Search on a Nutrition Scatter Chart. Example Task: Among all Fruits and Vegetables with CO 2 $>0.15$ kg per 100 calories and price tier of either Highest or Medium, which single food has the lowest Glycemic Index (GI)?

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Accuracy trajectories of locally deployed open-weight Qwen and Gemma series across the six task groups. Thin lines denote per-group accuracies; the bold red line and shading denote Overall, computed from the displayed Text and multimodal averages with task-count weights 162 and 71.

ID	Definition
SRU	Semantic Reasoning & Understanding: violates constraints, misses aggregation, or answers off-question.
TUIE	Tool Use & Information Extraction: insufficient web search/fetch retrieval capability.
PPPA	Path Planning & Protocol Adherence: fails to finish within the time limit or to write the required contract artifact (e.g., answer.txt ).
FCR	Fine-grained Chart Reading: inspects the figure but misreads legends, curve names, markers, or numeric values.
VEB	Visual Evidence Bypass: the task depends on an image, yet the model barely looks at it and guesses via web search or priors.
VCS	Visual Common Sense & Entity Recognition: misidentifies the target object, attribute, or person in the image.

Appendix

Table 4: Capability-oriented failure taxonomy.

Model	Sampled	SRU	TUIE	PPPA	FCR	VEB	VCS	OTH
API Deployment
GPT-6-Astra	1	1	0	0	0	0	0	0
GPT-6.1-Sol	1	1	0	0	0	0	0	0
Claude-Opus-5.5	2	1	0	0	0	0	0	1
Claude-Opus-4.8	3	1	0	0	0	1	0	1
Kimi-K3	3	1	0	0	0	0	0	2

Appendix

Table 5: Fine-grained failure distribution over zero-score trajectories from 42 models on 30 stratified tasks. Models are grouped by deployment mode.

(a) Tokens/task and context limits: Local Deployment
	Text				MM
Model	SS	MS	MT	Avg	SS	MS	MT	Avg	Ctx
Qwen3.6-27B	47k	52k	83k	62k	58k	54k	128k	81k	256K
Qwen3.6-35B-A3B	53k	57k	86k	66k	57k	68k	114k	81k	256K
Qwen3.5-27B	39k	64k	85k	68k	49k	53k	126k	78k	256K
Qwen3.5-35B-A3B	35k	45k	78k	55k	61k	62k	105k	77k	262K

Appendix

Table 6: Per-model token consumption and dollar cost per task for locally deployed open-weight models on OmniStepBench. Tokens and per-model context-limit figures are shown in panel (a); Ctx does not represent the evaluation window, which was fixed at 32K for every local run. Hosted-reference costs are shown in panel (b). Notation and ranking follow the main paper.

	Text				MM
Model	SS	MS	MT	Avg	SS	MS	MT	Avg
GPT-6-Astra	108k	137k	761k	345k	176k	84k	569k	281k
GPT-6.1-Sol	106k	135k	671k	313k	189k	75k	730k	338k
Claude-Opus-5.5	62k	76k	210k	119k	86k	75k	276k	149k
Claude-Opus-4.8	52k	85k	301k	154k	77k	98k	147k	109k
Kimi-K3	164k	93k	284k	166k	59k	110k	188k	123k

Appendix

Table 7: Per-model API token consumption per task, by workload group.

Model	Provider	Pool	Input	Output	Cache Read	Cache Write
API Deployment
GPT-6-Astra	OpenAI	API	10.00	50.00	1.00	12.50
GPT-6.1-Sol	OpenAI	API	2.00	10.00	0.10	2.50
Claude-Opus-5.5	Anthropic	API	4.00	20.00	0.20	5.00
Claude-Opus-4.8	Anthropic	API	5.00	25.00	0.50	6.25
Claude-Opus-4.6	Anthropic	API	5.00	25.00	0.50	6.25

Appendix

Table 9: Per-million-token prices (USD/1M tokens) for all 42 evaluated models. Provider indicates the inference backend from which the rate is sourced. “—” denotes that the provider does not expose a separate cache-read or cache-write rate. Entries marked

\dagger

use the official provider rate (model not listed on OpenRouter).

Figure 7: Task composition of OmniStepBench. The inner ring partitions the 233 tasks along the $2\times 3$ modality $\times$ execution-complexity taxonomy; the outer ring shows the 22 contributing source benchmarks. Segment sizes are proportional to task counts.

Group	#Tasks	Auto	Judge	Hybrid
Text-SS	20	10	0	10
Text-MS	87	64	23	0
Text-MT	55	35	20	0
MM-SS	20	0	20	0
MM-MS	26	16	10	0
MM-MT	25	0	0	25

Appendix

Table 10: Distribution of the three scoring modes across the six OmniStepBench workload groups. Text/MM = Text/Multi-Modal; SS/MS/MT = Single-Step / Multi-Step / Multi-Step+Tools. Auto = automated (rule-based) scoring; Judge = LLM-based scoring; Hybrid = weighted combination of the two.

Benchmark	Domain / Capability	#Tasks	Group	Scoring
Text Single-Step (Text-SS)
LogiQA	Reading comprehension with logical reasoning	10	Text-SS	Auto
GPQA	Graduate-level science multiple-choice	10	Text-SS	Hybrid
Text Multi-Step (Text-MS)
AdvancedIF	Complex multi-constraint instruction following	10	Text-MS	Judge
Bamboogle	Multi-hop question answering	10	Text-MS	Auto

Appendix

Table 11: Source benchmarks underlying OmniStepBench. #Tasks: number of tasks retained after the three-stage curation. Group: the workload group each benchmark is mapped to under the modality

\times

execution-complexity taxonomy (Text/MM = Text/Multi-Modal; SS/MS/MT = Single-Step / Multi-Step / Multi-Step+Tools). Scoring: the scoring mode(s) used by the benchmark’s tasks (Auto = automated, Judge = LLM-based, Hybrid = weighted combination; see § F.2 ).

Panel of Figure 8 — Figure 8: LLM-as-a-Judge prompt template. Field placeholders are denoted in curly braces.

Panel of Figure 9 — Figure 9: Meta-judge system prompt.

Multimodal Agent Evaluation

See all 22 papers

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses AI Agent Benchmarks
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
May 28, 2026Yinsicheng Jiang, Liang Cheng, Yeqi Huang +6Multi-Agent LLM Systems LLM Agent Evaluation
University of Edinburgh · 2Microsoft Research
BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
Aug 27, 2026Mesut TorukLLM Agent Skill Evaluation LLM Agent Evaluation
Istanbul, Turkiye

May 11, 2026J/K move · Enter open · S save

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding+13

1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China+8

LLM Agent Harnesses AI Agent Benchmarks Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents Terminal Agents LLM Agent Evaluation+4

Open paper AskPDF arXiv

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.

Abstract

Multimodal Agent Evaluation

Explore similar work

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click