AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments

Wang Yang; Chaoda Song; Xinpeng Li; Debargha Ganguly; Chuang Ma; Shouren Wang; Zhihao Dou; Yuli Zhou

cs.AIApr 7, 2026

AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments

Authors: Wang Yang, Chaoda Song, Xinpeng Li, Debargha Ganguly, Chuang Ma, Shouren Wang, Zhihao Dou, Yuli Zhou,

Organizations: Case Western Reserve University · NII LLMC (Japan) · University of Zurich

Abstract

Existing Agent benchmarks suffer from two critical limitations: high environment interaction overhead (up to 41% of total evaluation time) and imbalanced task horizon and difficulty distributions that make aggregate scores unreliable. To address these issues, we propose AgentCE-Bench built around a unified grid-based planning task, where agents must fill hidden slots in a partially completed schedule subject to both local slot constraints and global constraints. Our benchmark offers fine-grained control through two orthogonal axes: Scalable Horizons, controlled by the number of hidden slots $H$ , and Controllable Difficulty, governed by a decoy budget $B$ that determines the number of globally misleading decoy candidates. Crucially, all tool calls are resolved via static JSON files under a Lightweight Environment design, eliminating setup overhead and enabling fast, reproducible evaluation suitable for training-time validation. We first validate that $H$ and $B$ provide reliable control over task horizon and difficulty, and that AgentCE-Bench exhibits strong domain consistency and model discriminability. We then conduct comprehensive experiments across 13 models of diverse sizes and families over 6 domains, revealing significant cross-model performance variation and confirming that AgentCE-Bench provides interpretable and controllable evaluation of agent reasoning.

Figures & tables

Panel of Figure 1 — Figure 1: This figure demonstrates the key benefits of AgentCE-Bench as a benchmark: (1) it clearly distinguishes the agentic capabilities of models of different sizes, and (2) it identifies the capability boundary of each model under controlled agent task horizon and difficulty. Heatmaps of average reward (%) for Qwen3.5 Dense models show that harder tasks (more hidden slots and larger decoy budget) yield lower rewards within each model, confirming configurable difficulty; larger models score higher, confirming discriminability across scales.

Figure 2: Heatmap of average reward (%) across models and domains. Each cell represents the average task completion reward of a model on a specific domain, highlighting cross-domain performance consistency and inter-model differences.

Figure 3: Overview of AgentCE-Bench framework. Left (Data Examples): Each task is controlled by two configurable axes — hidden slots ( $H$ ) and decoy budget ( $B$ ). Each hidden slot has a candidate pool containing one truth (green), filtered candidates that violate slot constraints (red), and decoy candidates that pass slot constraints but cause global conflicts (orange). Right (Interaction): The agent operates in a lightweight JSON-based environment, iteratively querying and filling hidden slots through multi-turn tool-use dialogue, enabling precise measurement of agent capability across configurable difficulty levels.

Panel of Figure 5 — Figure 5: Top row: Distribution of task steps across three domains (airline, retail, and telecom), showing highly skewed and uneven distributions with large variance in task complexity. Bottom row: Reward scores by domain for Claude ( Anthropic, 2024 ) , GPT-4.1, and o4-mini, revealing significant difficulty imbalance across domains — some domains are consistently harder than others, with noticeably unequal performance distributions.

Figure 6: Illustration of a benchmark instance in the course scheduling domain. Left: A $5{\times}7$ course schedule grid with one hidden slot at (Wed, Time 5) that the agent must fill. Global constraints apply to the entire grid. Right: The candidate pool for the hidden slot, categorized into three types: Truth (green) — the unique correct answer satisfying; Filtered Candidates (red) — items that violate slot-level constraints; and Decoy Candidates (orange) — items that pass slot constraints but cause global constraint violations

Panel of Figure 7 — Figure 7: Verification of benchmark properties. (a) Task steps scale consistently with the number of hidden slots, confirming controllable horizon. (b) Score decreases as decoy budget increases, demonstrating effective difficulty control. (c) Domain-level scores of Qwen2.7B remain consistent across domains, suggesting low sensitivity to domain variation. (d–f) Score distributions across Qwen3.5 Dense, Qwen3.5 MoE, and MiniMax model families show strong discriminability, with rankings aligned with model scale.

		Usage (steps / Hour / K)			Score (%)
Arch.	Size	Steps	Time	Tokens	Course	Meal	PC	Shop	Travel	Work	Avg.
Qwen3.5 Dense
Dense	0.8B	283.5	3.9	67.1	1.9	0.0	1.9	0.0	0.0	0.0	0.6
Dense	2B	174.4	3.4	61.9	3.7	1.9	7.4	1.9	1.9	3.7	3.4
Dense	4B	144.5	2.4	29.8	38.9	46.3	42.6	29.6	33.3	25.9	36.1
Dense	9B	134.1	2.0	24.2	51.9	44.4	42.6	44.4	46.3	46.3	46.0

Table 2: Evaluation results of different models across six domains. Arch. denotes the model architecture (Dense or MoE). Steps , Time (total hours on 8

\times

H200), and Tokens (K) report the average interaction steps, total wall-clock time, and average completion tokens per task, respectively. The benchmark consists of

6\times 9

tasks per domain, with hidden slots sampled from

\{1,5,7,11,15,21\}

and decoy budgets from

\{0,2,4,8,10,15,19,21,25\}

. Domain scores and Avg. report the average reward (%) per domain and overall.

Panel of Figure 8 — Figure 8: Heatmaps of average reward (%) across hidden slots and decoy budgets for all evaluated models. Higher reward indicates better task completion under the corresponding horizon and difficulty setting.

Panel of Figure 9 — Figure 9: Heatmaps of average reward across hidden slots and decoy budgets on the Course domain under varying tool failure rates (0.0, 0.1, 0.3) of Qwen3.5-122B-A10B, simulating real-world tool call failures due to network or service instability. Higher failure rates consistently degrade agent performance across all horizon and difficulty levels.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Description
set_slot(row, col, id)	Place an item id into the grid at (row, col) , or clear the slot if id is None .
get_current_grid_state()	Return the full current state of the grid, including all filled and empty slots.
get_slot_id(row, col)	Query the item id currently occupying a specific grid cell.
get_hidden_slot_query_budget(row, col)	Return the remaining attribute query budget for a given hidden slot.
get_global_check_budget()	Return the remaining budget for global constraint checks.
done()	Signal that the agent has completed the task.

Appendix

Table 3: Common tools available across all domains.

Tool	Description
query_{domain}_candidate_from_attribute(row, col, field, operator, value)	Filter the candidate list for a given slot by an attribute condition (field operator value) , narrowing the search space.
get_{domain}_item_info(id)	Retrieve the full attribute profile of a single item.
get_{domain}_item_attributes(ids, field)	Batch-query a specific attribute field for a list of item ids.
check_{domain}_slot_constraints(row, col)	Check whether the item currently placed at (row, col) satisfies the local slot constraints for that position.
check_{domain}_global_constraints()	Check whether the entire current grid satisfies all global constraints.

Appendix

Table 4: Domain-specific tools (instantiated per domain by replacing {domain} with the domain name).

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints
Aug 1, 2026Meher Bhaskar Madiraju, Meher Sai Preetam MadirajuAI Agent Evaluation Agent Evaluation
Efficient Benchmarking of AI Agents
Mar 24, 2026Franck NdzomgaAI Agent Evaluation AI Agent Benchmarks
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
Oct 1, 2026Luis Wiedmann, Leander Girrbach, Cordelia Schmid +1AI Agent Reliability AI Agent Evaluation
Technical University of Munich, MCML · Helmholtz Munich · Inria, École normale supérieure, CNRS, PSL Research University

Aug 1, 2026J/K move · Enter open · S save

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints

Meher Bhaskar Madiraju, Meher Sai Preetam Madiraju

AI Agent Evaluation Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

We present AgentSLABench, a resource-aware evaluation framework for autonomous AI agents that measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets. Unlike standard benchmarks that report only accuracy, AgentSLABench produces a multi-dimensional profile per agent per task - the same way systems profilers (perf, pprof, cProfile) measure resource consumption of code, but extended with task correctness as a first-class dimension. AgentSLABench provides 16 task environments across 6 categories (5 core: multi-hop QA, retail substitution, code generation, web shopping, travel planning; 11 extended) with isolated Docker containers, declared CPU/memory/time/network budgets, sealed test sets with SHA256 hashes, and a standardized profiling protocol. We profile 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) plus 4 task-specialized agents, finding that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7-83.3% on retail and code_gen, while general baselines fail entirely on 4/5 domain tasks. Crucially, we report the Efficiency-Adjusted Success Rate (EASR) - success weighted by resource consumption relative to declared budgets - revealing that high accuracy at unbounded cost is not production-viable. We release the full infrastructure, sealed test sets, and profiling results to enable reproducible, resource-aware agent evaluation.

Abstract

AI Agent Benchmarks

Explore similar work

AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints

Efficient Benchmarking of AI Agents

Agents Are Systems, Not Models: Rethinking Agentic Evaluation