cs.AIMar 16, 2026
SaveCUBE: A Standard for Unifying Agent Benchmarks
Organizations: ServiceNow AI Research · Dalhousie · McGill · Mila ·
Abstract
The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating an "integration tax" that limits comprehensive evaluation. We propose CUBE (Common Unified Benchmark Environments), a universal protocol standard built on MCP and Gym that allows benchmarks to be wrapped once and used everywhere. By separating task, benchmark, package, and registry concerns into distinct API layers, CUBE enables any compliant platform to access any compliant benchmark for evaluation, RL training, or data generation without custom integration. We call on the community to contribute to the development of this standard before platform-specific implementations deepen fragmentation as benchmark production accelerates through 2026.
Figures & tables
| Feature | WebArena ( Zhou et al., 2024 ) | SWE-bench ( Jimenez et al., 2024 ) | osworld ( Xie et al., 2024 ) | GAIA ( Mialon et al., 2024 ) |
|---|---|---|---|---|
| Environment Type | Simulated Web : A private micro-internet (GitLab, Reddit clone, etc.). | Coding Workspace : A specific software repository and dev toolchain. | Desktop OS : A full graphical operating system (Ubuntu/Windows). | The Real World : Live internet access and local document sets. |
| Hosting Format | Benchmark-level VM : One shared world per task run. High AWS costs. | Task-level Container : Lightweight, ephemeral Docker images per task. | Benchmark-level VM : OS VM with Task-level Snapshots. | Static Files : Task files on HF Hub; users provide tool implementations. |
| Action Space (Tooling) | Flexible/Provided : Playwright basics provided, but usually swapped for custom abstractions (e.g., BrowserGym). | Fixed Shell : Standard terminal-based actions (Bash/Git). | Native GUI : Hard-coded mouse/keyboard coordinate actions. | None (BYOT) : Users must build tools (Search, PDF reader) from scratch. |
| Integration Effort | High : Requires solving rigid networking (port mapping) and building a perception bridge to the HTML. | Moderate : Standard Docker orchestration; agents speak Terminal natively. | Moderate : Authors provide the VM/API, but vision-to-action grounding is a high hurdle. | High : The researcher must provide a broad range of tools. |
| Scalability Bottleneck | State Reset Latency: VMs require high resources. Benchmark recommends sequential evaluation of all tasks. | Disk I/O Churn : Constant building/pulling of unique task images saturates the filesystem. | RAM & Snapshots : 20 GB+ memory per agent and heavy disk I/O for state resets. | API & Rate Limits : Capped by external search quotas and LLM token costs. |
Table 1: A comparative analysis of the infrastructure and operational requirements across current agentic benchmarks. The lack of uniformity in how these environments are hosted, controlled, and reset creates a significant overhead for cross-benchmark evaluation.
Figure 1: Task-level diagram for CUBE’s API. Left: Separation between tasks and tools and the possibility to reconfigure the tools. Only the Gym-API is exposed. Right: By implementing base classes, CUBE’s implementation can automatically expose an RPC layer, including the well-known MCP API. On the client side, the Gym-API is exposed.
| Method | Namespace | For | Arguments | Returns | Description |
| tools/list | mcp | Agent | none | Tool[] | List actions |
| tools/call | mcp | Agent | {name, args} | {content[], isError} | Execute action |
| resources/list | mcp | Agent | none | Resource[] | List resources |
| resources/read | mcp | Agent | {uri} | {contents[]} | Read obs/task |
| cube/evaluate | cube | Harness | none | {obs, reward, terminated, truncated, info} | Eval state |
| cube/reset | cube | Harness | {seed?} | {obs, info} | Reset task |
Table 2: Task-Level API. Methods available as both Python class methods (e.g., task.reset() ) and RPC endpoints (e.g., cube/reset ). The first part is the well-known MCP protocol. The second part adds observation, step, and evaluation functions aligned with the Gym API.
| Method | Namespace | For | Arguments | Returns | Description |
|---|---|---|---|---|---|
| cube/info | cube | Harness | none | BenchmarkInfo | Benchmark metadata |
| cube/tasks | cube | Harness | {filter?} | TaskList | List available tasks |
| cube/spawn | cube | Harness | {task_id, seed?} | url | Start task, return endpoint |
| cube/status | cube | Harness | none | TaskStatus[] | Health of running tasks |
| cube/shutdown | cube | Harness | {session_id?} | void | Cleanup all or specific |
Table 3: Benchmark-Level API. Methods available as both Python class methods and RPC endpoints for discovering benchmarks, listing tasks, and orchestrating task instances. Each spawned task exposes its own Task-Level API endpoint.
| Field | Type | Description |
|---|---|---|
| id | string | Unique identifier (e.g., webarena-verified ) |
| name | string | Human-readable name |
| version | string | Semantic version (e.g., 1.2.0 ) |
| authors | string[] | Package authors |
| paper | string? | Related paper URL (if any) |
| package | string | PyPI package name for pip install |
Table 4: Registry Fields. Each registered benchmark exposes this metadata for discovery, installation, and compliance verification.
| Feature | CUBE | NeMo Gym | AgentBeats | OpenEnv | Harbor |
|---|---|---|---|---|---|
| Primary Focus | Protocol standard for wrapping benchmarks once, usable for evaluation, RL training, and data generation | Infrastructure to develop environments and scale rollout collection, battle-tested in Nemotron 3 | Evaluation orchestration; benchmarks become judge agents assessing subject agents via A2A and MCP | Framework for creating and sharing new RL environments; standardizes the environment side of post-training | Evaluation and RL rollout framework; adapts existing benchmarks into a standard container-based format |
| Coverage | 9 CUBEs (early stage); wraps any benchmark type including shared-infrastructure and VM-based ones | 40+ environments across math, coding, tool use, and safety, and integrations with other environment libraries and benchmarks | 250+ benchmarks across 17 domains, covering both static datasets (GAIA, SWE-Bench) and interactive environments | 30+ environments across coding, games, web, and simulation; designed for new environment creation | 46+ adapters for established benchmarks (SWE-Bench, Terminal-Bench, GPQA, ARC-AGI-2) with parity validation |
| Agent Interface | MCP tools/call for actions; Gym-style cube/reset , cube/step , cube/evaluate | Tools contract through OpenAI Responses API spec; flexible Agent scaffolding, easy to plug in any environment and Gym-like APIs implementable through HTTP servers | A2A for task delegation; MCP for tool access; judge + subject + delegator roles | HTTP Gym API ( reset / step / state ); MCP tool-calling interface for agent-environment interaction | Wraps full coding agents (Claude Code, OpenHands, etc.); MCP servers configurable per task; ATIF trajectory format |
| RL Training | Gym-compatible evaluate / reset / step ; Python in-process for low-latency loops | Ray-powered rollout; NeMo RL, OpenRLHF, Unsloth integrations | Gym-to-MCP bridge available; primarily an evaluation platform | TRL, TorchForge, Unsloth, SkyRL integrations; composable Rubric reward system | QueueOrchestrator for dynamic RL loops; ATIF captures token IDs and logprobs |
| Adding a New Benchmark | Implement a Python class once; works across all CUBE-compatible platforms via thin connectors | Supports integrating benchmarks and creating new environments via separation of concerns between the Agent Harness and Environment Resources | Implement a judge agent (A2A + MCP) | Design a Docker environment with FastAPI and MCP tools; supports both new environments and wrappers around existing frameworks | Write task.toml + instruction.md + Dockerfile + test.sh ; Harbor-specific |
| Scale & Deployment | CUBE-harness uses Ray for parallel rollout; benchmarks declare resource requirements ( what ), while pluggable backends dispatch to local or cloud infrastructure ( how ) | Async first, server-based design; Ray for thousands of parallel rollouts | Five operation modes from local development to hosted deployment; GitHub Actions CI used for competition leaderboards | Docker locally; HuggingFace Spaces for sharing; Kubernetes for scaling | Docker, Daytona, Modal, E2B, GKE; QueueOrchestrator for parallel trials |
Table 5: Comparison of CUBE with existing agent benchmark frameworks, based on features and documentation at the time of writing.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,497 papers