cs.CLApr 13, 2026
SaveCocoaBench: Evaluating Unified Digital Agents in the Wild
Abstract
LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.
Figures & tables
Figure 1: CocoaBench evaluates agents on complex digital tasks that require flexible composition of core capabilities such as vision, search, coding. The shopping example shown here illustrates one such task and highlights the multi step, compositional nature of the benchmark.
| Benchmark | Application focus | Infra coupling | Verif. reward | Capabilities | ||
|---|---|---|---|---|---|---|
| V | S | C | ||||
| OSWorld | Computer use | VM + task setup | ✓ | ✓ | ✗ | ✗ |
| BrowseComp | Web research | Open | ✓ | ✗ | ✓ | ✗ |
| SWE-bench Pro | Repository-issue resolution | Repo container | ✓ | ✗ | ✗ | ✓ |
| TerminalBench-2 | CLI task execution | Task container | ✓ | ✗ | ✗ | ✓ |
| MCP Atlas | MCP server orchestration | Fixed apps | ✓ | ✗ | ✗ | ✓ |
Table 1: Comparison of representative agent benchmarks by application focus, infrastructure coupling, reward verifiability, and core capability coverage (V=Vision, S=Search, C=Coding).
Figure 2: Statistics of CocoaBench . ( a ) Distribution of task domains, covering a wide range of everyday topics. ( b ) Distribution of resource types required by the tasks. Documents include .csv , .pdf , and .bibtex . ( c ) Human-annotated key capabilities required for each task, including Vision, Search, and Coding. The Vision, Search, and Coding types are not mutually exclusive; numbers on the bars indicate the proportion of all tasks, and the x-axis represents the number of tasks.
Figure 3: Co-occurrence matrix of required capabilities (Vision, Search, and Coding).
Figure 4: Overall performance on CocoaBench for representative agent systems and model backbones under the shared Cocoa-Agent scaffold.
Figure 5: Accuracy-cost (left) and accuracy–time (right) trade-offs across agents. Marker shapes denote agent/scaffold types and colors denote base models.
Figure 6: Top 10 most frequently used tools, ranked by total call count.
Figure 7: Per-model distribution of tool calls across the three capability categories under Cocoa-Agent .
Figure 8: Left: Error type distribution across all six models evaluated on CocoaBench using the Cocoa-Agent framework (based on 712 failure trajectories). Right: Comparisons of error distributions between GPT-5.4 and Kimi K2.5.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Ability | Tool | Description |
| Vision | browser_click | Click at screen coordinates (left/right/middle; single/double/triple) |
| browser_type | Type text into the focused element | |
| browser_press | Press a single keyboard key | |
| browser_hotkey | Press a key combination (e.g., Ctrl+C ) | |
| browser_scroll | Scroll the page by pixel offset | |
| browser_move_to | Move the cursor to absolute coordinates |
Table 2: Complete tool inventory of Cocoa-Agent . Tools are grouped by the three key capabilities defined in CocoaBench : Vision (GUI-level browser interaction), Search (DOM-level content access and navigation), and Coding (code execution, shell commands, and file operations). The Control category contains only the task completion signal.
Figure 9: Aggregate failure distribution across all 6 models evaluated under Cocoa-Agent on CocoaBench . The inner ring: Reasoning & Planning (54%), Tool & Execution (17%), Visual Grounding (29%). The outer ring details all 9 active subcategories (E1.1–E3.3). Displayed percentages reflect each subcategory’s share of total failure-mode mentions across all 722 failed runs.
Figure 10: Cocoa-Agent with Claude Sonnet 4.6 failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (50%), Tool & Execution (17%), Visual Grounding (33%). The outer ring details 9 active subcategories. Displayed percentages reflect each subcategory’s share of total failure-mode mentions across all 114 failed runs.
Figure 11: Cocoa-Agent with GPT-5.4 failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (57%), Tool & Execution (11%), Visual Grounding (32%). The outer ring details 8 active subcategories. Displayed percentages reflect each subcategory’s share of the total failure-mode mentions across all 96 failed runs.
Figure 12: Cocoa-Agent with Kimi-k2.5 failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (55%), Tool & Execution (15%), Visual Grounding (30%). The outer ring details 9 active subcategories. Displayed percentages reflect each subcategory’s share of the total failure-mode mentions across all 137 failed runs.
Figure 13: Cocoa-Agent with Qwen3.5-397B failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (56%), Tool & Execution (19%), Visual Grounding (25%). The outer ring details 9 active subcategories. Displayed percentages reflect each subcategory’s share of the total failure-mode mentions across all 139 failed runs.
Figure 14: Cocoa-Agent with Gemini 3.1 Pro Thinking failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (55%), Tool & Execution (20%), Visual Grounding (25%). The outer ring details 9 active subcategories. Displayed percentages reflect each subcategory’s share of the total failure-mode mentions across all 107 failed runs.
Figure 15: Cocoa-Agent with Gemini 3 Flash failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (53%), Tool & Execution (17%), Visual Grounding (30%). The outer ring details 9 active subcategories. Displayed percentages reflect each subcategory’s share of the total failure-mode mentions across all 125 failed runs.
Figure 16: OpenAI Codex failures on CocoaBench mapped to the failure taxonomy. The inner ring: Reasoning & Planning (59%), Tool & Execution (5%), Visual Grounding (36%). The outer ring details 8 active subcategories. Displayed percentages reflect each subcategory’s share of total failure-mode mentions across all 86 failed runs.
| Action Type | Limit (chars) | Rationale |
|---|---|---|
| DOM & Navigation (Prone to long lists; only page context needed) | ||
| dom_get_text | 500 | |
| dom_mark_elements | 400 | |
| dom_get_html | 300 | |
| browser_navigate | 300 | |
| browser_get_viewport_info | 200 | |
Table 3: Observation truncation limits per action type used when building the judge prompt.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,427 papers