SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories
Organizations: University of California, Berkeley · University of Chicago · Bespoke Labs
Abstract
AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement. SlackBench thus provides an extensible framework for evaluating agents in collaborative work environments. The SlackBench dataset is available at https://huggingface.co/datasets/HuanzhiMao/SlackBench .
Figures & tables
| \toprule Harness | Model | AS | HA | SU | CI | Overall |
|---|---|---|---|---|---|---|
| \midrule Antigravity | Gemini 4 Argon | 70.69 | 90.28 | 76.68 | 83.33 | 80.25 |
| Muse Code | Muse Wonder | 81.03 | 62.50 | 71.42 | 80.00 | 73.74 |
| Claude Code | Claude Fable 5.1 | 67.82 | 71.53 | 71.98 | 83.33 | 73.67 |
| Claude Code | Claude Opus 5.5 | 71.84 | 75.00 | 70.42 | 73.33 | 72.65 |
| Claude Code | Muse Wonder | 68.97 | 66.67 | 67.25 | 80.00 | 70.72 |
| Claude Code | GLM 5.3 | 69.54 | 70.83 | 65.12 | 58.33 | 65.96 |
| \toprule Harness | Model | Access | AS | HA | SU | CI | Overall |
| \midrule Claude Code | Claude Opus 5.5 | CLI | 66.67 | 72.92 | 70.12 | 85.00 | 73.68 |
| Claude Code | Claude Opus 5.5 | MCP | 71.84 | 75.00 | 70.42 | 73.33 | 72.65 |
| \midrule Claude Code | Claude Opus 4.7 | CLI | 57.47 | 55.56 | 60.48 | 65.00 | 59.63 |
| Claude Code | Claude Opus 4.7 | MCP | 52.87 | 60.42 | 52.69 | 60.00 | 56.49 |
| \midrule Claude Code | Claude Sonnet 4.6 | CLI | 58.62 | 41.67 | 55.52 | 63.33 | 54.78 |
| Claude Code | Claude Sonnet 4.6 | MCP | 50.00 | 40.97 | 47.66 | 45.00 | 45.91 |
| \toprule | CLI | MCP | ||
| \cmidrule (lr)2-3 \cmidrule (lr)4-5 Tool-call category | Calls | % | Calls | % |
| \midrule Search & filter ( grep / jq / awk vs. search_messages ) | 4,095 | 30.0% | 6,243 | 50.9% |
| Read & extract ( cat / head / jq vs. get_channel_history ) | 7,999 | 58.6% | 3,298 | 26.9% |
| List & lookup ( ls vs. list_channels , get_user_info ) | 1,164 | 8.5% | 873 | 7.1% |
| Tool / schema loading (built-in vs. ToolSearch ) | – | – | 1,848 | 15.1% |
| Other (HTTP fetch, etc.) | 393 | 2.9% | 7 | 0.1% |
| \toprule Harness | Model | Effort | AS | HA | SU | CI | Overall |
| \midrule Claude Code | Claude Opus 4.7 | Low | 50.00 | 50.00 | 45.19 | 50.00 | 48.80 |
| Claude Code | Claude Opus 4.7 | Medium | 52.87 | 60.42 | 52.69 | 60.00 | 56.49 |
| Claude Code | Claude Opus 4.7 | High | 58.62 | 72.92 | 63.94 | 75.00 | 67.62 |
| \midrule Claude Code | Claude Sonnet 4.6 | Low | 39.66 | 41.67 | 40.86 | 55.00 | 44.30 |
| Claude Code | Claude Sonnet 4.6 | Medium | 50.00 | 40.97 | 47.66 | 45.00 | 45.91 |
| Claude Code | Claude Sonnet 4.6 | High | 55.17 | 33.33 | 44.96 | 60.00 | 48.37 |
| \toprule Harness | Model | Access | AS (filtered) | HA (filtered) |
| \midrule Claude Code | Claude Opus 4.7 | Ours | 53.33 | 53.19 |
| Claude Code | Claude Opus 4.7 | Official MCP | 57.78 | 59.57 |
| \midrule Claude Code | Claude Sonnet 4.6 | Ours | 46.67 | 34.04 |
| Claude Code | Claude Sonnet 4.6 | Official MCP | 57.78 | 38.30 |
| \midrule Codex CLI | GPT-5.5 | Ours | 53.33 | 18.09 |
| Codex CLI | GPT-5.5 | Official MCP | 55.56 | 25.53 |
| \toprule | GPT-5.4 | Opus 4.7 | Gemini 3.1 Pro |
|---|---|---|---|
| \midrule GPT-5.4 | 100% | 95.1% | 96.3% |
| Opus 4.7 | – | 100% | 96.6% |
| Gemini 3.1 Pro | – | – | 100% |
| \bottomrule |
| \toprule Channel | Description |
|---|---|
| \midrule bfcl-pv-01 | Prompt variation experiments on the BFCL benchmark |
| bfcl-thousand-tools | Scaling function-calling evaluation to 16–1024 tools |
| code-misalignment-capabilities | Running coding and math benchmarks on local models |
| cognitive-foundations… | Extending a cognitive foundations for reasoning paper |
| cognitive-paper-code-base | Codebase development for the cognitive paper experiments |
| hypothesis-literature-agent | Building an agent for hypothesis generation from literature |
| \toprule Category | Example query |
| \midrule Agentic Search | What are the currently remaining TODOs for Ryan in the tool-bench-langvar project? |
| Hallucination | I remember that for our prompt variation project for BFCL, changing return format to typescript + tool call tags would decrease the performance for gpt-4o, could you find how much the decrease is? |
| Summarization | I’m trying to write a recommendation letter for Marcus for his PhD application. Could you summarize his research contributions? |
| Circumstance inference | one of our team keeps taking credit for someone else’s work. who’s the credit-stealer and who’s getting taken from? |
| \bottomrule |
| \toprule Tool | Description |
| \midrule Slack Tools (7) | |
| list_channels | Lists all channels in the Slack workspace |
| get_channel_history | Fetches message history for a channel |
| get_channel_members | Lists members of a channel |
| get_thread_replies | Fetches replies in a message thread |
| get_user_info | Gets information about a user by ID |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,313 papers