TandemBench: Benchmarking Decision Models for Multi-Agent Collaboration
Organizations: Harbin Institute of Technology · Peng Cheng Laboratory
Abstract
Decision models choose among predefined options in tasks such as routing and action selection. In multi-agent collaboration, a decision determines how members divide work, exchange resources, and act in parallel. We introduce TandemBench, a benchmark of 1,200 questions in multi-agent cooking scenarios where coordination choices affect goal completion. Models receive complete global information and candidate team plans, with reference answers established through execution and exact search. Models select an optimal team plan and, in paired variants that retain or remove it, judge whether better arrangements exist beyond the supplied options. We evaluate 40 native decision models and compare them with 10 general-purpose language models. The best native decision model achieves 62.50% decision accuracy. Native decision models remain close to chance in optimality verification. Paired tests reveal that some report better alternatives regardless of whether an optimal plan is already present. These findings show that team-plan decision accuracy alone can conceal systematic errors in judging whether the supplied options contain an optimal arrangement. The dataset and evaluation code are publicly available at https://github.com/lililibaohang/TandemBench.
Figures & tables
| Rank | Model | Params. (B) | DA | VA |
|---|---|---|---|---|
| 1 | KnowLine-31B-Gen1 | 31 | 62.50 | 51.44 |
| 2 | Basal-Omni-1.0-Ultra | 31 | 60.75 | 51.97 |
| 3 | Quyet-1.0-Large | 31 | 60.75 | 52.06 |
| 4 | DOPEST-1.0-31B | 31 | 60.38 | 52.16 |
| 5 | PPLX-Decider-v1.2 | 27 | 60.38 | 55.31 |
| 6 | PPLX-Decider-v1.1-27B | 27 | 59.44 | 54.28 |
| Rank | Model | Avg. output tokens | DA | VA |
|---|---|---|---|---|
| 1 | Qwen3.8-27B | 8652 | 82.80 | 71.88 |
| 2 | Gemma4-31B | 1072 | 73.11 | 57.03 |
| 3 | Gemma4-26B-A4B | 1516 | 60.58 | 53.66 |
| 4 | Muse-Glimmer-30B | 1415 | 59.64 | 60.35 |
| 5 | Qwen3-32B | 925 | 46.33 | 49.45 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Plan | Program ID | Stage | Remaining | Total |
|---|---|---|---|---|
| A | opt04 | 13 | 8 | 21 |
| B | opt01 | 10 | 3 | 13 |
| C | opt02 | 7 | 8 | 15 |
| D | opt03 | 9 | 6 | 15 |
| Unit or property | Count |
|---|---|
| Recipe tasks | 30 |
| Decision scenarios | 200 |
| Short-/long-horizon question pairs | 200 |
| Total evaluation questions | 1,200 |
| Base team-plan decision questions | 400 |
| Optimality-verification variants | 800 |
| Rank | Model | Avg. output tokens | DA | VA |
|---|---|---|---|---|
| 1 | Qwen3.8-27B | 8652 | 82.80 | 71.88 |
| 2 | Gemma4-31B | 1072 | 73.11 | 57.03 |
| 3 | Gemma4-26B-A4B | 1516 | 60.58 | 53.66 |
| 4 | Muse-Glimmer-30B | 1415 | 59.64 | 60.35 |
| 5 | Qwen3-32B | 925 | 46.33 | 49.45 |
| 6 | Gemma4-E4B | 2564 | 45.03 | 51.16 |
| Model | Feasible (%) | Noncompletion (%) | Regret | Consistency (%) | Range (pp) |
|---|---|---|---|---|---|
| Basal-Omni-1.0-Ultra | 99.75 | 0.25 | 1.44 | 31.75 | 11.25 |
| KnowLine-31B-Gen1 | 99.94 | 0.06 | 1.68 | 45.25 | 11.75 |
| Quyet-1.0-Large | 98.75 | 1.25 | 1.22 | 37.25 | 25.75 |
| PPLX-Decider-v1.1-27B | 100.00 | 0.00 | 1.74 | 37.00 | 9.00 |
| Torchcast-4B-v1.2 | 95.69 | 4.31 | 1.25 | 43.25 | 2.00 |
| CLM-8B | 78.56 | 21.44 | 4.50 | 92.00 | 1.00 |
| Model | Same-plan short | Same-plan long | Conflict DA | Fast-stage choice |
|---|---|---|---|---|
| Basal-Omni-1.0-Ultra | 72.27 | 65.23 | 35.00 | 36.67 |
| KnowLine-31B-Gen1 | 72.27 | 59.77 | 55.00 | 0.00 |
| Quyet-1.0-Large | 56.64 | 61.33 | 48.33 | 10.00 |
| PPLX-Decider-v1.1-27B | 58.20 | 57.81 | 45.00 | 35.00 |
| Torchcast-4B-v1.2 | 55.47 | 47.66 | 73.33 | 15.00 |
| CLM-8B | 4.30 | 5.86 | 0.00 | 66.67 |
| Model | Item-weighted | Task-macro |
|---|---|---|
| Basal-Omni-1.0-Ultra | 60.75 | 65.45 |
| KnowLine-31B-Gen1 | 62.50 | 66.96 |
| Quyet-1.0-Large | 60.75 | 61.00 |
| PPLX-Decider-v1.1-27B | 59.44 | 62.90 |
| Torchcast-4B-v1.2 | 58.94 | 56.95 |
| CLM-8B | 15.44 | 10.52 |
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,571 papers