Existing Multi-Agent Systems (MAS) typically rely on homogeneous model configurations, failing to exploit the diverse expertise inherent in different post-trained architectures. We propose Team-of-Thoughts, a heterogeneous MAS framework that treats diverse models as specialized tools within an orchestrator-driven paradigm. Team-of-Thoughts introduces two novel components: (1) Orchestrator Calibration, which identifies models with superior coordination and synthesis capabilities, and (2) Agent Self-Assessment, a protocol where tool agents profile their own domain-specific strengths to guide selection. At inference, the orchestrator dynamically activates the most compatible agents based on these profiles to maximize capability coverage. Across five mathematical reasoning and code generation benchmarks, Team-of-Thoughts consistently outperforms individual models and existing MAS baselines. Notably, on AIME24 and LiveCodeBench, Team-of-Thoughts achieves 96.00% and 77.91% accuracy, respectively, significantly improving over homogeneous role-play baselines (80.00% and 65.93%).
Figures & tables
Figure 1: Pass@5 accuracy versus cost on MBPP+ for Team-of-Thoughts, single models, and AgentVerse. The dashed line denotes the Pareto front of single models.
Figure 2: Overview of Team-of-Thoughts. (Left) Unlike single-model reasoning or homogeneous MAS (role-play), Team-of-Thoughts leverages heterogeneous model priors to maximize solution space coverage. (Right) Our architecture utilizes a pre-inference pipeline—comprising orchestrator calibration and agent self-profiling—to enable dynamic agent selection. During inference, the orchestrator selectively invokes optimal tool agents and synthesizes their outputs into a high-confidence final response.
Figure 3: Schematic comparison of Chain-of-Thought (CoT) and Team-of-Thoughts. (Left) Standard agentic reasoning via CoT generates sequential intermediate steps to refine the prediction distribution heuristically. However, on complex tasks, CoT may fail to converge on distant targets. (Right) Team-of-Thoughts utilizes heterogeneous tool agents to explore a broader solution space. During inference, these agents refine their individual distributions through local reasoning, which an orchestrator then prioritizes and aggregates into the final output.
AIME 2024
AIME 2025
Humaneval+
MBPP+
LiveCodeBench v6
Model
pass@1
@3
@5
@1
@3
@5
@1
@3
@5
@1
@3
@5
@1
@3
@5
Claude Sonnet 4.5
84.00
86.67
86.67
74.67
87.67
90.00
95.61
96.34
96.34
81.80
84.29
85.45
48.24
55.33
57.69
GPT-5-mini
88.67
93.33
93.33
73.33
83.50
86.11
92.68
94.88
95.12
82.38
86.19
87.57
64.84
74.40
77.20
Gemini 3 Flash
86.67
90.67
93.33
71.33
81.33
86.67
96.46
96.95
96.95
84.44
85.13
85.45
72.53
75.71
77.47
Deepseek v3.2 Exp
94.67
96.67
96.67
94.00
98.67
100.00
91.26
95.88
96.65
80.20
84.59
85.83
76.48
86.65
88.46
GPT OSS 20b
75.33
88.00
90.00
73.33
83.50
86.11
89.23
95.12
95.63
79.23
84.88
86.29
46.04
54.34
58.24
Table 1: Performance comparison of Team-of-Thoughts across five benchmark tasks. Success rates (%) are reported using pass@1, 3, and 5. Team-of-Thoughts is evaluated against individual model baselines and state-of-the-art multi-agent methods. The Theoretical Limit represents the oracle upper bound achieved by selecting the best result among all individual models for each problem.
Model
AIME2024
MBPP+
Budget Cost ($)
0.03
0.02
Avg
0.03
0.02
Avg
Claude Sonnet 4.5
86.67
46.67
66.67
78.38
78.38
78.38
GPT-5 Mini
86.67
93.33
90.00
86.49
83.78
85.14
Gemini 3 Flash
86.67
40.00
63.34
83.78
83.78
83.78
DeepSeek v3.2
93.33
93.33
93.33
81.08
81.08
81.08
GPT-OSS 20B
80.00
80.00
80.00
75.68
75.68
75.68
Table 2: Orchestration performance across candidate models and budget constraints. Calibration accuracy (%) is evaluated on AIME2024 and MBPP+ benchmarks under two distinct budgetary tiers (USD). Avg denotes the mean accuracy across all evaluated settings.
Benchmark
Single
Self-Assess
Orch-Based
Random
AIME2024
94.67
96.00
92.67
94.00
MBPP+
82.38
85.93
85.77
84.44
Table 3: Performance evaluation of tool agent assessment strategies. Comparison of Pass@1 accuracy (%) with no tool agent (“Single”), Orchestrator-Led Assessment, Agent Self-Assessment, and Random Invocation across AIME2024 and MBPP+ benchmarks.
AIME 2025
MBPP+
Config
pass@1
@3
@5
pass@1
@3
@5
No traces
95.33
99.33
100.00
85.93
88.84
89.68
Include traces
78.67
93.33
96.67
80.90
85.61
87.04
Table 4: Performance of orchestrator aggregation strategies. Comparison of task accuracy (%) achieved when incorporating versus excluding the intermediate reasoning traces of tool agents.
Activated tools
Pass@1
Pass@3
Pass@5
2
82.75%
85.58%
86.77%
4
82.96%
85.32%
85.98%
8
60.32%
83.76%
86.24%
16
60.63%
83.10%
85.71%
Table 5: Performance under different numbers of activated tool agents, with 25 total tool agents available.
Activate 2
Activate 8
Agreement
Avail Tools
Pass@1
@3
@5
Pass@1
@3
@5
A (%)
10
82.70
86.32
87.57
74.97
85.26
85.98
90.74
25
82.75
85.58
86.77
60.32
83.76
86.24
91.80
50
82.43
86.59
87.83
54.76
82.17
86.51
61.64
75
82.49
87.09
88.36
63.65
84.21
86.24
75.13
100
82.49
86.56
87.57
61.43
84.13
86.51
71.43
Table 6: Orchestrator performance and tool selection consistency (Agreement) across varying ensemble sizes. Performance is compared between configurations activating two and eight agents per tool call.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Construction Method
pass@1
pass@3
pass@5
Random Sample
77.91%
85.05%
87.91%
Categorized Sample
81.65%
85.93%
86.81%
Appendix
Table 7: Performance in pass@k on LiveCodeBench v6 using different calibration construction strategies.
Figure 4: Comparing the times of tool agent invocations between two calibration dataset construction methods.
Orch context len
Pass@1
Pass@3
Pass@5
16K
60.32%
83.76%
86.24%
32K
59.79%
82.99%
85.71%
64K
59.42%
83.39%
86.51%
128K
60.32%
82.96%
85.45%
Appendix
Table 8: Performance under different orchestrator max context lengths, with eight of 25 available tool agents activated on each tool call.