Collective Reasoning, Single-Writer Execution: Fusion-MoA for Long-Horizon Agents
Authors: Zongyou Yang, Yinghan Hou, Botao Su, Ercheng Zhao
Organizations: Dyson School of Design Engineering, Imperial College London, London, United Kingdom · Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom
Mixture-of-Agents systems combine several language models into a stronger response, but tool-using agents act over long horizons in which every step changes external state and the history keeps growing. We present Fusion-MoA, a serving architecture that brings multi-model collaboration into this loop through executor-anchored consultation. One Executor model holds the full context and emits every text and tool call; when consultation is admitted, three specialist models review its proposed next action through a compact view under a shared 120-second deadline. The fleet sits behind one OpenAI-compatible endpoint, so an unmodified agent client drives it as a single model. Fusion-MoA serves an in-car assistant and an in-car personal agent for intelligent cockpits and is deployed on MetaX C550 servers. Evaluated as four quantized open-weight models on five NVIDIA L40S GPUs and driven by Claude Code, it passes 78 of the 89 Terminal-Bench 2.1 tasks, 16 more than its Executor alone and 29 more than a self-hosted quantized Kimi K3 on the same client. On the 63 tasks that both it and the API-served LongCat-2.5-Preview pass, it needs 14.5 minutes on average against 17.5. Specialist reviews use 0.4% of the Executor's tokens, and every tool call comes from the Executor.
Figures & tables
Fig. 1: Fusion-MoA. (a) An unmodified client sends the history Ht and tool schema Tt to one tool-calling API. D12, the Executor, holds the full context and returns every response at ; the client executes tool calls and appends observations ot , and the official verifier scores the final task artifacts. When the admission predicate gt fires, D12 writes a memo mt 1 , the service builds the review view zt 2 , three specialists review it in parallel 3 , and the memo and completed reviews Rt return to D12 as read-only advice 4 . (b) Specialist output reaches the client only through a new D12 generation. (c) The review view keeps three slots while Ht grows. (d) Memo and reviews share one 120-second deadline; late or truncated reviews are dropped.
Cell
Model; quantization; GPUs
Role
D12
Qwen3.8-Flash-Next; IQ3_S; 2
Executor
D16
Qwopus3.8-27B-Flash-V2; Q8_0; 1
Implementation
D9
Qwopus3.6-27B-v2; Q6_K; 1
Alternatives
D13
Muse-Glimmer-30B; Q4_K_XL; 1
State check
TABLE I: Cells of the deployed Profile.
Workload
Typical requests
Fusion-MoA path
In-car assistant
Question answering, travel and schedule lookups
One model API; fast Executor path
In-car personal agent
Multi-step trip planning, cross-service information processing, tool-execution workflows
On-demand consultation; long-horizon state; single tool-action writer
TABLE II: Intelligent-cockpit workloads served by Fusion-MoA.
Fig. 2: Accuracy and time on Terminal-Bench 2.1. (a) Tasks passed against median Agent minutes per solved task. (b) Median Agent minutes on the tasks that Fusion-MoA and each reference both pass; n is the number of such tasks. (c) Mean prompt tokens per Executor call and per specialist review for each of the 89 tasks, sorted by Executor context.
Run
PASS
U
Median
Mean
Claude Code
Fusion-MoA
78
0
5.9
15.2
Executor only (D12)
62
2
3.7
8.4
Kimi K3 (self-hosted)
49
1
4.3
7.3
LongCat-2.5-Preview
69
0
10.8
21.9
Terminus-2
TABLE III: Official outcomes and Agent time on the 89 tasks. U: unscored. Median, Mean: Agent minutes per passed task. With the runtime assets of six tasks restored, DeepSeek, HY3, and LongCat-2.0 pass 62, 56, and 46 (Appendix B ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Fusion-MoA
Claude Code references
Terminus references
Agent
Claude Code 2.1.220
Claude Code 2.1.220
Terminus 2.0.0
Agent allowance
Original task limits; given the speed of the local hardware, tasks not resolved within them were run again with a longer allowance
Kimi K3: original limits; LongCat-2.5-Preview: extended
10,800 s per task; HY3 also extended; Qwen3.6 original limits
Concurrency
One task
One task per model
Two tasks per model (Qwen3.6: six)
Sampling
D12 T=0.6 , p=0.95
Provider default; maximum effort
T=0.7 , p=1.0
Request output cap
D12 16,384 tokens
Client default
8,192 tokens
Model access
Local GPUs
Kimi K3: quantized deployment on in-house B200 GPUs; LongCat-2.5-Preview: official API; both through a relay
DeepSeek, HY3, and Qwen3.6: self-deployed; LongCat-2.0: official API
Appendix
TABLE IV: Protocol details for the runs in Table III .
Run
Exceptions
PASS with one
Fusion-MoA
13
7
Executor only
20
4
Kimi K3
15
1
LongCat-2.5-Preview
11
4
DeepSeek V4 Flash (Terminus)
8
0
HY3 (Terminus)
11
0
Appendix
TABLE V: Exception diagnostics overlap official outcomes; they are not extra outcome categories.
Measure
Value
Rounds: task entry / recall
89 / 133
Tasks with a recall round
51 of 89
Memo delivered
186 of 222
Rounds with ≥ 2 usable reviews
181 of 186
Round time, median / p90
43.3 / 120.0 s
Rounds completed before 120 s
191 of 222
Appendix
TABLE VI: Consultation path in the Fusion-MoA runs behind the outcomes in Table III (89 runs). A review is no-op when it is exactly the role’s no-op reply; every other usable review is substantive. Three further rounds were in flight when the Agent limit ended their session, and specialist requests include those rounds. In 36 rounds the memo fell back to direct execution (27 at the deadline, 9 at the context limit). Of the 133 recalls, 112 came from the periodic check.
Task
Review
Advice
Response
R00
protein-assembly
D16, D9
departs
partly
R01
build-cython-ext
D9
departs
kept plan
R02
build-cython-ext
D9
matches
same plan
R03
crack-7z-hash
D9
matches
other step
R04
git-multibranch
D9
departs
kept plan
R05
mailman
D9
departs
kept plan
Appendix
TABLE VII: Consultation rounds with substantive advice in the 89 runs, in log order.
Specialist
Candidate model
Role in review
General reasoning
General open-weight LLM
Reasoning on complex tasks
Vehicle domain
Automotive-domain fine-tuned model
Vehicle knowledge, control intents, feature constraints
Trip planning
Planning-tuned model
Multi-step itinerary planning
Tool use
Tool-use fine-tuned model
API argument checks, tool-execution suggestions
Appendix
TABLE VIII: Example specialist roles for an intelligent cockpit (illustrative configurations, not evaluated here).
Executor and client
Allowance
PASS
FAIL
Infra
Qwopus3.6-27B-Coder, Terminus 2
14,400 s
45
27
17
Qwen3.8-27B, OpenCode (medium)
18,000 s
42
42
5
Qwen3.8-27B, OpenCode (xhigh)
18,000 s
31
49
9
Qwen3.8-27B, Claude Code (xhigh)
18,000 s
31
29
29
Appendix
TABLE IX: Earlier Fusion-MoA runs on the 89 tasks, one run per task. Infra: unscored or infrastructure error.
Autonomous agents powered by large language models (LLMs) perform complex tasks through long-horizon reasoning and tool interaction, where a fundamental trade-off arises between execution efficiency and reasoning robustness. Models at different capability-cost levels offer complementary advantages: lower-cost models enable fast execution but may struggle on difficult reasoning segments, while stronger models provide more robust reasoning at higher computational cost. We present AgentCollab, a self-driven collaborative inference framework that dynamically coordinates models with different reasoning capacities during agent execution. Instead of relying on external routing modules, the framework uses the agent's own self-reflection signal to determine whether the current reasoning trajectory is making meaningful progress, and escalates control to a stronger reasoning tier only when necessary. To further stabilize long-horizon execution, we introduce a difficulty-aware cumulative escalation strategy that allocates additional reasoning budget based on recent failure signals. In our experiments, we instantiate this framework using a two-level small-large model setting. Experiments on diverse multi-step agent benchmarks show that AgentCollab consistently improves the accuracy-efficiency Pareto frontier of LLM agents. The code is available at https://github.com/SparkEngineAI/AgentFlashInfer.
Mixture-of-Agents (MoA) systems improve reasoning accuracy by routing each query to multiple expert LLMs and aggregating their outputs. Efficiently executing this workload on limited GPU resources has bottlenecks. Skill-based routing creates skewed expert demand, and combining instruction-tuned LLMs with long-reasoning models results in extreme variability in generation lengths. Consequently, traditional scheduling strategies suffer from significant GPU idling and throughput collapse due to load imbalances. We present MOSAIC, a scheduling framework for accelerating batch-oriented MoA workloads. MOSAIC uses two-stage workload-aware scheduling: an expert placement stage, where an integer linear program (ILP) decides expert placement, followed by a query-sequencing stage that schedules requests based on predicted output lengths. Additionally, MOSAIC uses confidence-aware adaptive aggregation, leveraging inter-expert agreement to bypass the heavy aggregator LLM for consensus queries. On a 4-GPU system, MOSAIC achieves 2.7-3.4x end-to-end speedups over round-robin scheduling and reduces the mean time-to-answer by 3.9-7.8x, while matching accuracy within +/-1pp.
Saptarshi Mitra, Yifan Zhang, Rachid Karami +5
University of California, Irvine, USA · University of Southern California, Los Angeles, USA · Intel, USA
Specializing Large Language Models (LLMs) toward distinct abilities underpins successes ranging from personalized assistants to multi-agent systems (MAS). Single-agent paradigms rely on pre-defined personas or steering vectors to induce specialization, yet they impose a single fixed specialization that fails to adapt to diverse queries. Conversely, MAS achieves dynamic multi-perspective problem solving by orchestrating agents with distinct text-based roles, but fusing these specializations requires multi-turn interactions that inflate context length and inference cost. To address these limitations, we propose Mixture of Roles (MoRe), which adaptively composes multiple specializations into a single steering vector for single-turn inference. Specifically, MoRe learns a diversified codeboox of steering vectors, each of which encodes a latent role. A query-aware router dynamically fuses the codebook into a steering vector that encompasses multiple roles. By steering the backbone LLM with the composed vector, MoRe enables multi-perspective specialization in a single-agent, single-turn inference process. The proposed MoRe can be efficiently trained via a three-stage SFT curriculum and GRPO post-training, while the backbone LLM remains frozen. Experiments across reasoning and personality benchmarks show that MoRe outperforms single-agent baselines by 2.2% on average, and achieves performance on par with MAS while reducing token cost by 20x.