Large language model (LLM) based multi-agent systems often rely on predefined collaboration structures, limiting their adaptability to evolving task requirements. Although recent approaches enable dynamic orchestration, trajectory-level rewards they apply assign one outcome to each decision, ofen struggling to identify the contributions of individual structural changes. We propose RoleFlow, a self-evolving multi-role agentic system that refines collaboration structures through feedback-driven atomic edits and on-demand role synthesis. To make structural adaptation learnable, we introduce Single-Edit Counterfactual Policy Optimization (SECPO), which evaluates each edit against its no-edit counterpart at the same chain state and assigns targeted credit to edit-specific tokens. Across seven benchmarks spanning four domains, RoleFlow achieves the optimal average performance, surpassing the strongest competitor by 4.7 points. It leads on all three multi-hop QA benchmarks by 3.2-7.5 F1 points and also shows supremacy on mathematical reasoning, web interaction, and code generation. Further analyses demonstrate cross-benchmark transferability, training-data efficiency, and cross-executor generalization.
Figures & tables
Figure 1: Motivation of RoleFlow. Hand-crafted structures lack adaptability (top) , while trajectory-level rewards obscure the contributions of individual structural edits (middle) . Our RoleFlow enables effective structural evolution through same-state counterfactual evaluation and edit-level credit assignment (bottom) .
Figure 2: Overview of RoleFlow. The Global Router assembles an initial sequence of roles from the role library R and applies one structural edit at∈E upon validation failure, for at most Tmax adaptation cycles. The role library grows through the selective retention of synthesized roles during training.
Figure 3: The single-edit advantage of SECPO. A certain chain state is executed with and without the candidate edit, and the reward difference Ai(t) (Eq. 2 ) credits that edit and gates its commitment, enabling edit-level credit assignment under evolving structural dynamics.
Figure 4: Token-level advantage of SECPO. Every plan token receives the group-relative advantage AℓGRPO , and the single-edit advantage Ai(t) is added with weight λ on the edit span Et(i) .
Method
Multi-hop QA
Reasoning
Coding
Web
HotpotQA
2Wiki OOD
MuSiQue OOD
MATH-500
MBPP
HumanEval OOD
WebShop
Avg.
Direct Prompting
GPT-4o-mini
45.6
24.5
23.4
76.5
83.7
88.3
59.0
57.3
Claude-Haiku-4.5
47.2
26.4
28.1
92.0
90.6
96.9
59.5
63.0
Qwen2.5-7B-Instruct
30.5
18.8
11.6
77.5
76.8
82.4
58.9
50.9
Llama-3.1-8B-Instruct
32.6
15.1
9.5
53.3
70.7
71.8
57.2
44.3
Table 1: Performance comparison with direct prompting, hand-crafted pipelines, automated workflow search, trained executors and trained orchestrators on seven benchmarks. The executor LLM is consistently set as gpt-4o-mini for all multi-agent systems and trained orchestrators. We bold the best results and underline the runner-ups among all methods for direct prompting and remaining methods, respectively; OOD marks out-of-distribution benchmarks.
Inference
Training
Multi-hop QA
Coding
GR
R
Tmax
SECPO
HotpotQA
2Wiki OOD
MuSiQue OOD
MBPP
HumanEval OOD
✓
✓
3
✓
55.6 ± 2.2
43.5 ± 1.8
42.1 ± 1.4
89.0 ± 0.2
95.7 ± 0.9
✗
✓
3
✓
45.7 ± 1.1
29.6 ± 2.8
25.0 ± 5.0
88.1 ± 0.3
93.4 ± 0.4
✓
✗
3
✓
45.3 ± 1.4
30.6 ± 1.1
16.3 ± 0.6
86.6 ± 0.8
91.3 ± 0.9
✓
✓
1
✓
51.8 ± 2.4
36.4 ± 3.5
33.6 ± 3.0
88.1 ± 0.4
93.9 ± 1.3
✓
✓
3
✗
53.5 ± 3.9
41.0 ± 5.0
39.5 ± 3.2
89.2 ± 0.3
95.7 ± 0.9
Table 2: Ablation study of RoleFlow. GR: Global Router; R : role library with specialist synthesis; Tmax : adaptation cycles allowed after cycle 0 (3 in RoleFlow; 1 allows a single edit). Models are trained with K=5 .
Method
Configuration
Credit
Chain Executions
Training Run
Test F1 and Δ F1 vs. GRPO
per Question
Tokens
GPU-h
HotpotQA
2Wiki OOD
MuSiQue OOD
K
Updates
c
λ
Reference
Bound
Realized
Rel.
(M)
F1
Δ
F1
Δ
F1
Δ
Group-Level Credit (GRPO) 4=0
GRPO
5
5
1
✗
Group Mean
20
0 9.1
1.00 ×
4.26
2.6
53.5
ref.
41.0
ref.
39.5
ref.
GRPO, Optimizer Updates ↑
5
10
1
✗
Group Mean
20
10.0
1.09 ×
7.33
3.4
53.4
( − 0.1)
40.2
( − 0.8)
39.4
( − 0.1)
GRPO, Rollouts ↑ ( K=9 )
9
5
1
✗
Group Mean
36
16.2
1.78 ×
7.55
3.5
54.5
( + 1.0)
41.6
( + 0.6)
39.3
( − 0.2)
Table 3: Efficiency comparison of SECPO with alternative credit signals and training budgets for the Global Router on HotpotQA. c : the number of chain executions per edit cycle; λ : whether the per-edit advantage is applied on the edit tokens; Bound: the maximum number of chain executions per question =K,(1+c,Tmax) ; Realized: the measured mean; Rel. : the ratio to GRPO.
Figure 5: API Cost and data efficiency of RoleFlow on HotpotQA. Panel (a) plots F1 against the API cost per question and panel (b) against the number of labeled training questions, where RoleFlow † denotes the GR trained on 100 labeled questions.
Figure 6: Performance of RoleFlow with different executor backbones on the HotpotQA, MBPP and HumanEval benchmarks.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
GR backbone
Qwen2.5-7B-Instruct
Executor LLM
gpt-4o-mini
Group size K
5
Tasks per training step
32
Learning rate
2×10−6
KL coefficient β
0.001
Appendix
Table 4: SECPO training hyperparameters.
Benchmark
Domain
Source
Train
Test
Metric
HotpotQA
Multi-hop QA
Yang et al. (2018)
200
200
F1
2WikiMultiHopQA
Multi-hop QA
Ho et al. (2020)
0
200
F1
MuSiQue
Multi-hop QA
Trivedi et al. (2022)
0
200
F1
MATH-500
Mathematical reasoning
Lightman et al. (2023)
200
500
Accuracy
MBPP
Code generation
Austin et al. (2021)
324
341
Pass@1
HumanEval
Code generation
Chen et al. (2021)
0
131
Pass@1
Appendix
Table 5: Benchmark catalog.
(a) Role Library and RoleFlowChains
Multi-hop QA
Code
HotpotQA
2Wiki OOD
MuSiQue OOD
MBPP
HumanEval OOD
Role Library
3=0
Role Library Size ∣R∣
47.7 (4 + 43.7)
15.0 (8 + 7.0)
New Roles per Task
3.66
3.98
4.07
0.22
0.08
RoleFlowChains
Appendix
Table 6: Role library and specialist recruitment of RoleFlow. (a) Statistics of the role library R and the role structures of the Global Router; (b) Performance on HotpotQA with specialists disabled or instantiated from generic specifications.
Role
Capabilities
Seed roles
Planner
planning, coordination, edit_proposal
Validator
validation, quality_assessment
OutputFormatter
formatting, extraction
Retriever
retrieval, wikipedia
Specialists offered by name in the GR plan prompt, synthesized when named
Appendix
Table 7: Role library specification on QA benchmarks.
Role
Capabilities
Planner
planning, task_analysis, edge_case_listing
AlgorithmDesigner
algorithm_design, complexity_analysis
Coder
code_generation, python
TestDesigner
test_design, edge_case_analysis
Debugger
debugging, code_repair
CodeReviewer
code_review, specification_check
Appendix
Table 8: Role library specification on coding benchmarks.
Figure 7: Case study of deficit-driven synthesis on a MATH-500 problem. Left: Twelve of the eighteen compared methods, including RoleFlow’s executor GPT-4o-mini, give a wrong answer. Right: In a replayed chain of RoleFlow, the validator V rejects the first answer, 5040; one edit adds a role synthesized for the reported deficit, and the edited chain, executed from its first role, answers 10080, which V accepts.
Figure 8: Case study comparing direct prompting with RoleFlow, whose trajectory comes from a logged run of an earlier GR checkpoint not reported in Table 1 . GPT-4o-mini prompted directly (left) answers incorrectly, and RoleFlow (right) answers correctly in cycle 0 with Planner, InformationExtractor, Reasoner, OutputFormatter and Validator, where the Validator role accepts the output and no edit follows.
Figure 9: Case study of adaptive routing with GR . After cycle 0 is rejected, rerunning the initial roles without GR (left) still fails, whereas GR (right) inserts a Reasoner in cycle 1 and answers correctly.
Figure 10: Case study of RoleFlow before (top) and after (bottom) training on a coding task. The trained GR composes Analyst, Coder and Debugger into a correct patch, whereas the untrained system produces an incorrect one.
Figure 11: Case study of RoleFlow with and without SECPO on the task of Figure 10 . Without SECPO (left) the incorrect cycle-0 patch becomes the final output, whereas with SECPO (right) cycle 1 runs Analyst, Coder and Debugger and produces a correct patch.
Role-based LLM multi-agent systems need adaptive role pools, yet adapting such systems is not merely a matter of prompt optimization: roles often carry structural obligations, including capability coverage, message compatibility, validation, final-answer aggregation, and parser-compatible output protocols. Existing systems either fix the role inventory and lose adaptivity, or allow unconstrained generation to induce role drift, removing structurally necessary roles and breaking answer contracts. We formulate this as contract-preserving role evolution, requiring every committed edit to preserve five structural contracts (capability, communication, validation, aggregation, output protocol). We instantiate this formulation in SERO, a Self-Evolving Role Orchestration framework that evolves a typed role-card pool through credit-guided retrieval, a credit-ranked communication DAG with a protected terminal aggregator and conditional validator repair, and a contextual-bandit controller whose LLM-proposed edits are committed only when they preserve the contracts and improve task score. Experiments on real-world reasoning benchmarks across three LLM backbones confirm the value of contract-preserving role evolution.
Ling-Yue Ge, Lan-Zhe Guo
1National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China
Recursive self-improvement (RSI) lets a system improve from its own outcomes; in LLM-based multi-agent systems, Agents refine one another within a task, and outcomes improve how they collaborate across tasks. However, existing multi-agent collaboration leaves this loop open: collaboration is pre-defined at the operator level, topology-only learning keeps verbatim exchange that propagates errors, and reward maximization on a system's own outcomes concentrates on a few teams. To address these challenges, we propose CollabFlow, an RSI system of Learned Agent Collaboration: a trainable Collab-Director constructs teams of complete Agents, a frozen executor runs them, and each round's outcomes retrain the director. Within each round, the edges of a collaboration graph carry protocols of Evidence-Conditioned Communication: a receiver adopts a differing answer only when the sender's evidence is stronger by a margin, so the director learns who communicates and how. Across rounds, we further propose Collaborative Trajectory Balance (CTB), a flow-based objective that credits each team once across its construction orders and targets a reward-proportional distribution over teams, so several good teams stay in play. We also bound how far this self-generated target moves between rounds, which shrinks as records accumulate. On twelve datasets, CollabFlow outperforms all baselines and keeps improving across rounds. Code is available at https://anonymous.4open.science/r/CollabFlow-631E.
Xiao Huang, Mingda Zhang, Junming Zhang +5
The Chinese University of Hong Kong, Shenzhen · Fudan University · University of Oxford
Multi-agent systems often rely on predefined collaboration structures, limiting their ability to adapt to diverse task requirements. We introduce DynaFlow, a framework that integrates free-form executable workflow synthesis into the action space of the main agent. The agent decides when collaboration is needed and generates task-specific programs that coordinate sub-agents through sequential or parallel execution, structured information exchange, and verification. Execution results and traces inform subsequent actions, enabling the agent to adapt collaboration as reasoning progresses. We train only the main agent through supervised fine-tuning on distilled trajectories, followed by refinement on examples filtered for unsupported premises, biased verification, and information-flow defects. All sub-agents remain fixed, separating orchestration learning from executor capability improvement. Across six benchmarks spanning mathematical reasoning, scientific question answering, and multi-hop retrieval, DynaFlow achieves 88.08% overall accuracy in Forced mode, outperforming the strongest evaluated baseline by 10.90 percentage points with the same Qwen3.5-27B backbone. Workflow analysis finds more explicit use, verification, and refinement of upstream outputs after training, highlighting cross-stage information flow as a key dimension of multi-agent orchestration.