RoleFlow: Self-Evolving Multi-Role Agentic Collaboration via Structural Adaptation

Yi Ge; Pan Wang; Xuehao Cui; Zian Zeng; Xu Hu; Heng Huang; Furong Huang

cs.AIOct 21, 2026

RoleFlow: Self-Evolving Multi-Role Agentic Collaboration via Structural Adaptation

Authors: Yi Ge, Pan Wang, Xuehao Cui, Zian Zeng, Xu Hu, Heng Huang, Furong Huang

Organizations: University of Maryland, College Park · University of Pittsburgh · The University of Texas at Dallas

Abstract

Large language model (LLM) based multi-agent systems often rely on predefined collaboration structures, limiting their adaptability to evolving task requirements. Although recent approaches enable dynamic orchestration, trajectory-level rewards they apply assign one outcome to each decision, ofen struggling to identify the contributions of individual structural changes. We propose RoleFlow, a self-evolving multi-role agentic system that refines collaboration structures through feedback-driven atomic edits and on-demand role synthesis. To make structural adaptation learnable, we introduce Single-Edit Counterfactual Policy Optimization (SECPO), which evaluates each edit against its no-edit counterpart at the same chain state and assigns targeted credit to edit-specific tokens. Across seven benchmarks spanning four domains, RoleFlow achieves the optimal average performance, surpassing the strongest competitor by 4.7 points. It leads on all three multi-hop QA benchmarks by 3.2-7.5 F1 points and also shows supremacy on mathematical reasoning, web interaction, and code generation. Further analyses demonstrate cross-benchmark transferability, training-data efficiency, and cross-executor generalization.

Figures & tables

Figure 1: Motivation of RoleFlow. Hand-crafted structures lack adaptability (top) , while trajectory-level rewards obscure the contributions of individual structural edits (middle) . Our RoleFlow enables effective structural evolution through same-state counterfactual evaluation and edit-level credit assignment (bottom) .

Figure 2: Overview of RoleFlow. The Global Router assembles an initial sequence of roles from the role library $\mathcal{R}$ and applies one structural edit $a_{t}\in\mathbb{E}$ upon validation failure, for at most $T_{\max}$ adaptation cycles. The role library grows through the selective retention of synthesized roles during training.

Figure 3: The single-edit advantage of SECPO. A certain chain state is executed with and without the candidate edit, and the reward difference $A^{(t)}_{i}$ (Eq. 2 ) credits that edit and gates its commitment, enabling edit-level credit assignment under evolving structural dynamics.

Figure 4: Token-level advantage of SECPO. Every plan token receives the group-relative advantage $A^{\mathrm{GRPO}}_{\ell}$ , and the single-edit advantage $A^{(t)}_{i}$ is added with weight $\lambda$ on the edit span $\mathcal{E}^{(i)}_{t}$ .

Method	Multi-hop QA			Reasoning	Coding		Web
Method	HotpotQA	2Wiki OOD	MuSiQue OOD	MATH-500	MBPP	HumanEval OOD	WebShop	Avg.
Direct Prompting
GPT-4o-mini	45.6	24.5	23.4	76.5	83.7	88.3	59.0	57.3
Claude-Haiku-4.5	47.2	26.4	28.1	92.0	90.6	96.9	59.5	63.0
Qwen2.5-7B-Instruct	30.5	18.8	11.6	77.5	76.8	82.4	58.9	50.9
Llama-3.1-8B-Instruct	32.6	15.1	9.5	53.3	70.7	71.8	57.2	44.3

Table 1: Performance comparison with direct prompting, hand-crafted pipelines, automated workflow search, trained executors and trained orchestrators on seven benchmarks. The executor LLM is consistently set as gpt-4o-mini for all multi-agent systems and trained orchestrators. We bold the best results and underline the runner-ups among all methods for direct prompting and remaining methods, respectively; OOD marks out-of-distribution benchmarks.

Inference			Training	Multi-hop QA			Coding
$\mathrm{GR}$	$\mathcal{R}$	$T_{\max}$	SECPO	HotpotQA	2Wiki OOD	MuSiQue OOD	MBPP	HumanEval OOD
✓	✓	3	✓	55.6 $\pm$ 2.2	43.5 $\pm$ 1.8	42.1 $\pm$ 1.4	89.0 $\pm$ 0.2	95.7 $\pm$ 0.9
✗	✓	3	✓	45.7 $\pm$ 1.1	29.6 $\pm$ 2.8	25.0 $\pm$ 5.0	88.1 $\pm$ 0.3	93.4 $\pm$ 0.4
✓	✗	3	✓	45.3 $\pm$ 1.4	30.6 $\pm$ 1.1	16.3 $\pm$ 0.6	86.6 $\pm$ 0.8	91.3 $\pm$ 0.9
✓	✓	1	✓	51.8 $\pm$ 2.4	36.4 $\pm$ 3.5	33.6 $\pm$ 3.0	88.1 $\pm$ 0.4	93.9 $\pm$ 1.3
✓	✓	3	✗	53.5 $\pm$ 3.9	41.0 $\pm$ 5.0	39.5 $\pm$ 3.2	89.2 $\pm$ 0.3	95.7 $\pm$ 0.9

Table 2: Ablation study of RoleFlow. GR: Global Router;

\mathcal{R}

: role library with specialist synthesis;

T_{\max}

: adaptation cycles allowed after cycle 0 (3 in RoleFlow; 1 allows a single edit). Models are trained with

K=5

Method	Configuration			Credit		Chain Executions			Training Run		Test F1 and $\boldsymbol{\Delta}$ F1 vs. GRPO
	Configuration			Credit		per Question			Tokens	GPU-h	HotpotQA		2Wiki OOD		MuSiQue OOD
	$K$	Updates	$c$	$\lambda$	Reference	Bound	Realized	Rel.	(M)		F1	$\Delta$	F1	$\Delta$	F1	$\Delta$
Group-Level Credit (GRPO) 4=0
GRPO	5	5	1	✗	Group Mean	20	0 9.1	1.00 $\times$	4.26	2.6	53.5	ref.	41.0	ref.	39.5	ref.
GRPO, Optimizer Updates $\uparrow$	5	10	1	✗	Group Mean	20	10.0	1.09 $\times$	7.33	3.4	53.4	( $-$ 0.1)	40.2	( $-$ 0.8)	39.4	( $-$ 0.1)
GRPO, Rollouts $\uparrow$ ( $K=9$ )	9	5	1	✗	Group Mean	36	16.2	1.78 $\times$	7.55	3.5	54.5	( $+$ 1.0)	41.6	( $+$ 0.6)	39.3	( $-$ 0.2)

Table 3: Efficiency comparison of SECPO with alternative credit signals and training budgets for the Global Router on HotpotQA.

c

: the number of chain executions per edit cycle;

\lambda

: whether the per-edit advantage is applied on the edit tokens; Bound: the maximum number of chain executions per question

=K,(1+c,T_{\max})

; Realized: the measured mean; Rel. : the ratio to GRPO.

Figure 5: API Cost and data efficiency of RoleFlow on HotpotQA. Panel (a) plots F1 against the API cost per question and panel (b) against the number of labeled training questions, where RoleFlow † denotes the GR trained on 100 labeled questions.

Figure 6: Performance of RoleFlow with different executor backbones on the HotpotQA, MBPP and HumanEval benchmarks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
GR backbone	Qwen2.5-7B-Instruct
Executor LLM	gpt-4o-mini
Group size $K$	5
Tasks per training step	32
Learning rate	$2\times 10^{-6}$
KL coefficient $\beta$	0.001

Appendix

Table 4: SECPO training hyperparameters.

Benchmark	Domain	Source	Train	Test	Metric
HotpotQA	Multi-hop QA	Yang et al. (2018)	200	200	F1
2WikiMultiHopQA	Multi-hop QA	Ho et al. (2020)	0	200	F1
MuSiQue	Multi-hop QA	Trivedi et al. (2022)	0	200	F1
MATH-500	Mathematical reasoning	Lightman et al. (2023)	200	500	Accuracy
MBPP	Code generation	Austin et al. (2021)	324	341	Pass@1
HumanEval	Code generation	Chen et al. (2021)	0	131	Pass@1

Appendix

Table 5: Benchmark catalog.

(a) Role Library and RoleFlowChains	Multi-hop QA			Code
	HotpotQA	2Wiki OOD	MuSiQue OOD	MBPP	HumanEval OOD
Role Library
3=0
Role Library Size $\|\mathcal{R}\|$	47.7 (4 + 43.7)			15.0 (8 + 7.0)
New Roles per Task	3.66	3.98	4.07	0.22	0.08
RoleFlowChains

Appendix

Table 6: Role library and specialist recruitment of RoleFlow. (a) Statistics of the role library

\mathcal{R}

and the role structures of the Global Router; (b) Performance on HotpotQA with specialists disabled or instantiated from generic specifications.

Role	Capabilities
Seed roles
Planner	planning, coordination, edit_proposal
Validator	validation, quality_assessment
OutputFormatter	formatting, extraction
Retriever	retrieval, wikipedia
Specialists offered by name in the GR plan prompt, synthesized when named

Appendix

Table 7: Role library specification on QA benchmarks.

Role	Capabilities
Planner	planning, task_analysis, edge_case_listing
AlgorithmDesigner	algorithm_design, complexity_analysis
Coder	code_generation, python
TestDesigner	test_design, edge_case_analysis
Debugger	debugging, code_repair
CodeReviewer	code_review, specification_check

Appendix

Table 8: Role library specification on coding benchmarks.

Figure 7: Case study of deficit-driven synthesis on a MATH-500 problem. Left: Twelve of the eighteen compared methods, including RoleFlow’s executor GPT-4o-mini, give a wrong answer. Right: In a replayed chain of RoleFlow, the validator $V$ rejects the first answer, 5040; one edit adds a role synthesized for the reported deficit, and the edited chain, executed from its first role, answers 10080, which $V$ accepts.

Figure 8: Case study comparing direct prompting with RoleFlow, whose trajectory comes from a logged run of an earlier GR checkpoint not reported in Table 1 . GPT-4o-mini prompted directly (left) answers incorrectly, and RoleFlow (right) answers correctly in cycle 0 with Planner, InformationExtractor, Reasoner, OutputFormatter and Validator, where the Validator role accepts the output and no edit follows.

Figure 9: Case study of adaptive routing with $\mathrm{GR}$ . After cycle 0 is rejected, rerunning the initial roles without $\mathrm{GR}$ (left) still fails, whereas $\mathrm{GR}$ (right) inserts a Reasoner in cycle 1 and answers correctly.

Figure 10: Case study of RoleFlow before (top) and after (bottom) training on a coding task. The trained $\mathrm{GR}$ composes Analyst, Coder and Debugger into a correct patch, whereas the untrained system produces an incorrect one.

Figure 11: Case study of RoleFlow with and without SECPO on the task of Figure 10 . Without SECPO (left) the incorrect cycle-0 patch becomes the final output, whereas with SECPO (right) cycle 1 runs Analyst, Coder and Debugger and produces a correct patch.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

Roles with Rails: Contract-Preserving Role Evolution in Multi-Agent Structured Reasoning
May 27, 2026Ling-Yue Ge, Lan-Zhe GuoMulti-Agent LLM Systems LLM Agent Orchestration
1National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China
CollabFlow: Recursive Self-Improvement of Agent Collaboration
Sep 29, 2026Xiao Huang, Mingda Zhang, Junming Zhang +5Multi-Agent LLM Systems Multi-Agent Collaboration
The Chinese University of Hong Kong, Shenzhen · Fudan University · University of Oxford
DynaFlow: Learning Free-Form Multi-Agent Orchestration
Oct 25, 2026Ruhao Liu, Qi Li, Shizun Wang +1Multi-Agent Orchestration LLM Agent Orchestration
National University of Singapore

May 27, 2026J/K move · Enter open · S save

Roles with Rails: Contract-Preserving Role Evolution in Multi-Agent Structured Reasoning

Ling-Yue Ge, Lan-Zhe Guo

1National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China

Multi-Agent LLM Systems LLM Agent Orchestration Agentic Reasoning Multi-Agent Systems Multi-Agent Reasoning

Open paper AskPDF arXiv

Role-based LLM multi-agent systems need adaptive role pools, yet adapting such systems is not merely a matter of prompt optimization: roles often carry structural obligations, including capability coverage, message compatibility, validation, final-answer aggregation, and parser-compatible output protocols. Existing systems either fix the role inventory and lose adaptivity, or allow unconstrained generation to induce role drift, removing structurally necessary roles and breaking answer contracts. We formulate this as contract-preserving role evolution, requiring every committed edit to preserve five structural contracts (capability, communication, validation, aggregation, output protocol). We instantiate this formulation in SERO, a Self-Evolving Role Orchestration framework that evolves a typed role-card pool through credit-guided retrieval, a credit-ranked communication DAG with a protected terminal aggregator and conditional validator repair, and a contextual-bandit controller whose LLM-proposed edits are committed only when they preserve the contracts and improve task score. Experiments on real-world reasoning benchmarks across three LLM backbones confirm the value of contract-preserving role evolution.

Abstract

Multi-Agent LLM Systems

Explore similar work

Roles with Rails: Contract-Preserving Role Evolution in Multi-Agent Structured Reasoning

CollabFlow: Recursive Self-Improvement of Agent Collaboration

DynaFlow: Learning Free-Form Multi-Agent Orchestration