Roles in multi-agent large language model (LLM) systems allocate decisions as well as work. In a Planner--Executor--Reviewer team, the Reviewer's approval can determine when the team submits an answer and receives corrective feedback. We study both functions by varying role instructions while holding agents, topology, call budget, and ground-truth evaluator constant, using three open-weight models on Omni-MATH and two on SciBench. Removing roles largely eliminates functional specialization without lowering first-pass accuracy. After a failed first answer, however, submission decisions matter: a role-structured Reviewer often withholds approval, so its team submits less often and receives fewer evaluator verdicts. In an exploratory comparison with feedback available to both teams, the role-free team recovers more often across all five model--benchmark settings. Two further analyses support the importance of feedback opportunities. In a post hoc comparison at equal numbers of verdicts, the pooled recovery difference is small (role-structured minus role-free: +0.009, 95% interval [−0.014,+0.033]). In an exploratory intervention on 400 Omni-MATH problems with gemma-4, keeping the role prompts but forcing submission in every cycle raises recovery from 0.198 to 0.613; the paired comparison detects no difference from the role-free team. The takeaway is to evaluate roles through recovery after failure as well as specialization. For systems with reliable external feedback, this requires measuring who may submit, how often the team receives feedback, and which agents can act on it.
Figures & tables
Figure 1. One deliberation cycle and where the feedback enters. A trajectory (one problem’s run, at most six cycles) advances one cycle per pass of Planner, Executor and Reviewer. The third agent’s approval (the bar marks this gate) is the only route to the evaluator, which knows the reference answer and returns PASS, ending the trajectory, or FAIL with a repair hint; without approval nothing is submitted and the team deliberates internally in the next cycle. The key gives where the feedback enters per condition: the advice slot of each role prompt for the role-structured team (Planner, Executor and Reviewer prompts) and the structurally enforced one (the same roles imposed by typed message formats); for the role-free team (one generic prompt for all three agents, with no slot), an [Evaluator] entry in the shared history, or nowhere in the feedback-withheld diagnostic. A trajectory that never submits is graded once when the budget runs out. A diagram of one deliberation cycle: Planner, Executor and Reviewer in sequence, then an arrow marked approve and submit to the Evaluator. A line from the Evaluator returns the FAIL verdict and repair hint to all three agents in the next cycle; a dashed line from the Reviewer back to the Planner marks a cycle without approval, in which nothing is submitted. Below, a table lists for each of the four conditions who approves submission and through which channel the feedback reaches the agents.
functional collapse
adjudicated no division
actor
RS
RF -
RF
RS
RF -
gemma-4-31B
0.000
0.976
0.974
0.000
0.857
gpt-oss-120b
0.120
0.846
0.883
0.000
1.000
Llama-3.3-70B
0.000
0.872
0.900
0.000
1.000
Table 1. Functional collapse on the Omni-MATH benchmark. Share of first cycles (one pass of Planner, Executor and Reviewer) in which all three agents assert their own final answer, per actor (the model playing all three agents). RS: role-structured (Planner, Executor and Reviewer prompts); RF - and RF: role-free (one generic prompt for all three agents) with the evaluator’s feedback withheld or shown. Adjudicated no division : the share of a judged sample of first cycles in which a model judge with a fixed rubric finds no division of labor; the judge did not see the grade but could infer the condition. The byte-duplication lower bound, the enforced condition and a sensitivity analysis over answer-marker definitions are in the supplement.
Figure 2. Roles organize the team; the Reviewer’s gate limits feedback and recovery. Conditions: role-structured (Planner, Executor and Reviewer prompts); role-free (one generic prompt for all three agents) with the evaluator’s feedback shown, or withheld as a diagnostic (hollow); structurally enforced (roles imposed by typed message formats; Omni-MATH only). Actors are the models playing all three agents. (A) First-pass accuracy: whether a team’s first submitted answer is correct. (B) Recovery: the share of first-pass-wrong trajectories (runs whose first graded answer was wrong) that finish correct, each condition over its own set; brackets give the exact McNemar p for role-structured against role-free with feedback, paired on problems both first answer wrongly. (C) How gemma-4’s first-pass-wrong Omni-MATH trajectories spend the six-cycle budget: cycles ending in a submission (the third agent approved) or internal cycles without submission (it declined). The bottom row is the exploratory forced-submission run on its own 400 -problem subset: role prompts kept, the answer submitted in every cycle. (A) is over trajectories with a first verdict; in (B) trajectories that never finish count as unrecovered; error bars are bootstrap 95% intervals. Three panels. A: first-pass accuracy per actor and domain, about equal for the role-structured team and both role-free teams except gpt-oss on Omni-MATH, where the role-free teams are higher. B: recovery after a failed first answer; the role-free team with feedback recovers most, the role-structured and enforced teams less, and the role-free team whose feedback is withheld least, tying the role-structured team for Llama-3.3. C: stacked bars of deliberation cycles that end in a submission versus internal cycles without submission for gemma-4 in four conditions, and a fifth bar for the forced-submission run, which submits in every cycle.
recovery
recovered only by (paired)
verdict-matched
final accuracy
submissions
RS never
actor, domain ( n )
RS
RF
RF -
n
RS
RF
p
RS
RF
p
RS
RF
p
RS
RF
submits
gemma-4, Omni-MATH ( 1,600 )
0.222
0.609
0.051
236
4
89
6.2×10−22
0.165
0.153
0.69
0.846
0.921
1.7×10−24
1.27
3.20
84/316
gpt-oss, Omni-MATH ( 1,600 )
0.324
0.552
0.247
319
17
101
1.0×10−15
0.229
0.216
0.72
0.803
0.894
2.7×10−28
2.10
3.53
51/463
Llama-3.3, Omni-MATH ( 400 )
0.014
0.147
0.015
250
1
26
4.2×10−7
0.012
0.008
1.0
0.315
0.415
1.6×10−7
0.32
2.71
205/276
gemma-4, SciBench ( 574 )
0.254
0.419
0.051
100
7
21
0.013
0.230
0.190
0.48
0.847
0.880
0.0054
1.95
3.62
3/118
gpt-oss, SciBench ( 574 )
0.315
0.483
0.231
93
7
18
0.043
0.215
0.247
0.68
0.852
0.892
1.9×10−4
2.68
3.95
1/124
Table 2. Role-structured against role-free with feedback, in every cell (one actor, the model playing all three agents, on one benchmark; n problems). RS: role-structured (Planner, Executor and Reviewer prompts); RF: role-free (one generic prompt for all three agents) with the evaluator’s feedback shown; RF - : role-free with it withheld. A trajectory (one problem’s run) is first-pass wrong when its first submitted answer is graded wrong. Recovery : share of first-pass-wrong trajectories that finish correct, each condition over its own set. Recovered only by : on the n problems both RS and RF first answer wrongly, those recovered by one condition and not the other, with the exact McNemar p for the two counts. Verdict-matched (post hoc): the same paired recovery with each RF trajectory cut off after as many evaluator verdicts (PASS or FAIL decisions) as RS received on that problem, so both get the same number of verdicts and hints. Final accuracy : correct at the end, over all problems, paired; never-graded trajectories count as incorrect. Submissions : answers the team submits (third-agent approvals) per first-pass-wrong trajectory, excluding the single end-of-budget grading. RS never submits : RS first-pass-wrong trajectories with no submission in six cycles, graded once at the end.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
functional
exact byte
adjudicated
actor
collapse
duplication
no division
role-structured (RS)
gemma-4-31B-it
0.000
0.000
0.000
gpt-oss-120b
0.120
0.000
0.000
Llama-3.3-70B-Instruct
0.000
0.000
0.000
role-free, feedback withheld (RF - )
Appendix
Table S1. Functional collapse on the Omni-MATH benchmark under three measures, with the structurally enforced condition and a sensitivity analysis over answer markers.
Figure S1. Which agent changes when the role clauses are deleted. For each actor, the share of first cycles in which the Planner, Executor and Reviewer assert their own final answer under role-structured prompts (filled) and under role-free prompts with the evaluator’s feedback withheld (hollow), on the Omni-MATH benchmark. The Planner moves from almost never answering to almost always answering; the Executor answers under both; the Reviewer moves less. Three panels, one per actor. Each shows three horizontal dumbbells, for the Planner, Executor and Reviewer, from the role-structured value to the role-free value of the share of first cycles in which that agent asserts its own final answer. The Planner's dumbbell spans almost the whole axis in every panel; the Executor's is near one at both ends; the Reviewer's is short.
Marginal population: first pass paired; recovery on each condition’s own denominator
first pass
recovery ∣ first verdict wrong
actor
RS
RF -
difference
RS ( n )
RF - ( n )
difference
gemma-4-31B-it
0.802
0.802
+0.0000[−0.0156,+0.0150]
0.222 (316)
0.051 (316)
−0.1709[−0.2247,−0.1171]
gpt-oss-120b
0.713
0.763
+0.0503[+0.0339,+0.0666]
0.324 (463)
0.247 (380)
−0.0766[−0.1361,−0.0153]
Llama-3.3-70B-Instruct
0.308
0.316
+0.0077[−0.0283,+0.0437]
0.014 (276)
0.015 (268)
+0.0004[−0.0214,+0.0192]
Paired population: problems first-pass wrong under both conditions; one shared denominator
Appendix
Table S2. First-pass accuracy and recovery with roles and without them (feedback withheld) on the Omni-MATH benchmark, on both populations.
first pass
recovery
paired recovery
final
paired final
Omni-MATH
role-structured (frozen)
0.802
70/316=0.222
–
0.846
–
role-free, with feedback, run 1
0.802
193/317=0.609
89 vs 4 , p=6.2×10−22
0.921
137 vs 17 , p=1.7×10−24
role-free, with feedback, run 2
0.811
177/302=0.586
95 vs 7 , p=7.8×10−21
0.922
139 vs 18 , p=2.4×10−24
SciBench
role-structured (frozen)
0.794
30/118=0.254
–
0.847
–
Appendix
Table S3. The two gemma-4-31B-it cells of the role-free condition with feedback, run twice, against the frozen role-structured cell.
RS declining cycles
clause
RS never
free-cycle approval
by 2nd verdict
per later verdict
approvals
actor, domain
before
FAIL
free
share
submits
RS
RF
RS
RF
RS
RF
RS
RF
gemma-4-31B-it, Omni-MATH
2.24
0.81
1.20
0.19
84/316
0.25
0.96
0.206
0.438
0.41
0.28
1.27
3.20
gpt-oss-120b, Omni-MATH
1.33
1.17
0.71
0.37
51/463
0.53
0.68
0.246
0.296
0.27
0.22
2.10
3.53
Llama-3.3-70B-Instruct, Omni-MATH
4.49
0.25
0.88
0.04
205/276
0.01
0.82
0.014
0.092
0.25
0.08
0.32
2.71
gemma-4-31B-it, SciBench
0.55
1.38
1.48
0.40
3/118
0.35
0.99
0.237
0.239
0.26
0.16
1.95
3.62
gpt-oss-120b, SciBench
0.41
1.53
0.60
0.60
1/124
0.67
0.78
0.258
0.317
0.19
0.16
2.68
3.95
Appendix
Table S4. Where the role-structured Reviewer declines, per cell, and what each team’s allocation of the budget buys.
Figure S2. Recovery follows exposure to the evaluator. Top: the share of first-pass-wrong trajectories that receive a k -th evaluator verdict. Bottom: the cumulative share recovered by the k -th verdict, with the final value labeled. One panel per cell (actor on benchmark): role-structured (purple), role-free with feedback (rose) and role-free with the feedback withheld (hollow, dashed). The role-structured recovery curve flattens as fewer trajectories receive further verdicts; these curves do not isolate how effectively each team uses a verdict. Two rows of five panels, one column per cell. Top row: the share of first-pass-wrong trajectories that receive a k-th verdict, falling steeply for the role-structured team, more slowly for the role-free team with feedback, and hardly at all for the role-free team without feedback. Bottom row: cumulative recovery by the k-th verdict; the role-structured curve flattens after the second verdict while the role-free curve with feedback keeps rising.
cell
paired n
forced ties
RS never submitted
problems left
held-answer recodes (RS / RF)
gemma-4-31B-it, Omni-MATH
236
141
66
95
12 / 3
gpt-oss-120b, Omni-MATH
319
84
36
235
4 / 0
Llama-3.3-70B-Instruct, Omni-MATH
250
242
187
8
2 / 1
gemma-4-31B-it, SciBench
100
29
2
71
2 / 0
gpt-oss-120b, SciBench
93
11
1
82
0 / 0
Appendix
Table S5. Robustness of the verdict-matched recovery contrast: forced ties, six definitions of the contrast, and per-verdict pass rates.
Figure S3. Two robustness checks. (a) The verdict-matched recovery difference, role-structured minus role-free with feedback, pooled over the five cells (filled) and without Llama-3.3-70B-Instruct (hollow), under the six definitions of Table S5 , with 95% intervals; the band marks the ±0.05 equivalence margin. (b) The pre-registered interaction P4 (Section F ): each difference between two actors in enforcement’s effect on final accuracy, with its 95% interval; the shaded bar behind it spans the smallest interaction detectable at 80% power. Where that bar is wider than any plausible effect, the null is uninformative. Two bar charts with error bars. Left: six pairs of bars, one pair per definition of the verdict-matched contrast, all small and with intervals that cross or nearly cross zero, inside a shaded band from minus 0.05 to plus 0.05. Right: three bars for the pre-registered interaction, each with an interval, drawn over a wider shaded bar showing the smallest detectable effect; for two of the three the shaded bar is wider than the band.
condition
first pass
recovery
final accuracy
verdicts
Reviewer approvals
role-structured, every cycle submitted
0.768
57/93=0.613
0.910
4.20
0.98
role-structured, re-run after the freeze
0.773
18/91=0.198
0.818
1.44
1.13
role-structured, frozen snapshot
0.795
12/82=0.146
0.825
1.41
1.15
role-free with feedback
0.795
45/82=0.549
0.908
3.34
3.34
Appendix
Table S6. Removing the submission gate: gemma-4-31B-it on the same 400 Omni-MATH problems in every row.
first pass (paired, all problems)
recovery (paired on first-pass wrong under both)
actor
n
RS
RF -
diff.
exact p
nboth
RS
RF -
diff.
RS only
RF - only
exact p
gemma-4-31B-it
574
0.794
0.794
+0.0000
1
99
0.212
0.020
−0.1919
21
2
6.60×10−5
gpt-oss-120b
574
0.784
0.789
+0.0052
0.784
96
0.219
0.135
−0.0833
17
9
0.169
Appendix
Table S7. The SciBench benchmark: first pass and recovery with roles and without them (feedback withheld), two actors, frozen separately.
Planner authors
locked answer
recovery
final accuracy, SE − RS
actor
answer (RF - )
(RF - / RS)
(RS)
attrition
pre-registered
as graded
gemma-4-31B-it
0.000 (0.986)
0.119 (0.845/0.288)
0.314 (0.222)
8.0%
−0.0225 [-0.0387,-0.0063]
+0.0190 [+0.0054,+0.0333]
gpt-oss-120b
0.000 (0.916)
0.092 (0.208/0.052)
0.395 (0.324)
9.9%
−0.0050 [-0.0219,+0.0112]
+0.0361 [+0.0215,+0.0507]
Llama-3.3-70B-Instruct
0.000 (0.958)
0.072 (0.806/0.750)
0.139 (0.014)
17.0%
+0.0175 [-0.0200,+0.0550]
+0.0453 [+0.0030,+0.0876]
Appendix
Table S8. Structural enforcement against the role-structured baseline on the Omni-MATH benchmark: the manipulation check, attrition, and final accuracy under both scorings.
Figure S4. Enforcement’s effect on final accuracy under the two scorings. Enforced minus role-structured final accuracy per actor with 95% intervals, under the pre-registered scoring (unfinished trajectories count as incorrect; filled) and as graded (they are dropped; hollow). The tick labels give the share of enforced trajectories that never finish, which is what separates the two bars. The sign flips between the scorings for two of the three actors, and the as-graded gains are ordered like the attrition. A bar chart with error bars. For each of three actors, two bars of the accuracy difference between the enforced and the role-structured condition, one under the pre-registered scoring and one as graded; the as-graded bar is higher than the pre-registered one in every group. The tick labels give each actor's share of enforced trajectories that never finish, from 8 percent to 17 percent.
endpoint
disposition
P1 intervention → fidelity
failed as written; re-designated. The interface forbids an answer field, not
(also the manipulation check)
computation. Re-scoped to answer authorship, which it drives to 0.000 in all three
actors (Table S8 ). Reported as a second dissociation, not as a pass.
P2 actor × interv. on fidelity
not estimable across the panel: the fidelity clauses are condition-specific, so no
common scale exists. Reported as unresolved rather than substituted.
P3 intervention → accuracy (enforced − role-structured), Holm across three:
Appendix
Table S9. The five pre-registered endpoints and what each returned. P4, the primary, is null before and after Holm correction.
RS
RF -
context quartile
recovery
n
recovery
n
difference
gemma-4-31B-it
Q1 (least context)
0.662
65
0.151
93
+0.511
Q2
0.280
50
0.009
108
+0.271
Q3
0.047
85
0.014
73
+0.033
Q4 (most)
0.078
116
0.000
42
+0.078
Appendix
Table S10. Recovery by accumulated-context quartile, with roles and without them (feedback withheld), on the Omni-MATH benchmark; directional only.
organization
team
single agent
difference
gemma-4-31B-it
role-structured
0.8462
0.8688
−0.0225
[−0.0400,−0.0056] , p=0.0122 , n=1600
role-free
0.8125
0.8688
−0.0563
[−0.0737,−0.0394] , p=1.22×10−10 , n=1600
gpt-oss-120b
Appendix
Table S11. Each three-agent team against a single agent of the same actor under the same call cap, on the Omni-MATH benchmark.