Multi-agent discussion produces reasoning collaboratively, while deployment still relies on a smaller student that must answer alone. Full-graph distillation transfers such discussions by treating each correct solution as a separate training target, but what teachers provide need not match what the student learns. This mismatch appears in two ways: several correct solutions may amount to the same lesson in the student's representation (over-teaching), while some training questions remain missed after learning (under-learning). We quantify both with rates that are high on discussion graphs. To address them, we introduce FAR (Fold--Attempt--Revisit), a learner-centered distillation procedure that organizes supervision around the student's own state. Before training, FAR folds solutions within each discussion state into representative lessons. After training, the student's own attempts identify missed questions, while poorly represented clusters indicate where folding may have lost useful variation; FAR then writes targeted lessons from retained discussion contexts. Across two students and four reasoning tasks, FAR outperforms full-graph distillation with far fewer training targets and lower under-learning. Further analyses examine the behavior of FAR and the sources of its gains. Overall, the results show that organizing discussion supervision around the student can make distillation both more effective and more efficient.
Figures & tables
Figure 1: Learner-centered distillation from multi-agent discussion. Repeated correct solutions can over-teach the student while leaving questions unsolved. The human-learning sequence motivates two learner-centered decisions: which solutions need separate study, and which questions need another lesson after an unaided attempt.
Figure 2: Overview of FAR. Fold selects weighted representatives; Attempt identifies remaining misses; Revisit uses retained sources and the fixed Fold student θ1 to write further lessons. Residual denotes variation in the material not fully covered by its representative solution.
Method
SQA
ECQA
ARC
GSM8K
Avg ↑
ULR ↓
TPC ↓
Mistral-7B-Instruct-v0.2
Base
61.57
55.79
42.83
43.52
50.93
45.80
347.3
SMAGDi ‡
66.81
56.15
60.92
41.32
56.30
39.22
1065.3
SFT
69.00
71.83
77.30
56.56
68.67
14.89
136.8
MAGDi
72.93
71.74
77.47
56.86
69.75
14.40
135.7
Fold
70.31
72.97
77.65
56.71
69.41
14.84
134.7
Table 1: Held-out accuracy and answer efficiency. Accuracy and ULR are percentages; Avg and TPC equally weight their four task values. Fold is FAR’s first stage. Bold marks the best accuracy, ULR, and TPC; ‡ denotes multi-call inference. Task-specific TPC appears in Appendix Table 11 .
Method
PIQA
SVAMP
MedMCQA
Avg ↑
Δ Fold ↑
Train ULR ↓
TPC ↓
Mistral-7B-Instruct-v0.2
Base
72.80
60.90
40.76
58.15
−5.88
39.70
198.0
SFT
77.58
70.70
43.65
63.98
−0.06
15.67
169.6
MAGDi
77.09
69.70
43.80
63.53
−0.51
14.93
163.3
Fold
77.64
70.70
43.77
64.04
0.00
15.91
167.0
FAR
78.00
72.50
44.81
65.10
+1.06
14.60
160.0
Table 2: Joint training transfers beyond the source tasks. Accuracies and train ULR are percentages; Avg equally weights the three transfer tasks. Δ is the gain over Fold. TPC counts generated tokens per correct transfer answer, including tokens from wrong answers. Bold marks the best value within each student.
Method
Further material
Avg ↑
Δ Full ↑
TPC ↓
ULR ↓
FAR (Full)
F and R lessons
74.29
0.00
125.7
14.29
Fold only
No further material
69.41
−4.88
134.7
14.84
F-only
Missed-question lessons
72.53
−1.76
128.3
14.68
R-only
Residual-selected lessons
71.42
−2.87
126.3
14.89
E-replay
Original correct references
69.87
−4.42
150.3
16.26
Table 3: Revisit ablations on Mistral. Avg and ULR are four-task percentages; TPC averages task ratios. Δ is the accuracy difference from complete FAR. F selects missed questions; R selects retained variation on solved questions. Bold marks the best result.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Setting
Fold
Five epochs; base learning rate 10−5 ; mass-scaled AdamW steps
Fold optimizer
AdamW moment coefficients (0.9,0.999) , stabilizer 10−8 , zero decay; linear schedule, 3% warmup; gradient norm cap 1
Revisit
One pass; 5×10−6 ; complete-response token-mean CE, including EOS; equal F/R weights; no mass scaling
Adapters
LoRA ( Hu et al., 2022 ) rank 16, scale 32, dropout 0.05; query and value projections
Training batch
One example; gradient accumulation one
Lesson writer
Fixed θ1 during writing; greedy; batch eight; 512 new tokens
Appendix
Table 4: Fold and Revisit settings. The representative count follows the state spectrum; cluster mass is a count of represented solutions, not a searched parameter. Acceptance uses training gold and mechanical checks, without held-out test answers.
Student
Task
Lessons
Lesson fraction
Response tokens
Added exposure
Δ Acc ↑
Mistral
SQA
166
8.92%
12,983
1.28%
+7.42
Mistral
ECQA
163
6.83%
11,871
1.17%
+3.65
Mistral
ARC
125
5.53%
15,543
1.36%
+4.52
Mistral
GSM8K
309
15.34%
47,828
3.59%
+3.94
Qwen
SQA
215
11.56%
6,768
0.73%
+3.93
Qwen
ECQA
163
7.01%
10,990
1.17%
+3.60
Appendix
Table 5: A single Revisit pass improves held-out accuracy. Counts and response tokens describe the selected one-pass lesson sets. Lesson fraction is relative to Fold’s representatives; added exposure is relative to all five Fold passes. Δ Acc is FAR minus Fold in percentage points.
Student
Task
Original nodes
Representatives
Tokens saved ↑
GPUh saved ↑
Mistral
SQA
3,223
1,861
40.63%
24.79%
Mistral
ECQA
3,678
2,385
32.38%
34.25%
Mistral
ARC
3,738
2,261
38.80%
31.24%
Mistral
GSM8K
4,043
2,014
48.08%
60.35%
Qwen
SQA
3,223
1,860
40.90%
42.86%
Qwen
ECQA
3,678
2,325
33.96%
27.07%
Appendix
Table 6: Fold changes the volume of initial study. Original nodes precede exact deduplication; representatives are the actual Fold training targets. Target-token reductions compare those representatives with the original correct-node pool. GPU-hour reductions use recorded Fold and SFT training spans.
Student
SQA
ECQA
ARC
GSM8K
Mean
Mistral
39.53
31.50
36.92
48.06
39.00
Qwen
39.34
33.08
37.86
50.66
40.24
Appendix
Table 7: Over-teaching by task. OTR describes repeated solutions in the shared correct-solution pool under each initial student’s representation. Values are percentages; Mean equally weights the four tasks.
Student
Task
States
k : 1/2/3
n
Δ
Lower
Upper
dr
Mistral
SQA
1379
897/482/0
195
-0.006
-0.018
0.005
0.055
Mistral
ECQA
1982
1579/403/0
490
-0.007
-0.012
-0.002
0.054
Mistral
ARC
1542
823/719/0
252
-0.012
-0.019
-0.005
0.058
Mistral
GSM8K
1767
1522/243/2
393
-0.014
-0.019
-0.009
0.049
Qwen
SQA
1379
898/481/0
195
-0.007
-0.017
0.004
0.055
Qwen
ECQA
1982
1639/343/0
490
-0.008
-0.012
-0.004
0.055
Appendix
Table 8: State allocation and distances. k lists state counts at rounded effective rank 1/2/3. n counts paired questions; Δ is cross-state minus within-state distance with a 95% interval. dr is mean representative-to-member distance.
Student
Task
Target
C
Q
Z
rs
Lower
Upper
Mistral
SQA
Rep.
1861
931
916
-0.045
-0.093
0.003
Mistral
SQA
Nonrep.
945
839
n.a.
-0.003
-0.069
0.061
Mistral
ECQA
Rep.
2385
885
1401
-0.072
-0.111
-0.032
Mistral
ECQA
Nonrep.
984
727
n.a.
0.087
0.028
0.148
Mistral
ARC
Rep.
2261
989
1109
0.088
0.045
0.130
Mistral
ARC
Nonrep.
1152
967
n.a.
0.048
-0.008
0.104
Appendix
Table 9: Residual–target NLL correlations. C and Q count clusters and questions for each target object. Z counts zero-residual clusters in the full representative pool. rs is Spearman correlation; bounds form a 95% question-bootstrap interval.
Task
Single-round graphs
Correct nodes / graph (single-round)
Correct nodes / state (multi-round)
Share of correct nodes
GSM8K
557
2.98
1.90
41.0%
StrategyQA
719
2.71
1.60
60.5%
ARC
736
2.96
1.89
58.3%
ECQA
306
2.58
1.58
21.5%
Appendix
Table 10: Released training graphs (1,000 per task). Counts use correct nodes before deduplication. Single-round graph means include zero-correct graphs; multi-round state means exclude empty states. The last column gives the single-round share of all correct nodes.
Method
SQA
ECQA
ARC
GSM8K
Mean
Mistral-7B-Instruct-v0.2
Base
175.7
220.9
369.8
622.6
347.3
SMAGDi
868.9
851.5
711.2
1829.8
1065.3
SFT
125.6
107.4
104.9
209.2
136.8
MAGDi
119.0
108.7
103.9
211.0
135.7
Fold
118.1
103.1
104.4
213.4
134.7
Appendix
Table 11: Generated tokens per correct answer by task. Lower is better. TPC includes output tokens from incorrect answers and all calls of multi-call inference. Mean equally weights the four task ratios.
Method
Mistral GPUh
Qwen GPUh
SFT
4.28
4.87
MAGDi
9.77
11.33
Fold
2.55
2.97
SMAGDi
1.59
2.14
Appendix
Table 12: Recorded training subtotals across four tasks. GPU-hours sum the recorded stage spans described above. These are training subtotals, not end-to-end FAR costs.
Student
Method
Optimizer steps
Target tokens
Mistral
SFT
72,245
7,756,040
Mistral
MAGDi
80,535
8,582,935
Mistral
Fold
42,605
4,506,535
Qwen
SFT
72,245
7,277,905
Qwen
MAGDi
80,535
8,059,635
Qwen
Fold
41,560
4,134,770
Appendix
Table 13: Recorded task-training exposure. Counts sum the four task-specific runs over all epochs. Targets are response tokens used in next-token supervision.
Case
Student / task
Gold
Base
Fold
FAR
C01
Mistral / GSM8K
140
40
120
140
C02
Qwen / GSM8K
5
3
10
5
C03
Qwen / SQA
yes
no
no
yes
C04
Mistral / ARC
B
C
D
B
C05
Mistral / ECQA
4
3
3
4
C06
Qwen / ECQA
1
5
2
1
Appendix
Table 14: Answers in the nine test cases. Labels follow the task scorer. C01–C06 are repairs, C07 remains incorrect, and C08–C09 are regressions. In C07, Base reports a loss, while its answer is parsed as 38000.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation aligns teacher feedback with a student's current behavior, yet acquiring useful supervision for multi-turn agents requires identifying both where guidance is needed and whether the student can sustain the interaction afterward. We introduce Frontier-OPD, which addresses this problem through the student's learning frontier: decision situations that remain challenging but admit useful autonomous continuation after brief teacher guidance. To operationalize this frontier, Frontier-OPD organizes on-policy experience into a shared-state trajectory graph, where trajectory-balanced aggregation of teacher-student disagreement identifies distinct intervention candidates while preserving the histories needed for replay. Short teacher bridges are then assessed by comparing agreement over the bridge and the ensuing student continuation with an equally long segment of the original trajectory, providing a behavioral proxy for guidance that the current student can build on. Accepted teacher responses augment ordinary distillation data, and fresh rollouts renew the frontier estimate as the policy changes. Experiments on ALFWorld, WebShop, and ScienceWorld report higher mean autonomous performance than the compared distillation baselines across the evaluated student scales and teacher configurations, with ablations indicating complementary effects of candidate selection and guidance filtering. In one ALFWorld timing setting, the additional collection cost is offset by a shorter time to a target success rate.
On-policy distillation (OPD) trains compact language agents with teacher feedback on student-generated trajectories. In multi-turn tasks, compounding errors can move students beyond the teacher's effective supervision. We introduce Graph-Conditioned On-Policy Agent Distillation (GC-OPD), which enriches an off-the-shelf teacher's scoring context with execution evidence. A graph indexes repeated teacher executions by shared states while preserving complete successful and failed histories. After each student episode, GC-OPD retrieves current-state references or historical alternatives and combines them with student hindsight to score the original thought-action tokens. Using the same original teachers, GC-OPD improves mean success over vanilla OPD from 24.70% to 48.78% on ScienceWorld (4B student), from 53.36% to 85.26% on ALFWorld Unseen, and from 29.10% to 37.65% on WebShop. At matched student sizes, it also achieves higher mean success than every evaluated OPD baseline using GRPO-trained teachers on ScienceWorld and ALFWorld; the strongest such ScienceWorld 4B baseline reaches 46.66%. GC-OPD requires no task-specific teacher optimization.
We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0% to 62.0%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2% to 35.4%.