Allspark: Weak to Strong Transfer via Alternating Chain of Thought

Kaizhao Liang; Junxiong Wang; Chen Liang; Zhendong Wang; Qiang Liu

cs.LGSep 26, 2026

Allspark: Weak to Strong Transfer via Alternating Chain of Thought

Authors: Kaizhao Liang, Junxiong Wang, Chen Liang, Zhendong Wang, Qiang Liu

Organizations: UT Austin · Together AI · Microsoft

Abstract

Recent progress in frontier models has renewed interest in large-scale reinforcement learning (RL), but the cost of generating large-model rollouts makes even testing RL recipes expensive. We ask whether reasoning improvements learned by a small, weak model can benefit a larger, stronger model without using the strong model's rollouts during training. We introduce Allspark, a training and inference framework for weak-to-strong transfer through alternating chains of thought. A weak teacher is trained alongside a frozen copy of the same model; the two alternate reasoning segments, and the frozen model produces the final answer. At inference time, a stronger student replaces the frozen training partner, while both models remain fixed. Because they communicate through text, the teacher can steer students from different model families and with different tokenizers. We study Allspark at two scales: controlled Qwen experiments across math and reasoning, and larger-scale Inkling experiments on ARC-AGI-2. The Inkling experiments show accuracy gains in within-family and cross-family settings, including transfer to Kimi and Nemotron, with benefits that vary across inference settings. These findings motivate reusing a trained weak teacher across strong students and examining the resulting accuracy--token tradeoff.

Figures & tables

Figure 1: Vanilla RL updates the full reasoning and answer. Allspark updates only the weak teacher’s reasoning beside a frozen small student. Transfer freezes the teacher and replaces the student with a stronger model. Blocks show authorship; underlines mark direct token loss.

Domain	Dataset	RL pool	Development
Math	MATH ( Hendrycks et al., 2021 )	142	128
Reasoning	LogiQA 2.0 ( Liu et al., 2023 )	162	128
Total		304	256

Table 1: Qwen training and development data for the two reported domains. Both methods share the screened RL pool. Development questions are disjoint from this pool but were used for checkpoint selection and subsequent evaluation. All samples come from the datasets’ original training splits.

Student	Teacher	Math	Reasoning
Qwen3-1.7B	None	81.3	73.4
	Ordinary RL	88.2	71.8
	Allspark	89.8	75.0
Qwen3-1.7B (RL-tuned)	None	89.8	70.3
Qwen3-4B	None	96.9	81.2
	Untrained	93.0	79.7

Table 2: Qwen development accuracy (%) on math and reasoning, with 128 questions per domain. The Allspark teacher is checkpoint 94; Ordinary RL is the separately trained Qwen3-1.7B policy. Dashes mark ongoing evaluations.

Figure 2: Selected paired math and reasoning rollouts with a Qwen3-4B student and a Qwen3-1.7B weak teacher. Blue bubbles are teacher-authored; white bubbles are student-authored. M3, R1, and R2 show teacher and next-student excerpts; P1 compares final-answer excerpts. Both P1 finals hit the answer-token limit. Quotes are verbatim; task summaries are abbreviated.

Figure 3: Two further recorded handoffs with Qwen3-4B students and the same Qwen3-1.7B weak teacher. Blue bubbles are teacher-authored; other excerpts are student-authored. Both conditions solve M2; R5 changes from B (incorrect) to C (correct). Quotes are verbatim and task summaries are abbreviated.

Figure 4: Accuracy–token tradeoff on 96 ARC development tasks, with three attempts per task. Paired runs reuse the rl’ed teacher with an Inkling-Small or Inkling student. Points from left to right use thinking efforts 0.40, 0.55, 0.70, and 0.85. Tokens include retained output from both writers and the final answer.

Figure 5: Changes versus each student alone: Nemotron-3-Ultra (medium/full) and Kimi-K2.6 (128K). Same weak teacher; 96 ARC development tasks, three attempts each. Bars: archived 95% task-bootstrap intervals. Tokens: retained output from both models, in native tokenizer units.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Optimizer	AdamW; constant learning rate $10^{-5}$
Adam moments / epsilon	$(0.9,0.999)$ / $10^{-8}$
Weight decay / gradient-norm ceiling	$0$ / $1$
LoRA rank $r$ / $\alpha$ / dropout	$16$ / $32$ / $0$
LoRA projections	$q,k,v,o$ , gate, up, down; no biases
Training context ceiling	8,192 tokens

Appendix

Table 3: Shared settings for the local Qwen training runs.

Checkpoint	Math	Reasoning
55	116	93
91	114	96
94	119	96

Appendix

Table 4: Finalized Allspark checkpoints under the revised evaluation protocol. Entries are correct answers on the two reported domains; each has 128 development questions. Checkpoint 94 is the fixed selection from the original three-domain protocol. Bold marks the selected checkpoint. These are development results, not public-test scores.

Figure 6: Development accuracy of the ordinary-RL-trained Qwen3-1.7B teacher alternating with the frozen student, under the historical 8K protocol (4,096 reasoning and 1,536 answer tokens). Each domain has 128 fixed questions; the figure reports math and reasoning only. Filled circles connect the initial checkpoint (step 0) to checkpoints 10, 20, 50, and 100 without smoothing.

Figure 7: Qwen3-1.7B training diagnostics over the first 100 committed updates of each method on a logarithmic step axis (one seed per method). Lines show full trailing-five-update means, starting at update 5. Pass rates are measured on accepted, filtered training rollouts; overall is the equal-domain mean. Learner entropy is measured on the respective trainable-token contexts. The accepted prompts and rollout policies differ between methods, so these curves are descriptive training diagnostics, not held-out accuracy or paired treatment effects.

Student setting	Teacher effort	Student context	Reasoning limit
Nemotron-3-Ultra, medium	0.40	65,536	24,576
Nemotron-3-Ultra, full	0.85	65,536	49,152
Kimi-K2.6, 128K	0.85	131,072	49,152

Appendix

Table 5: Cross-family inference settings. Context and reasoning limits are in tokens. The context limit applies to the student; the reasoning guard is checked in the receiving model’s tokenizer at each handoff.

Student	Teacher	Accuracy (%)
Inkling-Small	None	60.4
Inkling-Small	Untrained	60.4
Inkling-Small	Allspark-trained	70.8
Inkling	None	75.0
Inkling	Untrained	66.7
Inkling	Allspark-trained	76.0

Appendix

Table 6: Step 35, effort 0.7, one attempt on each of 96 ARC development tasks.

Model / serving tier	Fresh prefill	Cached read	Sampling
Inkling-Small	0.580	0.058	1.440
Inkling	1.870	0.187	4.680
Nemotron-3-Ultra, 64K	2.490	0.249	6.225
Kimi-K2.6, 128K	2.575	0.515	6.405

Appendix

Table 7: Historical rate assumptions, in USD per million tokens.

Student setting	Effort	Student alone	Weak + teacher	Allspark
Inkling	0.40	0.0266	0.0198	0.0483
	0.55	0.0311	0.0199	0.0546
	0.70	0.0426	0.0232	0.0606
	0.85	0.0623	0.0354	0.0895
Nemotron, medium	0.40	0.0459	0.0196	0.0757
Nemotron, full	0.85	0.1230	0.0352	0.1407

Appendix

Table 8: Archived modeled inference cost, in USD per attempt, over 96 development tasks with three attempts each. Weak + teacher always uses an Inkling-Small student and the trained teacher; the cross-family rows reuse that reference. For the Inkling rows, effort applies to both writers; for cross-family rows it denotes the teacher’s effort, with the target student using its native thinking setting.

Weak-to-Strong Generalization

Weak-to-strong generalization studies how a capable learner can improve from supervision produced by a less capable teacher.

See all 22 papers

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
Apr 14, 2026Linhao Yu, Tianmeng Yang, Siyu Ding +8RL for Language Model Reasoning LLM Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Baidu Inc. · Institute of Information Engineering, Chinese Academy of Sciences.
Weak-to-Strong Generalization via Direct On-Policy Distillation
Jul 6, 2026Shiyuan Feng, Huan-ang Gao, Haohan Chi +7RL for Language Model Reasoning Weak-to-Strong Generalization
1SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · 4Peking University +1
Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimRL for Language Model Reasoning Efficient Language Model Reasoning
1Princeton University · University of California, Berkeley.

Apr 14, 2026J/K move · Enter open · S save

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Linhao Yu, Tianmeng Yang, Siyu Ding, Renren Jin+7

TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · Baidu Inc. · Institute of Information Engineering, Chinese Academy of Sciences.

RL for Language Model Reasoning LLM Reasoning RL for Language Models LLM-Guided RL

Open paper AskPDF arXiv

RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose KnowRL (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.

Abstract

Weak-to-Strong Generalization

Explore similar work

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Weak-to-Strong Generalization via Direct On-Policy Distillation

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently