How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data

Zixian Huang; Kaichen Yang; Xu Huang; Feiyang Hao; Qiming Ge; Bowen Li; He Du; Kai Chen

cs.CLMar 23, 2026

How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data

Authors: Zixian Huang, Kaichen Yang, Xu Huang, Feiyang Hao, Qiming Ge, Bowen Li, He Du, Kai Chen, +1 more

Organizations: Shanghai AI Laboratory · Dalian University of Technology · Nanjing University

Abstract

A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.

Figures & tables

Figure 1: Illustration of the Teacher–Student Cooperation Data Synthesis Framework. Blue and Pink denote the capability-related text spans generated in the responses of GPT-OSS-120B and Qwen3-8B, respectively. Green and Purple indicate the stylistic text segments produced by GPT-OSS-120B and Qwen3-8B. The objective of our synthesis framework is to delegate the generation of capability-related text to GPT-OSS-120B, while assigning the generation of stylistic text to the student model Qwen3-8B.

Figure 2: Overview of the teacher–student cooperation data synthesis framework. The teacher and student models alternately generate capability and style tokens, while the boundary predictors enforce truncation to the exact capability or style spans.

	In-domain Test Sets				Out-of-domain Test Sets
	LCB-V5	LCB-V6	LCB-Pro	OJBench	GPQA-Diamond	AIME2024	AIME2025	OlympiadBench
Qwen3-8B	55.09	49.58	25.35	18.75	60.16	76.67	69.17	69.81
+ Reject-Sampling	54.55	48.00	26.06	18.32	60.61	75.42	66.77	69.53
+ Self-Distillation	51.50	43.43	19.26	14.22	60.61	71.98	62.29	67.80
+ Teacher-Answer	35.33	33.14	21.39	14.01	60.42	74.79	64.48	69.40
+ Teacher-Think	47.90	43.43	24.36	12.50	54.29	77.81	69.06	69.59

Table 1: Performance comparison between TESSY and SFT baselines on in-domain code generation and out-of-domain test sets, where LiveCodeBench is abbreviated as LCB for brevity.

Panel of Figure 7 — Figure 7: Comparison of TESSY and Teacher-Only across Base and Thinking versions of models.

Panel of Figure 8 — Figure 8: Visualization of the data distribution changes caused by TESSY across three teacher models.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	LCB-V5	LCB-V6	LCB-Pro	OJBench
DeepSeek-R1-Distill-Llama-8B	34.73	33.71	17.56	5.82
+ Self-Distillation	19.76	24.00	8.07	5.82
+ Teacher-Only	38.32	36.57	24.22	11.42
+ TESSY	52.10	41.71	26.20	12.50

Appendix

Table 2: Performance comparison using the DeepSeek-R1-Distill-Llama-8B as the student model.

	LCB-V5	LCB-V6	LCB-Pro	OJBench
Qwen3-8B	55.09	49.58	25.35	18.75
+ TESSY (Predictors trained on 500 samples)	59.58	54.00	35.13	23.98
+ TESSY (Predictors trained on 100K samples)	60.68	55.15	35.84	26.51

Appendix

Table 3: Comparison of training performance using TESSY synthetic data with boundary predictors trained on different dataset sizes. Different from Table 1 (80K samples), we used 40K synthesized samples here to reduce computational cost for comparison.

Panel of Figure 17 — Figure 17: Comparison of the average output similarity between Qwen3-8B and three teacher models.

Panel of Figure 18 — Figure 18: Comparison of training loss curves on Base and Reasoning models using SFT data synthesized by TESSY and the Teacher-Only.

Figure 19: Comparison of frequently used words across models. Darker colors indicate higher word frequencies in the outputs of the corresponding models.

Figure 20: Average ratio of teacher-generated and student-generated tokens under the TESSY framework.

Prompt

You are a text analysis expert. Task: Extract all spans of text that are transitional, filler, or tone-setting phrases. What to extract: • Include phrases or sentences that: – Express hesitation, tone, or attitude (e.g., “well”, “okay”, “so”, “let’s see”, “I think”) – Indicate transition or setup (e.g., “to begin with”, “in this case”, “for example”, “but if”) – Serve as narration or connection, not analysis • Do not include: – Actual reasoning, deduction, or explanation – Code or formula descriptions – Problem-solving steps Output format (STRICT JSON): • Return a JSON array of strings, e.g.: ["<span 1>", "<span 2>", ...] • Rules: 1. Each span must be copied verbatim from the original text. 2. Preserve order of appearance. 3. If there are none, return an empty list: [] 4. Output only the JSON array — no explanation or extra text. <input_text> {think_text} </input_text>

Appendix

Table 4: Prompt used to annotate the training data of boundary predictor.

Problem Description
Given a transcript of a Fizz Buzz game (where numbers divisible by $a$ are replaced with “Fizz”, by $b$ with “Buzz”, and by both with “FizzBuzz”), find any valid values of $a$ and $b$ that could generate the transcript. Input: Two integers $c$ and $d$ ( $1\leq c\leq d\leq 10^{5}$ ) indicating the range, followed by $d-c+1$ tokens (integers or “Fizz”/“Buzz”/“FizzBuzz”). Output: Two positive integers $a$ and $b$ ( $1\leq a,b\leq 10^{6}$ ) consistent with the transcript. If multiple solutions exist, output any one.
Response of Qwen3-8B-Base
To solve this problem, we need to determine the values of $a$ and $b$ that could have generated the given transcript. The key insight is that the transcript consists of numbers and strings (“Fizz”, “Buzz”, “FizzBuzz”) that correspond to multiples of $a$ and $b$ … To find $a$ and $b$ , we can: … • Identify the positions of “Fizz” and “Buzz” in the transcript. • … Approach • Identify the positions of “Fizz” and “Buzz”: … Solution Code #include <iostream> ... Explanation • Input Reading: – The program reads the range $[c,d]$ and the transcript. • Position Identification: – The program identifies the positions of “Fizz” and “Buzz” in the transcript. • Validation Loop: – The program iterates over possible values of $a$ and $b$ and checks if they satisfy the conditions for the transcript. • Output: – The first valid pair $(a,b)$ is output, and the program terminates. This approach ensures that we find a valid pair $(a,b)$ that could have generated the given transcript.

Appendix

Table 5: Example output of Qwen3-8B-Base

Problem Description
Given a transcript of a Fizz Buzz game (where numbers divisible by $a$ are replaced with “Fizz”, by $b$ with “Buzz”, and by both with “FizzBuzz”), find any valid values of $a$ and $b$ that could generate the transcript. Input: Two integers $c$ and $d$ ( $1\leq c\leq d\leq 10^{5}$ ) indicating the range, followed by $d-c+1$ tokens (integers or “Fizz”/“Buzz”/“FizzBuzz”). Output: Two positive integers $a$ and $b$ ( $1\leq a,b\leq 10^{6}$ ) consistent with the transcript. If multiple solutions exist, output any one.
Thinking of Student Model (Qwen3-8B)
<think> Okay, let’s see. I need to solve this Fizz Buzz reverse problem … First, I need to understand what the transcript implies about a and b … Let me think about how to model this … But wait, the transcript could have numbers that are not divisible by a or b. So for each entry in the transcript, we can derive constraints on a and b. The challenge is to find a and b that satisfy all these constraints. Let me think about how to approach this … But how do I find a and b that satisfy all these constraints? … Thus, the code is implemented as described. </think>
Thinking of Teacher Model (GPT-OSS-120B)
<think> We need to solve the reverse FizzBuzz problem: given a transcript from c to d inclusive, each entry is either a number (as integer string) or one of the words “Fizz”, “Buzz”, “FizzBuzz”. The transcript is guaranteed to be consistent with some a,b (1 $\leq$ a,b $\leq$ 1e6). Need to output any such a,b. Constraints: c,d up to 1e5 length (max length of transcript is up to 1e5). a,b up to 1e6. Goal: find any pair (a,b) that matches. We need to parse transcript: for each integer i from c to d, we have output string s_i. If s_i is a number, it must be exactly the integer i (as decimal). If s_i is “Fizz”, then i is multiple of a but not b. If “Buzz”: multiple of b but not a. If “FizzBuzz”: multiple of both a and b (i.e., multiple of lcm(a,b)) … We need to find any a,b. This is like solving constraints. Observation: The numbers in transcript that are numeric give us direct constraints: … … Let’s write code. </think>
Thinking of TESSY (Teacher: GPT-OSS-120B, Student: Qwen3-8B)

Appendix

Table 6: Comparison of thinking content generated by the teacher, student, and TESSY for the same programming problem, with TESSY portions generated by the student highlighted in purple .

Problem Description
Given a transcript of a Fizz Buzz game (where numbers divisible by $a$ are replaced with “Fizz”, by $b$ with “Buzz”, and by both with “FizzBuzz”), find any valid values of $a$ and $b$ that could generate the transcript. Input: Two integers $c$ and $d$ ( $1\leq c\leq d\leq 10^{5}$ ) indicating the range, followed by $d-c+1$ tokens (integers or “Fizz”/“Buzz”/“FizzBuzz”). Output: Two positive integers $a$ and $b$ ( $1\leq a,b\leq 10^{6}$ ) consistent with the transcript. If multiple solutions exist, output any one.
Final Answer of Student Model (Qwen3-8B)
To solve the Fizz Buzz reverse problem , we need to determine possible values of integers $a$ and $b$ that could generate a given transcript of the game. The transcript consists of numbers and the words “Fizz”, “Buzz”, and “FizzBuzz” corresponding to multiples of $a$ , $b$ , and both $a$ and $b$ respectively. Approach • Parse Input: Read the range of the transcript and the transcript itself. • Classify Entries: Identify which entries are “Fizz”, “Buzz”, “FizzBuzz”, or numbers. This helps in determining constraints on $a$ and $b$ . • Determine Constraints: For entries that are “Fizz” or “FizzBuzz”, $a$ must divide the corresponding number… • Generate Candidates: For “Fizz” and “FizzBuzz” entries, compute the GCD of all such numbers and generate all its divisors as potential values for $a$ … • Validate Candidates: Check all combinations of $a$ and $b$ to find a valid pair that satisfies all constraints from the transcript. Solution Code #include <bits/stdc++.h> using namespace std; ... Explanation • Input Parsing: The input is read and stored in a vector of strings representing the transcript. • Classifying Entries: We classify each entry to determine which numbers are divisible by $a$ , $b$ , or both. • Generating Candidates: Using the GCD of relevant numbers, we generate potential values for $a$ and $b$ . If no such entries exist, we find the smallest number that does not divide any number in the transcript. • Validation: For each candidate pair $(a,b)$ , we check if it satisfies all constraints from the transcript … This approach ensures that we efficiently narrow down the possible values of $a$ and $b$ and validate them against the transcript constraints.
Final Answer of Teacher Model (GPT-OSS-120B)
Solution Explanation For every integer i , … 1. Reformulating the conditions S_Fizz = { i \| type(i) = Fizz } … 2. Determining a If A is empty, a never has to divide any number, we may choose any value that … 3. Algorithm read c , d … 4. Correctness Proof • Lemma 1: If A is non - empty, any integer x that divides every element of A must divide G = gcd(A) ; conversely every divisor of G divides every element of A . Proof: … • … 5. Complexity Analysis $N=d\textendash c+1(\leq 10^{5})$ … Both limits are easily satisfied. 6. Reference Implementation #include <bits/stdc++.h> using namespace std; ... The program follows exactly the algorithm proven correct above and conforms to the GNU C++17 compiler.

Appendix

Table 7: Comparison of final answer content generated by the teacher and student models for the same programming problem

Explore similar work

Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation
Oct 13, 2025Hengyuan Zhang, Shiping Yang, Xiao Liang +8Synthetic Data Generation Language Model Distillation
The University of Hong Kong · Simon Fraser University · University of California, Los Angeles +3
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding
May 4, 2026Taewon Yun, Jisu Shin, Jeonghwan Choi +2CoT Distillation Efficient Language Model Reasoning
KAIST · UNIST
Learning from Synthetic Data Improves Multi-hop Reasoning
Mar 2, 2026Anmol Kabra, Yilun Yin, Albert Gong +6RL for Language Model Reasoning Multi-Hop Reasoning
Cornell University · University of Cambridge · Stanford University

Oct 13, 2025J/K move · Enter open · S save

Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation

Hengyuan Zhang, Shiping Yang, Xiao Liang, Chenming Shang+7

The University of Hong Kong · Simon Fraser University · University of California, Los Angeles+3

Synthetic Data Generation Language Model Distillation Multi-Teacher Knowledge Distillation Synthetic Data Generation for Language Models Teacher-Student Learning

Open paper AskPDF arXiv

Training student models on synthetic data generated by strong teacher models is a promising way to distilling the capabilities of teachers. However, recent studies show that stronger models are not always optimal teachers, revealing a mismatch between teacher outputs and student learnability. To address this issue, we propose PerSyn (Personalized data Synthesis), a novel synthesis strategy that operates under a new

Route then Generate'' paradigm to create data tailored to each student model, enabling it to learn more effectively. Specifically, PerSyn first assigns each prompt to its optimal teacher via a query-level router that jointly considers student learnability and teacher response quality. Each teacher then synthesizes data only for its assigned prompts, making the process more efficient than the conventional

Generate then Select'' paradigm, where all teachers must generate parallel responses for the entire prompt set before constructing the final dataset. Extensive experiments across different model families and scales demonstrate that PerSyn consistently achieves superior or comparable performance to all baselines in instruct tuning and math reasoning settings. Further analysis verifies the effectiveness of PerSyn and offers extra insights to propel future research.

Abstract

Explore similar work

Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation

Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding

Learning from Synthetic Data Improves Multi-hop Reasoning