Knowledge distillation usually trains a compact student from scratch, which needs a large amount of data even though the teacher's weights already encode most of what the student has to learn. In this work, we explore building the student directly from the teacher's weights. We propose Concertina, which folds every group of N adjacent teacher layers into one shared block that is executed N times. Each loop step carries a low-rank correction initialized from the layer it replaces, and a per-token gate mixes these corrections, to let every token recover the behavior of the original layers from a single shared block. We further design a distillation data recipe that balances general, mathematical, academic and targeted multiple-choice data, to recover all capabilities of the teacher together rather than one at the expense of another. On Qwen3-4B, a 2.27B Concertina model distilled on 22B tokens retains about 90% of the teacher on commonsense and mathematical reasoning and 84% on average across eight benchmarks. It outperforms same-size dense and layer-sharing baselines, as well as Qwen3-1.7B, which is trained on over a thousand times more data. These results suggest that a compact model need not be retrained at scale, since folding the teacher into itself and distilling it on the right data recovers most of its capabilities. Code is available at https://github.com/myboker/concertina.
Figures & tables
Figure 1: Overview of Concertina. Left: the layers of the pretrained decoder are split into groups G1,…,GL/N of N adjacent layers. Middle: each group is replaced by one shared block that is executed N times, and red dots mark the folded linear maps. Right: each folded linear map adds low-rank corrections BkAk to the shared weight W0 , mixed by gate weights gk that depend on the token x and the loop step t . Residual connections are omitted for clarity.
Model
Params
Tokens
base-7
MMLU
Redux
Pro
GPQA
BBH
GSM8K
MATH
Qwen3-4B (teacher)
4.02B
36T
0.630
0.671
0.720
0.404
0.379
0.505
0.879
0.566
Qwen3-1.7B
1.72B
36T
0.562
0.557
0.591
0.297
0.293
0.363
0.710
0.460
Concertina
2.27B
22B
0.580
0.556
0.587
0.301
0.308
0.400
0.788
0.498
% of teacher
57%
92%
83%
82%
75%
81%
79%
90%
88%
Table 1: Main results on eight benchmarks. The last row gives the score of Concertina as a percentage of the Qwen3-4B teacher. Tokens are pretraining tokens for the Qwen3 models and distillation tokens for Concertina, and bold marks the better of the two compact models.
Figure 2: Scores of the main model on the eight benchmarks during training. Each line follows one benchmark at every 2.2B distillation tokens. Dashed lines mark the scores of the Qwen3-4B teacher in the color of each benchmark.
Model
Params
base-7
MMLU
Redux
Pro
GPQA
BBH
GSM8K
MATH
Qwen3-4B (teacher)
4.02B
0.630
0.671
0.720
0.404
0.379
0.505
0.879
0.566
Truncated dense
2.31B
0.570
0.434
0.453
0.157
0.298
0.289
0.017
0.038
Concertina
2.27B
0.590
0.533
0.566
0.238
0.283
0.332
0.718
0.408
Table 2: Folding versus a truncated dense model that keeps 19 of the 36 layers of Qwen3-4B. Under the same 20B-token distillation, the truncated model almost fails on GSM8K and MATH, while Concertina reaches 82% and 72% of the teacher.
Figure 3: Accuracy of Concertina, RRT and FoldGPT against the number of training tokens. (a) Base-7 average. (b) Average over eleven tasks. Both baselines are trained with their published recipes on the same data, and the token axis is logarithmic. The middle point of Concertina is at 786M tokens, and the dashed line in (a) marks the Qwen3-4B teacher.
Figure 4: Controlled comparison of the RRT design choices with ours on Qwen3-4B, all trained with the same distillation recipe. (a) Base-7 accuracy of cycle tying as used by RRT and of our adjacent tying at equal size (2.27B) and equal token budget. (b) Base-7 accuracy of the static depth-wise LoRA used by RRT and of our input-conditioned gate on the main configuration. (c) Mean cosine similarity between the residual updates of the two layers that each topology ties together, measured on the teacher.
Mixture
Tokens
base-7
MMLU
Redux
Pro
GPQA
BBH
GSM8K
MATH
Avg.
General, math, academic
20B
0.590
0.533
0.566
0.238
0.283
0.332
0.718
0.408
75.6%
Generic, with diverse text
22B
0.592
0.549
0.584
0.252
0.288
0.364
0.740
0.438
78.6%
Targeted (ours)
22B
0.580
0.556
0.587
0.301
0.308
0.400
0.788
0.498
83.6%
Table 3: Data mixture ablation for the 2.27B fold of Qwen3-4B. Every row is a separate run from the teacher weights with the same method and objective, and only the training data differs. The last column is the mean percentage of the teacher over the eight benchmarks.
Model
Params
base-7
MMLU
Redux
Pro
GPQA
BBH
GSM8K
MATH
Qwen3-8B (teacher)
8.19B
0.660
0.726
0.768
0.462
0.369
0.553
0.895
0.587
Concertina ( N=3 )
3.68B
0.601
0.539
0.571
0.269
0.268
0.353
0.745
0.442
% of teacher
45%
91%
74%
74%
58%
73%
64%
83%
75%
Table 4: Folding Qwen3-8B with groups of three layers. The last row gives the score of Concertina as a percentage of the teacher.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Tokens
Base-7
MMLU
Redux
Pro
SuperGPQA
11-task
PPL
FoldGPT
197M
0.410
0.247
0.258
0.133
0.093
0.327
23.1
FoldGPT
0.49B
0.419
0.234
0.234
0.110
0.080
0.327
21.2
FoldGPT
1B
0.426
0.224
0.228
0.111
0.085
0.330
20.3
FoldGPT
2B
0.430
0.238
0.242
0.113
0.083
0.335
19.8
RRT
197M
0.415
0.230
0.229
0.122
0.103
0.326
17.5
RRT
0.49B
0.457
0.256
0.269
0.106
0.111
0.358
15.6
Appendix
Table 5: Per-benchmark scores of the prior folding methods and Concertina on Qwen3-4B. Every model has 18 unique blocks, which gives 2.21B parameters for FoldGPT, 2.34B for RRT and 2.27B for Concertina. Each task is evaluated on at most 1000 examples and OpenBookQA on 500.
Variant
Params
Tokens
Base-7
% teacher
Δ
Cycle tying, RRT
2.27B
197M
0.468
74.2%
Adjacent tying, ours
2.27B
197M
\best 0.498
\best 78.9%
+0.030
Cycle tying, RRT
2.27B
2B
0.521
82.6%
Adjacent tying, ours
2.27B
2B
\best 0.535
\best 84.8%
+0.014
Appendix
Table 6: Controlled comparison of cycle tying as used by RRT with adjacent tying on Qwen3-4B. Both variants use static step-LoRA and keep all other settings fixed. Scores are base-7 averages on at most 1000 examples per task, and the percentage is relative to the Qwen3-4B teacher at 0.631.
Variant
Tokens
base-7
MMLU
Redux
Pro
GPQA
BBH
GSM8K
MATH
Static depth-wise LoRA
197M
0.455
0.261
0.260
0.113
0.242
0.234
0.106
0.077
Input-conditioned gate
197M
0.462
0.238
0.234
0.115
0.247
0.245
0.132
0.074
Gate, trainable first step
197M
0.462
0.259
0.260
0.118
0.237
0.246
0.140
0.085
Static depth-wise LoRA
490M
0.471
0.254
0.254
0.116
0.258
0.236
0.189
0.124
Input-conditioned gate
490M
0.477
0.262
0.267
0.116
0.278
0.235
0.235
0.142
Appendix
Table 7: Full evaluation of the static depth-wise LoRA and the input-conditioned gate on the main configuration of Qwen3-4B with 2.27B parameters.
Model
ARC-C
ARC-E
HellaSwag
PIQA
OBQA
WinoGrande
LAMBADA
base-7
norm
norm
norm
acc
norm
acc
acc
mean
Qwen3-4B (teacher)
0.543
0.785
0.676
0.750
0.402
0.659
0.594
0.630
Qwen3-1.7B
0.431
0.709
0.590
0.726
0.362
0.611
0.506
0.562
Truncated dense (20B)
0.445
0.729
0.577
0.734
0.404
0.620
0.482
0.570
Concertina (20B)
0.463
0.755
0.614
0.745
0.396
0.645
0.515
0.590
Concertina (22B, main)
0.458
0.746
0.585
0.731
0.382
0.646
0.511
0.580
Appendix
Table 8: Per-task zero-shot scores behind the base-7 average. The 4B rows correspond to Tables 1 and 2 , where both 20B models share the recipe of Section 4.4 , and the 8B rows correspond to Table 4 . The second header row gives the metric of each task, where norm denotes length-normalized accuracy.
Layer pair
(0,1)
(2,3)
(4,5)
(6,7)
(8,9)
(10,11)
(12,13)
(14,15)
(16,17)
NMSE
1.963
0.335
0.387
0.0083
0.0127
0.0103
0.0121
0.0177
0.0088
Layer pair
(18,19)
(20,21)
(22,23)
(24,25)
(26,27)
(28,29)
(30,31)
(32,33)
(34,35)
NMSE
0.0092
0.0109
0.0257
0.0328
0.0440
0.0684
0.0930
0.130
0.492
Appendix
Table 9: Normalized reconstruction error of each adjacent layer pair of Qwen3-4B under an untrained fold with rank-32 step corrections. Each column is one pair of layers, indexed from zero.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.