cs.CLNov 13, 2026

Folding Language Models into Themselves

Authors: Ziyang Zhang, Yubin Jing, Yuyao Li, Haofan Wang, Yichen Gong

Organizations: Peking University · Georgia Institute of Technology · Tsinghua University · Lovart AI

Abstract

Knowledge distillation usually trains a compact student from scratch, which needs a large amount of data even though the teacher's weights already encode most of what the student has to learn. In this work, we explore building the student directly from the teacher's weights. We propose Concertina, which folds every group of N adjacent teacher layers into one shared block that is executed N times. Each loop step carries a low-rank correction initialized from the layer it replaces, and a per-token gate mixes these corrections, to let every token recover the behavior of the original layers from a single shared block. We further design a distillation data recipe that balances general, mathematical, academic and targeted multiple-choice data, to recover all capabilities of the teacher together rather than one at the expense of another. On Qwen3-4B, a 2.27B Concertina model distilled on 22B tokens retains about 90% of the teacher on commonsense and mathematical reasoning and 84% on average across eight benchmarks. It outperforms same-size dense and layer-sharing baselines, as well as Qwen3-1.7B, which is trained on over a thousand times more data. These results suggest that a compact model need not be retrained at scale, since folding the teacher into itself and distilling it on the right data recovers most of its capabilities. Code is available at https://github.com/myboker/concertina.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Knowledge Distillation for Large Language Models

    Mar 14, 2026Alejandro Paredes La Torre, Barbara Flores, Diego RodriguezLanguage Model DistillationLLM Compression

  2. Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

    Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT DistillationSelective Knowledge Distillation