One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data
Organizations: Landi Lab
Abstract
Byte-level BPE fragments Chinese text (for Qwen3.8 on Chinese Wikipedia, 3.84 bytes per token on average; 64% of tokens are single characters) and hides spelling from the model. We ask whether an already-trained 27B subword model can be converted directly into a pure byte-level model (one byte, one token), without a patching structure or a local encoder, while keeping most of its ability. Starting from Qwen3.8-27B (48 Gated DeltaNet layers + 16 full-attention layers), we shrink the vocabulary from 248,320 to 384 entries (the 256 byte symbols already in the BPE vocabulary plus special ids), leave every other weight untouched, and continue training in a two-stage curriculum (encoding first, then reading) on 4.77 GB of supervised bytes: all dialogue/behavioral supervision is generated on-policy by the original model itself (character dictionary, word dictionary, general dialogue, coding sessions), and knowledge reading uses encyclopedia text (54% of supervised bytes). Training takes 178.6 h on 8xH200. On the same GPU and serving stack, the byte model's gap to the original falls within the paired bootstrap interval on MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K, and it loses 4-6 points on C-Eval, thinking-mode GSM8K and non-thinking IFEval. Without thinking it leads on every character-level task (CUTE 75.2 -> 84.0, EXECUTE-zho 81.6 -> 93.9, our Chinese character suite 54.4 -> 64.0), but lookup knowledge about Chinese characters such as pronunciation and stroke count drops, and thinking does not recover it. The cost is throughput: per-step speed is identical, but each step yields 3-5x less output; Chinese characters per second is about 1/4 of the original, and prefill at 64K bytes is 6.5x slower. We also document four byte-model failure modes (greedy runaway, sampling breaking UTF-8, ineffective penalty terms on a small vocabulary, repetition) and their fixes.
Figures & tables
| Domain | orig (BPE reading) | LD3-base | |
|---|---|---|---|
| zh (Chinese web text) | 0.778 0.021 | 0.742 0.018 | 0.036 |
| zh_ood (Chinese, out-of-domain) | 1.022 0.013 | 0.997 0.013 | 0.025 |
| en | 0.587 0.017 | 0.549 0.014 | 0.038 |
| code | 0.205 0.008 | 0.190 0.008 | 0.015 |
| Model / mode | Recitation | QA correct | Kept | Marginal | Degraded |
|---|---|---|---|---|---|
| orig non-thinking | 22/22 | 22/22 | 21 | 1 | 0 |
| orig thinking | 21/22 | 22/22 | 22 | 0 | 0 |
| LD3-base non-thinking | 22/22 | 13/22 | 11 | 0 | 11 |
| LD3-base thinking | 22/22 | 14/22 | 13 | 0 | 9 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Mode | orig | LD3-base | [95% CI] |
|---|---|---|---|---|
| GSM8K | thinking | 97.0 [96.1, 97.9] | 93.0 [91.7, 94.4] | 4.0 [ 5.3, 2.9] |
| MATH-500 | thinking | 94.0 [91.8, 96.0] | 95.4 [93.4, 97.0] | +1.4 [ 0.8, +3.6] |
| HumanEval+ | thinking | 90.2 [85.4, 94.5] | 88.4 [83.5, 93.3] | 1.8 [ 7.3, +3.7] |
| C-Eval | thinking | 89.9 [88.3, 91.5] | 85.4 [83.4, 87.2] | 4.5 [ 6.3, 2.8] |
| MMLU-Pro | thinking | 80.1 [77.6, 82.5] | 78.8 [76.3, 81.4] | 1.3 [ 3.3, +0.6] |
| LongBench-v2† | thinking | 27.4 [19.7, 35.9] | 39.3 [30.8, 47.9] | +11.9 [+3.4, +21.4] |
| 域 | orig(BPE 读法) | LD3-base | |
|---|---|---|---|
| zh(中文网文) | 0.778 0.021 | 0.742 0.018 | 0.036 |
| zh_ood(中文,域外) | 1.022 0.013 | 0.997 0.013 | 0.025 |
| en | 0.587 0.017 | 0.549 0.014 | 0.038 |
| code | 0.205 0.008 | 0.190 0.008 | 0.015 |
| 模型 / 档 | 复述 | 问答答对 | 保住 | 勉强 | 退化 |
|---|---|---|---|---|---|
| orig 非思考 | 22/22 | 22/22 | 21 | 1 | 0 |
| orig 思考 | 21/22 | 22/22 | 22 | 0 | 0 |
| LD3-base 非思考 | 22/22 | 13/22 | 11 | 0 | 11 |
| LD3-base 思考 | 22/22 | 14/22 | 13 | 0 | 9 |
| 数据集 | 档 | orig | LD3-base | [95% 区间] |
|---|---|---|---|---|
| GSM8K | 思考 | 97.0 [96.1, 97.9] | 93.0 [91.7, 94.4] | 4.0 [ 5.3, 2.9] |
| MATH-500 | 思考 | 94.0 [91.8, 96.0] | 95.4 [93.4, 97.0] | +1.4 [ 0.8, +3.6] |
| HumanEval+ | 思考 | 90.2 [85.4, 94.5] | 88.4 [83.5, 93.3] | 1.8 [ 7.3, +3.7] |
| C-Eval | 思考 | 89.9 [88.3, 91.5] | 85.4 [83.4, 87.2] | 4.5 [ 6.3, 2.8] |
| MMLU-Pro | 思考 | 80.1 [77.6, 82.5] | 78.8 [76.3, 81.4] | 1.3 [ 3.3, +0.6] |
| LongBench-v2† | 思考 | 27.4 [19.7, 35.9] | 39.3 [30.8, 47.9] | +11.9 [+3.4, +21.4] |
Byte-Level Language Model
Byte-level language models predict text directly from bytes, avoiding fixed token vocabularies while balancing character-level flexibility against sequence efficiency.
See all 34 papers