cs.CLOct 13, 2026
SaveQwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
Organizations: School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Abstract
This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.
Figures & tables
Figure 1: Preprocessing pipelines we investigate, including packed-translated ( PT ), packed-summarized ( PS ), and packed-summarized-translated ( PST ). Packing process not shown for brevity.
| Text | Summary |
| Well it’s just that, you know, a pound, or a hundred pounds today, is not the same as a hundred pounds in a year’s time, or two, two years’ time. Right. So that would be opportunity cost? That’s right, yes, that’s right, so, in actual fact, this area here, right, is going to be the discounted sum of all future rural incomes. | A pound today isn’t the same as a pound next year. This represents opportunity cost. The discounted sum of future rural incomes is represented here. |
| A: you can smell ether in the air sometimes A: It keeps the neighborhoods happy I guess. B: I guess so. A: Uh, cause they clean the, uh, the lenses for cameras A: and they, they make film here A: but it’s, you know, it’s under so much control. A: You know, sulphur dioxide is the big emittant from them A: but that’s really getting under control now. B: Right. B: I, you know, I don’t know in air what they do. B: Uh, I, I haven’t run across any major pollutants down there that I’ve, I’ve really seen a lot of, you know, about. B: Uh, I, I know I’ve seen like, uh, B: my grandparents live in Corpus Christi, Texas B: and I know they’ve, there’s a lot of refineries down there B: and that, that’s some pretty potent stuff they put up in the air. B: I, but I don’t know how, uh, you know, B: there’s a difference in what you can smell and what you, uh, you know, what’s bad. A: Be interesting to see when, as Mexico develops industrially whether, you know, without | A: You can smell ether in the air sometimes. It keeps the neighborhoods happy. They clean camera lenses and make film. But it’s controlled well. Sulphur dioxide is a big emitter, but it’s getting under control now. B: I guess so. I don’t know much about air pollution. I’ve never seen many major pollutants around. My grandparents live near Corpus Christi with lots of refineries, which can be potent. But smelling something doesn’t mean it’s harmful. It would be interesting to see how Mexico develops industrially. |
Table 1: Examples of original text and their respective LLM-generated summaries.
| Preprocessing | P | PT | PS | PST | |
| Zero-shot | |||||
| eng | 57.8 | 55.4 | 55.8 | 57.2 | 59.0 |
| nld | 59.9 | 58.2 | 54.0 | 55.6 | 56.5 |
| zho | 51.9 | 52.2 | 49.2 | 52.5 | 52.8 |
| Fine-tune | |||||
| eng | 29.1 | 41.0 | 43.8 | 40.9 | 45.8 |
Table 2: Effect of the 3 preprocessing steps: Packing ( P ), Summarization ( S ), and Translation ( T )
| Model size | Base | Large | ||
| LoRA? | No | Yes | No | Yes |
| Zero-shot | ||||
| eng | 59.0 | 59.0 | 59.5 | 59.5 |
| nld | 56.5 | 56.5 | 57.8 | 57.8 |
| zho | 52.8 | 52.8 | 52.9 | 52.9 |
| Fine-tune | ||||
Table 3: Effect of model size and using LoRA for fine-tuning.
| 20k | Char | |||
| LoRA | No | Yes | No | Yes |
| Zero-shot | ||||
| eng | 59.0 | 59.0 | 60.9 | 60.9 |
| nld | 56.5 | 56.5 | 58.6 | 58.6 |
| zho | 52.8 | 52.8 | 52.7 | 52.7 |
| Fine-tune | ||||
Table 4: Comparison of 20k vocabulary to character-level vocabulary, and the effect of LoRA.
| Model | O.B. | P | PST | PST Large | PST Char Large | |
| Zero-shot | ||||||
| eng | 56.4 | 57.8 | 55.4 | 59.0 | 59.5 | 61.4 |
| nld | 57.8 | 59.9 | 58.2 | 56.5 | 57.8 | 59.1 |
| zho | 51.3 | 51.9 | 52.2 | 52.8 | 52.9 | 52.9 |
| Fine-tune | ||||||
| eng | 28.3 | 29.1 | 41.1 | 45.8 | 45.0 | 47.5 |
Table 5: Comparison of the official baseline ( O.B. ) compared to various steps added, and final submitted model (right).
| Model | O.B. | PST Char | PST Char Large |
| Zero-shot | |||
| eng | 53.6 | 57.1 | 57.4 |
| nld | 55.0 | 55.3 | 56.1 |
| zho | 47.3 | 49.3 | 51.0 |
| Fine-tune | |||
| eng | 43.1 | 50.4 | 53.3 |
Table 6: Comparison of the official baseline ( O.B. ) to our two leaderboard submissions, including hidden tasks. PST Char and PST Char Large are equivalent to Qwenfant-0.1B and Qwenfant-0.4B , respectively.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value | |
| Model | Architecture | Qwen3 |
| Hidden Size | 768 | |
| Intermediate Size | 3072 | |
| Layers | 12 | |
| Vocabulary Size | 20000 | |
| Training | Context Size | 512 |
Table 7: Hyperparameters used (except where otherwise indicated).
Figure 2: Prompts used for translation and summarization.