Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text

Lukas Edman; Alexander Fraser

cs.CLOct 13, 2026

Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text

Authors: Lukas Edman, Alexander Fraser

Organizations: School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute

Abstract

This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.

Figures & tables

Figure 1: Preprocessing pipelines we investigate, including packed-translated ( PT ), packed-summarized ( PS ), and packed-summarized-translated ( PST ). Packing process not shown for brevity.

Text	Summary
Well it’s just that, you know, a pound, or a hundred pounds today, is not the same as a hundred pounds in a year’s time, or two, two years’ time. Right. So that would be opportunity cost? That’s right, yes, that’s right, so, in actual fact, this area here, right, is going to be the discounted sum of all future rural incomes.	A pound today isn’t the same as a pound next year. This represents opportunity cost. The discounted sum of future rural incomes is represented here.
A: you can smell ether in the air sometimes A: It keeps the neighborhoods happy I guess. B: I guess so. A: Uh, cause they clean the, uh, the lenses for cameras A: and they, they make film here A: but it’s, you know, it’s under so much control. A: You know, sulphur dioxide is the big emittant from them A: but that’s really getting under control now. B: Right. B: I, you know, I don’t know in air what they do. B: Uh, I, I haven’t run across any major pollutants down there that I’ve, I’ve really seen a lot of, you know, about. B: Uh, I, I know I’ve seen like, uh, B: my grandparents live in Corpus Christi, Texas B: and I know they’ve, there’s a lot of refineries down there B: and that, that’s some pretty potent stuff they put up in the air. B: I, but I don’t know how, uh, you know, B: there’s a difference in what you can smell and what you, uh, you know, what’s bad. A: Be interesting to see when, as Mexico develops industrially whether, you know, without	A: You can smell ether in the air sometimes. It keeps the neighborhoods happy. They clean camera lenses and make film. But it’s controlled well. Sulphur dioxide is a big emitter, but it’s getting under control now. B: I guess so. I don’t know much about air pollution. I’ve never seen many major pollutants around. My grandparents live near Corpus Christi with lots of refineries, which can be potent. But smelling something doesn’t mean it’s harmful. It would be interesting to see how Mexico develops industrially.

Table 1: Examples of original text and their respective LLM-generated summaries.

Preprocessing	$\mathit{\emptyset}$	P	PT	PS	PST
Zero-shot
eng	57.8	55.4	55.8	57.2	59.0
nld	59.9	58.2	54.0	55.6	56.5
zho	51.9	52.2	49.2	52.5	52.8
Fine-tune
eng	29.1	41.0	43.8	40.9	45.8

Table 2: Effect of the 3 preprocessing steps: Packing ( P ), Summarization ( S ), and Translation ( T )

Model size	Base		Large
LoRA?	No	Yes	No	Yes
Zero-shot
eng	59.0	59.0	59.5	59.5
nld	56.5	56.5	57.8	57.8
zho	52.8	52.8	52.9	52.9
Fine-tune

Table 3: Effect of model size and using LoRA for fine-tuning.

	20k		Char
LoRA	No	Yes	No	Yes
Zero-shot
eng	59.0	59.0	60.9	60.9
nld	56.5	56.5	58.6	58.6
zho	52.8	52.8	52.7	52.7
Fine-tune

Table 4: Comparison of 20k vocabulary to character-level vocabulary, and the effect of LoRA.

Model	O.B.	$\mathit{\emptyset}$	P	PST	PST Large	PST Char Large
Zero-shot
eng	56.4	57.8	55.4	59.0	59.5	61.4
nld	57.8	59.9	58.2	56.5	57.8	59.1
zho	51.3	51.9	52.2	52.8	52.9	52.9
Fine-tune
eng	28.3	29.1	41.1	45.8	45.0	47.5

Table 5: Comparison of the official baseline ( O.B. ) compared to various steps added, and final submitted model (right).

Model	O.B.	PST Char	PST Char Large
Zero-shot
eng	53.6	57.1	57.4
nld	55.0	55.3	56.1
zho	47.3	49.3	51.0
Fine-tune
eng	43.1	50.4	53.3

Table 6: Comparison of the official baseline ( O.B. ) to our two leaderboard submissions, including hidden tasks. PST Char and PST Char Large are equivalent to Qwenfant-0.1B and Qwenfant-0.4B , respectively.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Parameter	Value
Model	Architecture	Qwen3
	Hidden Size	768
	Intermediate Size	3072
	Layers	12
	Vocabulary Size	20000
Training	Context Size	512

Appendix

Table 7: Hyperparameters used (except where otherwise indicated).

Panel of Figure 2 — Figure 2: Prompts used for translation and summarization.

Explore similar work

Feb 23, 2026

BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

The goal of the BabyLM is to stimulate new research connections between cognitive modeling and language model pretraining. We invite contributions in this vein to the BabyLM Workshop, which will also include the 4th iteration of the BabyLM Challenge. As in previous years, the challenge features two ``standard'' tracks (Strict and Strict-Small), in which participants must train language models on under 100M or 10M words of data, respectively. This year, we move beyond our previous English-only pretraining datasets with a new Multilingual track, focusing on English, Dutch, and Chinese. For the workshop, we call for papers related to the overall theme of BabyLM, which includes training efficiency, small-scale training datasets, cognitive modeling, model evaluation, and architecture innovation.

Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul +7

IBM Research, MIT · ETH Zürich · Cornell University +7

Language Model Pretraining Small Language Models Multilingual Language Models

Ask PDF

Jul 12, 2026

The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese

This paper describes the first ChineseBabyLM challenge, which will be held in the 2026 NLPCC conference. The challenge calls for researchers to train language models from scratch with 100 million Chinese tokens and evaluates the models on 3 tracks of tasks: NLU, cognitive alignment and Hanzi knowledge. There is no restriction on tokenizer, model architecture and the number of training epochs. Details of the challenge can be found in https://chinese-babylm.github.io/.

Siyuan Song, Zhiheng Qian, Yunhao Zhang +11

Princeton University · Shanghai Jiao Tong University · Chinese Academy of Sciences +5

Language Model Pretraining LLM Evaluation Language Modeling

Ask PDF

Sep 24, 2026

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.

Dries Rooryck, Alex Cai, Yonatan Belinkov +2

Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel

Multilingual Language Models Language Model Pretraining Cross-Lingual Representation Alignment Curriculum Learning

Ask PDF