Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

Pedro Carvalho Brom

cs.CLOct 29, 2026

Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

Authors: Pedro Carvalho Brom

Organizations: Department of Mathematics Federal Institute of Brasília Brasília, Brazil

Abstract

Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It proposes no new method. It releases a tested Python package and two notebooks in Brazilian Portuguese with autoregressive and masked diffusion generation on one network, a preregistered comparison protocol, legal tasks built from text alone with a Kneser-Ney floor and public yardsticks, reference numbers for models of 11 to 12M parameters, and a catalogue of defects that only measurement revealed. On Brazilian prose and federal laws, at minutes per run, the autoregressive mode peaks at 64 epochs over the same text and then memorises it, while masked diffusion improves up to 256. At their best points on test, the diffusion bound exceeds the autoregressive likelihood by 0.06 bits per byte, within the preregistered minimum effect. On held-out laws and on acts published after training, the model ties a 6-gram floor in likelihood on the same windows, and it loses to the floor at continuing those acts. It beats the floor at detecting tampered text, and fills cross-references more often than the floor and public models of 160M and 630M parameters, only in decrees, a genre absent from training, where those scorers stay near or below chance, and in filling the blank only when each group weighs the same, since per item it ties the 630M model. On laws, the public models, trained with about 5x10^2 to 10^5 times its compute, lead in tampered text and continuation. The blind human reading is preregistered and not part of this version.

Figures & tables

Resource	Language	Tokenizer	Generation	Evaluation	Prereg.	Code licence
Annotated Transformer	English, German	BPE of the original setup	Encoder–decoder	BLEU	No	MIT
nanoGPT	English	GPT-2 BPE reused, or characters	Autoregressive	Validation loss	No	MIT
Cramming	English	WordPiece trained on the data	None, masked encoder	GLUE	No	MIT
TinyStories	English, synthetic	GPT-Neo BPE reused	Autoregressive	Grading by GPT-4	No	Data CDLA-Sharing 1.0
BabyLM	English	Participant’s choice	Participant’s choice	BLiMP, (Super)GLUE and others	No	Varies
This tutorial	Portuguese	Byte-level BPE trained on the data	Autoregressive and masked diffusion, one network	Kneser–Ney floor, public yardsticks, four domain tasks, calibration	Yes, hashed	CC BY-NC-SA 4.0

Table 1: Comparable resources on axes that can be checked in their papers and repositories.

Box	Module (gloss)	Responsibility	Checks
0	config (configuration)	Validated configuration of corpus, tokenizer, network, mode and budget
1	dados (data)	Domain adapters, cleaning, segmentation, block split, proper-name lists	Corpus G0 to G5
2	tokenizador (tokenizer)	Tokenizer contract, didactic BPE, library adapter	Tokenizer G0 to G5
3	janelas (windows)	Windows of $L$ tokens per document, with [EOS] and padding predicted	Formulation G1, G2
4	mascaramento (masking)	Stratified draw of $l$ and masking of exactly $l$ positions	Formulation G0
5	modelo (model)	Transformer with causal or bidirectional attention, [MASK] logit at $-\infty$	Formulation G0

Table 2: The eleven boxes of the pipeline, their modules with an English gloss of each Portuguese name, and the checks that guard them. The decision documents of the repository define each check.

Task	Items	Groups	Share of groups for power 0.8
T1 Filling in the blank	337	130	0.655
T2 Inserted or own provision	3,095 (313 inserted)	643	0.570
T3 Tampered text	1,185	469	0.581
T4 Continuation	2,251	615	0.571

Table 3: Items of the four tasks on the external acts of 2025 and 2026, and the share of groups that must favour the model for a sign test to reach power 0.8 under Holm’s correction in its worst case. The T2 share is kept as preregistered, although its sign test turned out not to measure the balanced accuracy ( Section 8.12 ). T1 holds 308 cross-references and 29 numbers.

Figure 1: Validation bits per byte against epochs over the same text, corpus A. Points are seeds (5 at the descriptive budgets, 6 at the decisive budget, dotted line), lines join the means. The dashed line is the 5-gram floor scored on the same windows of 256 tokens. The diffusion values are upper bounds.

Epochs	Seeds	Autoregressive	Diffusion	$\Delta$	95% interval	Randomization $p$
1	5	2.692	2.808	0.115	$[0.088,0.161]$
4	5	2.314	2.767	0.453	$[0.438,0.473]$
16	5	1.955	2.748	0.793	$[0.775,0.811]$
64	5	1.806	2.167	0.361	$[0.332,0.395]$
256	6	2.851	1.825	$-1.026$	$[-1.099,-0.954]$	0.031

Table 4: Mean validation bits per byte per mode and paired difference

\Delta=\mathrm{bpb}_{\mathrm{dif}}-\mathrm{bpb}_{\mathrm{AR}}

, corpus A. Intervals from the two-level bootstrap, whose coverage in the pilot was 0.918. At 256 epochs the decision rests on the randomization test.

Figure 2: Bits per byte on the training partition (dashed, open points) and on validation (solid) across the budget grid, mean over seeds. At 256 epochs the autoregressive mode nearly reproduces its training text, and the gap to validation is the overfitting of Figure 1 . The diffusion values are upper bounds.

Tokenizer	Epochs	$L$	Bits per byte	Difference to BPE
BPE, $V=4096$	16	256	1.954
Bytes, $V=260$	5.1	256	2.751	$+0.797$ $[+0.760,+0.834]$
Bytes, $V=260$	4.5	911	3.368	$+1.414$ $[+1.405,+1.422]$
BPE, $V=4096$	256	256	2.849
Bytes, $V=260$	81.3	256	1.849	$-1.000$ $[-1.024,-0.976]$
Bytes, $V=260$	71.8	911	1.840	$-1.010$ $[-1.026,-0.993]$

Table 5: Byte-level control at equal compute, autoregressive mode, corpus A, 3 seeds. Epochs are passes over the training text,

L

is the window in symbols, and the difference to BPE is paired by seed, with a 95% interval by

t

on 2 degrees of freedom.

Figure 3: Validation bits per byte against epochs over the same text, corpus B,

V=1024

. Points are seeds, lines join the means, and the dashed line is the 6-gram floor on the same windows of 256 tokens. The diffusion values are upper bounds.

Epochs	Seeds	Autoregressive	Diffusion	$\Delta$	95% interval	Randomization $p$
1	5	2.242	3.113	0.871	$[0.839,0.911]$
4	5	1.626	2.994	1.368	$[1.323,1.421]$
16	5	1.101	2.046	0.945	$[0.864,1.053]$
64	5	1.066	1.329	0.263	$[0.189,0.351]$
256	6	1.487	1.135	$-0.353$	$[-0.509,-0.188]$	0.031

Table 6: Mean validation bits per byte per mode and paired difference

\Delta=\mathrm{bpb}_{\mathrm{dif}}-\mathrm{bpb}_{\mathrm{AR}}

, corpus B. Intervals from the two-level bootstrap, whose coverage in the pilot was 0.919. At 256 epochs the decision rests on the randomization test.

Figure 4:

\Delta=\mathrm{bpb}_{\mathrm{dif}}-\mathrm{bpb}_{\mathrm{AR}}

on the test partition at 256 epochs, per work of corpus A and per group of corpus B, mean of the 6 seeds with a

t

interval, and the total with the two-level bootstrap interval. Left of zero, diffusion is better in likelihood.

Figure 5: Validation bits per byte at 256 epochs on corpus A with one factor changed at a time, seeds 300 to 302 as points and their mean as a line. The dotted line marks the value used in the comparison.

K

applies to the diffusion bound only, and the dashed line is the exact autoregressive value.

Figure 6: Where the autoregressive Transformer beats the 5-gram floor on the validation of corpus A, in nats per token, at 16 and 64 epochs. Left, by what the 5-gram knows of the context. Right, by how often the predicted token occurs in training. Above zero the Transformer is better.

Figure 7: Bits per byte on the odd validation documents of corpus A for the autoregressive Transformer, the 5-gram floor and their mixture, with the mixing weight

\lambda

fitted on the even documents. The weight of the Transformer grows with training.

Figure 8: Mean entropy of a sample over its own tokens against its generative perplexity under Tucano 630M, at sampling temperatures 0.6 to 1.2 (labels), 64 samples per seed and 3 seeds, corpus A. Dashed, 16 epochs. Solid, 256 epochs.

				ECE			Selective at 0.9		Deletion AUC
Corpus	Mode	Epochs	$T$	$T=1$	fitted	ROC AUC	Coverage	Kept	Importance	Random
A	AR	1	0.95	0.014	0.022	0.647	0.000	0/5	0.562	0.811
A	AR	4	0.96	0.012	0.025	0.748	0.000	0/5	0.234	0.702
A	AR	16	0.98	0.016	0.022	0.747	0.016	5/5	0.163	0.669
A	AR	64	1.18	0.106	0.030	0.779	0.029	5/5	0.102	0.501
A	AR	256	2.19	0.398	0.014	0.792	0.013	4/6	0.080	0.328

Table 7: Calibration and deletion per budget, mean over the seeds of each budget (5, or 6 at 256 epochs), on validation predictions for calibration and test passages for deletion. ECE at temperature 1 and at the fitted temperature

T

, ROC AUC of confidence as a detector of correct predictions, selective answer at a 90% target (coverage and seeds whose promise held on the held-out half), and area under the deletion curve by importance and at random.

Figure 9: Left, reliability of the first decisive seed of corpus A at 256 epochs, at temperature 1 (dashed, open points) and at the fitted temperature (solid). Right, expected calibration error across the budget grid, mean over seeds, corpus A (circles) and corpus B (squares, lighter).

Figure 10: Left, deletion curves of the first decisive seed of corpus A at 256 epochs, context words removed by importance (solid) and at random (dashed), mean over 300 test passages. Right, probability of the decision with the whole context removed, relative to the full context, across the budget grid, mean over seeds, corpus A (solid) and corpus B (dashed).

Figure 11: Bits per byte on the external laws of 2025 and 2026 against the training text of corpus B, nested fractions of the training groups. Autoregressive Transformer (points are seeds), 6-gram floor and their mixture, with the mixing weight fitted on validation per seed.

Scorer	T1 Filling in the blank	T2 Inserted or own	T3 Tampered text	T4 Continuation
All acts	130 groups	643 groups	469 groups	615 groups
6-gram floor	0.241 [0.179, 0.301]	0.502 [0.488, 0.515]	0.497 [0.456, 0.537]	1.355 [1.324, 1.385]
autoregressive	0.399 [0.349, 0.450]	0.467 [0.442, 0.492]	0.559 [0.528, 0.589]	1.425 [1.383, 1.468]
masked diffusion	0.310 [0.258, 0.361]	0.502 [0.476, 0.529]	0.550 [0.523, 0.578]	1.852 [1.816, 1.888]
TeenyTinyLlama 160M	0.269 [0.205, 0.331]	0.541 [0.495, 0.588]	0.567 [0.527, 0.607]	0.791 [0.779, 0.804]
Tucano 160M	0.294 [0.227, 0.358]	0.510 [0.491, 0.529]	0.458 [0.419, 0.498]	0.737 [0.726, 0.748]

Table 8: Scores on the four tasks over the external acts of 2025 and 2026, for all acts and by genre, mean of the per-group values with a 95% wild cluster bootstrap interval. T1 and T3 accuracy, T2 balanced accuracy, T4 bits per byte (lower is better). Complementary laws enter the total only, with 7 to 17 groups per task. The project’s model is at the best validation point of each mode, 5 seeds for the autoregressive mode and 6 for diffusion, whose T4 value is an upper bound. T2 is exploratory.

Figure 12: Score on each task by genre of the act, ordinary laws (filled squares) and decrees (open circles), with the 95% interval and chance as a dotted line. The T4 axis is reversed so that right is better in every panel. In decrees the floor and the public yardsticks stay near or below chance in T1 and T3.

	T1 accuracy by class			Picks the most frequent		Accuracy in decrees
Scorer	Numbers, laws (29)	Cross-ref., laws (77)	Cross-ref., decrees (231)	T1 of 4 options	T3 of 2 versions	T1, single- item groups	T3, single- item groups	T3, other groups
6-gram floor	0.379	0.286	0.234	0.487	0.557	0.149	0.377	0.695
autoregressive	0.345	0.395	0.410	0.476	0.676	0.385	0.482	0.659
masked diffusion	0.425	0.301	0.314	0.689	0.699	0.234	0.526	0.580
TeenyTinyLlama 160M	0.517	0.481	0.294	0.309	0.586	0.179	0.475	0.639
Tucano 160M	0.621	0.390	0.329	0.243	0.422	0.194	0.294	0.682

Table 9: Diagnostic of T1 and T3 on the external acts from the stored choices, item accuracy or rate, mean over seeds. Laws include the complementary laws. Most frequent refers to the count, in the training text of corpus B, of the value that tells the candidates apart, the number or the cited article in T1, and the term, number or cited article in T3, where shuffled subdivisions have no such value. Single-item groups are decree groups with one item, which weigh as much as any other group in Table 8 .

Figure 13: Score on each task of the external set against training compute,

6ND

, on a logarithmic scale. Filled points are the home-built model, open points the public yardsticks, with 95% intervals. The shaded steps are the Pareto frontier, where no cheaper scorer does better, with the 6-gram floor entering at the left edge because it trains in seconds on one CPU core. The T4 axis is reversed so that better is up in every panel. The legend gives parameters, compute relative to the home-built autoregressive model to two significant figures and training energy, an upper bound at the 500 W cap for the home-built model and the values reported by the authors for the yardsticks. Scores are over all acts, so the T1 and T3 panels carry the decrees, where the floor and the yardsticks stay near or below chance ( Figure 12 ).

	Score		Selective coverage
Decider	T2	T3	T2	T3
TF-IDF with logistic regression (floor)	0.544 [0.487, 0.600]	0.573 [0.553, 0.593]
autoregressive checkpoint	0.499 [0.459, 0.539]	0.501 [0.500, 0.503]	0.90% (0.89)	2.29% (0.51)
masked diffusion checkpoint	0.488 [0.441, 0.536]	0.514 [0.511, 0.517]	0.05% (0.67)	0.59% (0.54)
Laya multilingual, fine-tuned	0.485 [0.446, 0.524]	0.500 [0.494, 0.506]	0.00%	0.00%
Laya multilingual, no fine-tuning	0.515 [0.494, 0.535]	0.499 [0.492, 0.505]

Table 10: Typed deciders on the external acts of 2025 and 2026. T2 balanced accuracy and T3 accuracy, mean of the per-group values with a 95% interval. Selective answer at a 95% target, mean coverage over seeds and, in parentheses, the accuracy of the covered items.

Check	What it detected	Fix
Length test with exact conditionals	If diffusion receives the document length without paying for it, it gains the entropy of the length, 0.856 nat (1.235 bits) per document in the test, matched within $10^{-9}$	Predict [EOS] and padding in both modes
Weight of the loss	Weight $1/t$ unbounded near $t=0$ , infinite variance without a cut-off, and the cut-off truncates the bound	Number-of-masks form, weight at most $L$ ( Propositions 1 and 2 )
Logit gradient in fp16	With one masked token per sequence and scale $2^{16}$ , the logit gradient exceeds the fp16 maximum of 65,504 for the mean over masked tokens with weight $1/t$ at batch 2, 4 and 64	Number-of-masks form, peak bounded by scale over batch, 16,384 at batch 4
Precision sentinel	Median per-tensor error of 1.38 and 1.45, maximum 7.11, against a margin of 0.05, caused by dropout in the sentinel itself	Both passes of the sentinel in eval mode, with a test
Range of the sampling noise	In fp16 the uniform draw has 2,048 values and the largest Gumbel noise is 7.62, against 16.64 in fp32 and 36.74 in fp64	Categorical draws in float64 [ Zheng et al., 2024 ]
Enumeration of all sequences	With a free [MASK] logit the distribution sums to 0.5906 instead of 1, and with a target visible to its own prediction to 1.2837	[MASK] logit at $-\infty$

Table 11: Checks that detected a defect during construction, with what was measured and the fix adopted. Numbers come from the tests and scripts of the repository.

Figure 14: Cumulative distribution of the per-tensor p95 relative error between fp16 and fp32 gradients over all sentinel records of the 12 final runs at 256 epochs. Dashed, the sentinel in training mode, where each pass draws its own dropout mask. Solid, the sentinel in eval mode. The vertical line is the preregistered margin of 0.05.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Criterion	State
Formulation G0	Enumeration of all orders, distribution sums to 1 within $10^{-9}$ , NELBO bounds the exact value, injected defects fail	Passed. Defects give 0.5906 and 1.2837
Formulation G1	Length test through the package iterator, difference within $10^{-9}$	Passed
Formulation G2	Padding fraction measured before fixing $L$	$L=256$ , 12.2% padding
Tokenizer G0, G2	Equal merges and ids, vocabulary rule on the real held-out text	Passed, $V=4096$ (A), $1024$ (B)
Tokenizer G3, G4	Training and encoding times and equality on the Colab CPU	Not in this version
Corpus G1, G3	Segmentation and minimum block (A), pages, volume and largest group (B)	Passed. B: 444 of 444 pages, 4.76 MB, largest group 5.78%

Appendix

Table 12: Checks of the five design decisions and their state. “Not in this version” marks checks that depend on people or machines outside the scope of this version.

		Autoregressive		Diffusion
Learning rate	Dropout	16 epochs	256 epochs	16 epochs	256 epochs
$3\times 10^{-4}$	0	1.953	4.853	2.750	1.843
$3\times 10^{-4}$	0.1	1.931	3.668	2.756	1.863
$3\times 10^{-4}$	0.2	1.953	2.850	2.763	1.930
$10^{-3}$	0	1.939	5.602	2.752	1.873
$10^{-3}$	0.1	1.851	4.271	2.747	1.826

Appendix

Table 13: Hyperparameter search on corpus A, mean validation bits per byte of 2 seeds. In bold, the configuration chosen for each mode at 256 epochs.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 522 papers

Mar 23, 2026

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.

Caio Vicentino

Independent Researcher

Diffusion Language Models Autoregressive Language Modeling Diffusion Language Model Benchmarking LLM Training Diffusion Language Model Training Language Model Generation Evaluation

Ask PDF

Jun 17, 2026

Sumi: Open Uniform Diffusion Language Model from Scratch

Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.

Mengyu Ye, Keito Kudo, Wataru Ikeda +3

1Tohoku University

Language Model Pretraining Diffusion Language Models Diffusion Language Model Training

Ask PDF

Feb 16, 2026

Scaling Beyond Masked Diffusion Language Models

Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms

Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang +4

NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA +2

Diffusion Language Models Diffusion Models Diffusion Language Model Training Diffusion Language Model Benchmarking Masked Diffusion Models Diffusion Language Model Inference

Ask PDF

Abstract

Language Modeling

Explore similar work

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Sumi: Open Uniform Diffusion Language Model from Scratch

Scaling Beyond Masked Diffusion Language Models