Tucano 2 Cool: Better Open Source LLMs for Portuguese

Nicholas Kluge Corrêa; Aniket Sen; Shiza Fatimah; Sophia Falk; Lennard Landgraf; Julia Kastner; Lucie Flek

cs.CLMar 3, 2026

Tucano 2 Cool: Better Open Source LLMs for Portuguese

Authors: Nicholas Kluge Corrêa, Aniket Sen, Shiza Fatimah, Sophia Falk, Lennard Landgraf, Julia Kastner, Lucie Flek

Organizations: Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Center for Science and Thought · Helmholtz-Institut für Strahlen- und Kernphysik ·

Abstract

We present Tucano 2, a fully open suite of large language models (LLMs) with 0.5-3.7 billion parameters, designed to address certain gaps in open-source development for Portuguese LLMs. Following our previous works, we now extend our dataset, GigaVerbo-v2, to a new degree of quality and scale, while also introducing a new synthetic dataset, GigaVerbo-v2 Synth, aimed at filling missing gaps in GigaVerbo-v2, and two post-training datasets, GigaVerbo-v2 SFT and GigaVerbo-v2 Preferences, that allow Portuguese LLMs to be trained in domains like retrieval augmented generation, coding, tool use, chain-of-thought reasoning, and many other domains of interest. Through extensive ablation studies, we design both pretraining and continual pretraining recipes for the Tucano 2 suite (Base, Instruct, and Think), which achieve state-of-the-art performance on several Portuguese-language modeling benchmarks. We also extend and refine the evaluation harness introduced in our earlier work, yielding a comprehensive evaluation suite that provides strong signals across different pretraining, continual pretraining, and post-training regimes. All artifacts associated with Tucano 2 are openly released, including training recipes, logs, and source code, ensuring that our work is reproducible, accessible, and extendable by the broader Portuguese NLP community.

Figures & tables

Figure 1: Impact of Educational & Synthetic Data (46B tokens). The Edu+Synth and Edu mixtures achieve the best performance across benchmarks, substantially outperforming the Non-Edu mixture and the Tucano-2b4 baseline. The percentage values represent the relative increase/decrease in performance with regard to the Tucano-2b4 baseline.

Model	Tokens	Vocab Size	SF	PCW	Chars/Token
Tucano2-0.6B-Base	906,049	49,152	1.51	0.56	2.88
GlorIA-1.3B	950,349	50,258	1.58	0.47	3.04
Llama-3.2-1B	1,147,831	128,256	1.91	0.61	2.72
SmolLM3-3B-Base	1,147,831	128,256	1.91	0.61	2.72
OLMo-2-0425-1B	1,150,948	100,278	1.92	0.61	2.71
Qwen3-0.6B	1,155,951	151,669	1.93	0.61	2.68

Table 1: Tokenizer evaluation on Portuguese text (600,000 words). Our tokenizer achieves the best compression efficiency across all metrics.

Figure 2: Pretraining loss curve across 195,000 steps ( $\sim$ 408B tokens).

Figure 3: Normalized Preferred Metric (NPM) scores on the Easy Set evaluations.

Model	Total Avg.	Easy Set	Hard Set
Qwen3-0.6B-Base	29.40	26.41	32.38
Tucano2-0.6B-Base	20.64	40.28	0.99
Qwen2.5-0.5B	19.89	18.70	21.09
Curió-1.1B	19.23	39.16	$-0.69$
Tucano-2b4	17.88	33.55	2.20
Curió-Edu-1B	17.72	34.77	0.67

Table 2: Aggregate benchmark (NPM) results.

Figure 4: Comparison of our 0.6-Base model against Tucano-1b1 and Curió-edu-1b1.

Model	Total Avg.	Easy Set	Hard Set
Tucano2-qwen-3.7B-Base	59.21	57.41	61.00
Qwen2.5-7B	57.97	54.12	61.83
Qwen3-4B-Base	57.86	52.52	63.20
SmolLM3-3B-Base	50.25	54.06	46.44
Qwen2.5-3B	50.16	47.69	52.62
Tucano2-qwen-1.5B-Base	47.90	47.97	47.82

Table 3: Aggregate benchmark results (NPM) comparisons against other baselines.

Figure 5: Per-benchmark performance comparison. Bars indicate the absolute difference in evaluation scores between the continually pretrained model and its base.

Figure 6: Aggregate NPM score vs. total compute (FLOPs).

Subset	Samples	Tokens
Retrieval	1,977,667	1,013,172,488
General	1,235,976	700,483,545
Math	220,042	83,234,218
Structured	163,542	70,632,221
Summarization	128,669	90,310,108
Code	80,774	84,389,567

Table 4: Dataset composition after filtering.

	Total Avg.	K&R (NPM)	Instruct	Coding
Tucano2-qwen-3.7B-Instruct	53.64	56.22	41.67	47.56
Jurema-7B	53.03	50.66	47.00	75.61
Qwen2.5-3B-Instruct	51.71	47.34	63.33	70.73
Qwen3-4B	51.36	42.33	79.33	86.59
Gemma-3-Gaia-PT-BR-4b-it	49.93	45.00	70.33	64.02
SmolLM3-3B	49.54	43.99	69.67	68.29

Table 5: Post-training results for Instruct variants.

Figure 7: Per-benchmark comparison of Tucano2-qwen-3.7B-Instruct against Qwen3-4B and Gemma-3-Gaia-PT-BR-4b-it.

	Total Avg.	K&R (NPM)	Instruct
Tucano2-qwen-3.7B-Think	51.27	54.07	31.67
SmolLM3-3B	48.58	46.28	64.67
Qwen3-4B	46.35	40.97	84.00
Qwen3-1.7B	36.54	32.00	68.33
Tucano2-qwen-1.5B-Think	27.54	26.67	33.67
Qwen3-0.6B	24.11	19.22	58.33

Table 6: Post-training results for Think variants.

Figure 8: Per-benchmark comparison for the Think variant.

Model	1024	2048	4096
Qwen3-4B	0.966	0.984	0.979
Qwen3-1.7B	0.977	0.930	0.961
Qwen3-0.6B	0.924	0.861	0.885
Tucano2-qwen-3.7B-Think	0.817	0.765	0.707
Tucano2-qwen-3.7B-Instruct	0.795	0.710	0.686
Tucano2-qwen-1.5B-Think	0.603	0.501	0.440

Table 7: RULER-PT aggregate scores across context lengths.

Phase	Energy (kWh)	Carbon (kg CO 2 e)
GigaVerbo-v2 Synth (data generation)	14,400	5,472
Continual Pretraining	2,326	884
GigaVerbo-v2 Ablations	1,600	608
Evaluations (all models)	1,000	380
Post-Training (SFT + PO)	530	201
Pretraining	873	332

Table 8: Estimated energy consumption and carbon footprint per project phase.

Family	Params	Base	Instruct	Think
Tucano2-0.6	670M	408B	–	–
Tucano2-qwen-0.5	490M	50B	4.5B	250M
Tucano2-qwen-1.5	1.5B	100B	4.5B	250M
Tucano2-qwen-3.7	3.7B	50B	4.5B	250M

Table 9: Model configurations and training budgets (tokens) used for resource footprint estimation.

Figure 9: Aggregated material footprint tied to the training of our models. Estimations were made using the parameter count of each model, the amount of tokens each run took, an average MFU of 60%, and an assumed GPU lifespan of 3 years.

Appendix figures & tables79 assets

Supplementary material from the paper’s appendix.

Appendix

Training Configuration	Value
Epochs	20
Batch size	256
Maximum sequence length	512 tokens
Optimizer	AdamW ( Loshchilov and Hutter, 2019 )
Beta parameters	$\beta_{1}=0.9$ , $\beta_{2}=0.999$
Epsilon	$10^{-8}$

Appendix

Table B.1: Training configurations for Educational and Toxicity Annotators.

	Task	Precision	Recall	F1 Macro	Accuracy
BERTimbau-large-cased	Educational	0.63	0.54	0.57	0.72
BERTimbau-base-cased		0.57	0.52	0.54	0.71
BERTabaporu-large-uncased		0.56	0.51	0.52	0.70
RoBERTaCrawlPT-base		0.58	0.50	0.52	0.67
RoBERTaLexPT-base		0.56	0.50	0.52	0.70
BERTabaporu-base-uncased		0.55	0.50	0.51	0.69

Appendix

Table B.2: Performance metrics for Educational and Toxicity Annotators.

Source Type	Dataset / Crawl	License(s)
Common Crawl	CC-MAIN-2025-30	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2025-38	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2025-33	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2025-26	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2024-51	ODC-By v1.0, CommonCrawl ToU
	CC-MAIN-2023-50	ODC-By v1.0, CommonCrawl ToU

Appendix

Table C.3: Data sources organized by source type.

Subset	Documents	Size	Tokens
default	372,108,576	834 GB	317,688,116,144
excluded	2,892,095	7.8 GB	2,987,598,133
Total	375,000,671	841.8 GB	320,675,714,277

Appendix

Table C.4: Statistics for individual subsets.

Panel of Figure 23 — Figure C.1: Visualizations for the default subset. Left: Token distribution per data source. Right: Quality and toxicity score distributions per data source.

Panel of Figure 24 — Figure C.2: Visualizations for the excluded subset. Left: Token distribution per data source. Right: Quality and toxicity score distributions per data source.

Dataset	Generation Task/Type
FineWeb-Edu	Educational content generation/rephrasing
Cosmopedia-v2	Educational tutorials and explanatory articles
BDTD	Academic article summarization
BlogSet-BR	Blog-style content and informal writing
FineMath	Mathematical tutorials with CoT reasoning
CodeParrot Clean	Programming tutorials

Appendix

Table D.5: Seed datasets used for synthetic data generation.

Generator	Examples	Total Tokens
Qwen2.5-32B-Instruct	7,719,118	6,484,418,687
Qwen2.5-14B-Instruct	1,806,691	1,625,788,204
Qwen2.5-7B-Instruct	1,711,737	1,211,544,076
Total	11,237,546	9,321,750,967

Appendix

Table D.6: Statistics by generator.

Seed Dataset	Examples	Total Tokens
Cosmopedia-v2	1,896,692	1,669,702,784
FineMath	1,308,713	1,361,573,271
Wikipedia	1,205,667	996,465,458
LegalPT	2,012,941	990,403,657
FineWeb-Edu	1,074,616	978,048,012
CodeParrot	512,331	785,518,858

Appendix

Table D.7: Statistics by seed dataset.

Panel of Figure 28 — Figure D.3: Token distribution. (a) Distribution across the three Qwen2.5 generator models. (b) Distribution across the 15 seed datasets.

Category	Parameter	Value
Model Architecture	Vocabulary Size	49,152
	Hidden Layers	28
	Attention Heads	16
	Key-Value Heads	8
	Head Dimension	128
	Hidden Size	2,048

Appendix

Table E.8: Model Architecture and Training Configuration.

Panel of Figure 30 — Figure E.4: Per-benchmark performance comparisons across the four data mixtures and the Tucano-2b4 baseline. The Edu and Edu+Synth mixtures consistently outperform the Non-Edu mixture across most tasks, with particularly strong gains on ARC Challenge and HellaSwag.

Model	Param.	Tokens	NPM	ARC	Calame	Global PIQA	HellaSwag	Lambada
GigaVerbo-v2 (Edu)	1.5B	46B	39.31	32.80	57.90	82.00	44.90	37.70
Curió-1.1b	1.1B	1.15T	39.16	30.40	59.20	75.00	49.50	46.70
GigaVerbo-v2 (Edu+Synth)	1.5B	46B	37.49	34.40	57.90	75.00	46.00	39.00
Curió-edu-1b1	1.1B	1.02T	34.77	32.20	54.90	69.00	46.30	42.90
GigaVerbo-v2 (Synth)	1.5B	46B	33.86	32.60	56.10	72.00	43.90	33.90
Tucano-2b4	2.4B	500B	33.55	30.40	50.30	73.00	48.80	32.40

Appendix

Table E.9: Comprehensive comparison of our ablation models with other Portuguese and multilingual language models.

Model	Vocab Size	SF	PCW	Chars/Token
Tucano2-0.6B-Base	49,152	1.51	0.56	2.88
GlorIA-1.3B	50,258	1.58	0.47	3.04
Llama-3.2-1B	128,256	1.91	0.61	2.72
SmolLM3-3B-Base	128,256	1.91	0.61	2.72
OLMo-2-0425-1B	100,278	1.92	0.61	2.71
Qwen3-0.6B	151,669	1.93	0.61	2.68

Appendix

Table F.10: Evaluation results on Portuguese text (600,000 words).

Model	Vocab Size	SF	PCW	Chars/Token
Tucano2-0.6B-Base	49,152	1.48	0.48	2.94
Llama-3.2-1B	128,256	1.59	0.45	3.09
SmolLM3-3B-Base	128,256	1.59	0.45	3.09
OLMo-2-0425-1B	100,278	1.59	0.45	3.08
Qwen3-0.6B	151,669	1.60	0.46	3.05
Curió-1.1b	32,000	1.84	0.40	3.57

Appendix

Table F.11: Evaluation results on mixed data (Portuguese, English, and code; 600,000 words).

Tokenizer	SF	Vocab Size	Est. Compute (FLOPs)
Tucano2-0.6B-Base	1.51	49,152	7.26 $\times$ 10 21
GlorIA-1.3B	1.58	50,258	7.62 $\times$ 10 21
Llama-3.2-1B	1.91	128,256	9.89 $\times$ 10 21
SmolLM3-3B-Base	1.91	128,256	9.89 $\times$ 10 21
OLMo-2-0425-1B	1.92	100,278	9.67 $\times$ 10 21
Qwen3-0.6B	1.93	151,669	1.02 $\times$ 10 22

Appendix

Table F.12: Estimated compute cost for training a

\sim

1B-parameter model on 500 billion tokens using different tokenizers.

Benchmark	n-shot	Type	Baseline (%)
ENEM	3-shot	MC-Q&A	20.0
BLUEX	3-shot	MC-Q&A	22.5
OAB Exams	3-shot	MC-Q&A	25.0
ASSIN2 RTE	15-shot	Entailment	50.0
ASSIN2 STS	15-shot	Similarity	0.0
FAQUAD NLI	15-shot	Entailment	45.6

Appendix

Table G.13: Configuration of benchmarks in Garcia et al.’s evaluation suite. All benchmarks are implemented as generative (free-form generation) tasks.

Benchmark	Baseline	Threshold	First Step	Tokens (B)	Score	Improvement
ASSIN2 RTE	50.00	55.00	80,000	188	71.08	21.10
BLUEX	22.50	27.50	320,000	755	27.68	5.20
ENEM	20.00	25.00	320,000	755	25.05	5.10
FAQUAD NLI	45.60	50.60	40,000	94	53.74	8.10
HateBR	50.00	55.00	40,000	94	58.75	8.80
OAB	25.00	30.00	440,000	1,038	30.66	5.70

Appendix

Table G.14: Training tokens required for SmolLM3-3B to surpass baselines on Garcia et al.’s evaluation suite.

Benchmark	Mean Abs Change	SNR	Spearman
ASSIN2 RTE	0.127	4.953	0.606
ASSIN2 STS	0.085	2.492	0.792
BLUEX	0.024	5.993	0.859
ENEM	0.037	4.494	0.878
FAQUAD NLI	0.025	12.155	0.032
HateBR	0.092	6.266	0.290

Appendix

Table G.15: Signal-quality metrics for Garcia et al.’s evaluation suite.

Figure G.5: NPM aggregated score for SmolLM3-3b on Garcia et al.’s evaluation suite. The score shows an upward trend but with substantial volatility across checkpoints.

Panel of Figure 39 — Figure G.6: Individual benchmark performance on Garcia et al.’s evaluation suite.

Panel of Figure 40 — Figure G.7: Signal-quality metrics for Garcia et al.’s evaluation suite.

Benchmark	n-shot	Type	Baseline (%)
CALAME	5-shot	Completion (CF)	0
Global PIQA	5-shot	Completion (CF)	50
ASSIN2 Paraphrase	5-shot	Paraphrase Detection (MCF)	50
ASSIN2 Entailment	5-shot	Entailment (MCF)	50
BELEBELE	5-shot	MC-Q&A (MCF)	25
LAMBADA	5-shot	Completion (CF)	0

Appendix

Table G.16: Configuration of benchmarks in our new Portuguese evaluation suite. Benchmarks employ log-likelihood evaluation using either the Multiple-Choice Format (MCF) or the Cloze Formulation (CF).

Benchmark	Baseline	Threshold	First Step	Tokens (B)	Score	Improvement
ARC Challenge	25.00	30.00	80,000	188	30.77	5.80
ASSIN2 ENT	50.00	55.00	40,000	94	58.08	8.10
ASSIN2 PAR	50.00	55.00	40,000	94	63.65	13.60
BELEBELE	25.00	30.00	280,000	660	30.00	5.00
Global PIQA	50.00	55.00	40,000	94	65.00	15.00
HellaSwag	25.00	30.00	40,000	94	36.50	11.50

Appendix

Table G.17: Training tokens required for SmolLM3-3B to surpass baselines on our new evaluation suite. Most benchmarks require 94B–660B tokens, substantially less than the 755B–1T+ required by Garcia et al.’s harness.

Benchmark	Mean Abs Change	SNR	Spearman
ARC Challenge	0.012	11.777	0.931
ASSIN2 ENT	0.035	18.043	0.180
ASSIN2 PAR	0.025	28.970	0.273
BELEBELE	0.025	4.775	0.949
CALAME	0.009	41.838	0.621
Global PIQA	0.026	23.081	0.578

Appendix

Table G.18: Signal-quality metrics for our new Portuguese evaluation suite. Most benchmarks exhibit strong Spearman correlations (

>

0.5) and acceptable SNR, with CF tasks showing particularly high SNR (

>

19).

Figure G.8: NPM aggregated score for SmolLM3-3b on our new Portuguese evaluation suite. The score shows a clearer, monotonic upward trend with lower volatility than Garcia et al.’s evaluation suite.

Panel of Figure 45 — Figure G.9: Individual benchmark performance on our new Portuguese evaluation suite.

Panel of Figure 46 — Figure G.10: Signal-quality metrics for our new Portuguese evaluation suite.

Benchmark	n-shot	Type	Baseline (%)	Metric
Easy Set
CALAME	5-shot	Completion	0	acc
GlobalPIQA	5-shot	Completion	50	acc_norm
LAMBADA	5-shot	Completion	0	acc
ARC-Challenge	5-shot	MC-Q&A	25	acc_norm
HellaSwag	5-shot	MC-Q&A	25	acc_norm

Appendix

Table G.19: Configuration of our unified Portuguese evaluation suite. The Easy Set provides reliable signals at

<

200B tokens, while the Hard Set requires

>

660B tokens but offers complementary advanced evaluation.

Benchmark	Tokens to Surpass (B)	SNR	Spearman
Easy Set
ARC Challenge	188	11.78	0.931
CALAME	$<$ 40	41.84	0.621
Global PIQA	94	23.08	0.578
HellaSwag	94	26.17	0.937
LAMBADA	$<$ 40	19.58	0.576

Appendix

Table G.20: Comparison of signal-quality characteristics between Easy Set and Hard Set evaluations.

Panel of Figure 49 — Figure G.11: NPM aggregated scores for the Easy Set and Hard Set. The Easy Set provides clear signals in early training stages ( $<$ 200B tokens), whereas the Hard Set requires more training data but shows substantial improvements at later stages ( $>$ 660B tokens).

Metric	Description
prompt_level_strict_acc	Strict prompt accuracy: True only if all instructions are followed exactly with no formatting flexibility.
inst_level_strict_acc	Strict instruction accuracy: Checks each instruction separately with exact matching.
prompt_level_loose_acc	Loose prompt accuracy: Like strict prompt accuracy, but allows minor formatting variations.
inst_level_loose_acc	Loose instruction accuracy: Per-instruction check using loose (format-tolerant) matching.

Appendix

Table G.21: IFEval-PT evaluation metrics. Each metric is computed at either the prompt level (all instructions must be followed) or instruction level (per-instruction check).

Task	Category	Description
niah_pt_single_1	Retrieval (NIAH)	Find a hidden key/value pair embedded in random tokens.
niah_pt_single_2	Retrieval (NIAH)	Recover a key/value pair embedded in natural Portuguese text.
niah_pt_single_3	Retrieval (NIAH)	Complex retrieval: values are long/random; exact match required.
niah_pt_multikey_1	Retrieval (NIAH)	Retrieve the correct value for a queried key (simple).
niah_pt_multikey_2	Retrieval (NIAH)	Match key to value in structured KV pairs.
niah_pt_multikey_3	Retrieval (NIAH)	Retrieve correct values amid noise and distractors (advanced).

Appendix

Table G.22: RULER-PT task descriptions. Tasks span retrieval, multi-hop reasoning, and aggregation across varying context lengths.

Model Configuration
Parameters	7 billion
Architecture	Llama (32 layers, 4096 hidden dim, 32 attn heads)
Sequence Length	4096 tokens
Vocabulary Size	49,152 tokens
Activation Function	SwiGLU
Position Embedding	RoPE (Rotary Position Embedding)

Appendix

Table H.23: Experimental setup for scaling experiments on a 7B-parameter reference model.

Nodes	GPUs	Time/Step (s)	Tokens/s	TFLOPS/GPU	Speedup	Efficiency (%)
4	16	75.55	55,513	165.3	1.00 $\times$	100.0
8	32	38.24	109,731	163.5	1.98 $\times$	98.8
16	64	19.72	212,676	158.4	3.83 $\times$	95.8
32	128	10.11	414,968	154.5	7.47 $\times$	93.4
64	256	5.41	774,753	144.3	13.96 $\times$	87.3

Appendix

Table H.24: Scaling results for 7B-parameter model. Global batch size fixed at 1024 samples (

\sim

4M tokens).

Panel of Figure 54 — Figure H.12: Scaling analysis for 7B-parameter model. Left: Throughput (tokens/second) and achieved FLOPS per GPU as a function of node count. Right: Speedup relative to 4-node baseline, compared against ideal linear scaling. Both plots demonstrate near-linear scalability up to 64 nodes.

Component	Version/Configuration
Software Versions
PyTorch	2.8.0
CUDA	12.6.0
NCCL	2.18.3
Transformers	4.53.2
Flash-Attention	2.8.2

Appendix

Table H.25: Training system software and configuration details.

Parameter	Value
Total Parameters	670,127,616
Architecture	Llama
Hidden Activation	SwiGLU
Normalization	RMSNorm
Positional Embeddings	RoPE
Hidden Layer Size ( $d_{\text{model}}$ )	1,536

Appendix

Table I.26: Architectural specifications.

Metric	AdamW	AdamW+Muon
Variability ( $\sigma/\mu$ )	1.7906	1.6139
Volatility ( $\sigma(\Delta g_{t})$ )	0.1909	0.0769

Appendix

Table I.27: Gradient stability metrics for AdamW vs. AdamW+Muon over 12,000 steps (

\sim

25B tokens).

Panel of Figure 58 — Figure I.13: Gradient norm trajectories for AdamW (left) and AdamW+Muon (right) over 12,000 steps.

Training Configuration	Value
Parallelism Strategy	DDP
Activation Checkpointing	False
Total Batch Size	2,097,152 tokens ( $2^{21}$ )
Micro Batch Size (per GPU)	16 samples
Gradient Accumulation Steps	4
Maximum Context Length	4,096 tokens

Appendix

Table I.28: Hyperparameter settings for pretraining.

Stage	Portuguese (%)	English (%)	Steps	Total Tokens
Warmup+Stable	61% ( $\sim$ 180B)	39% ( $\sim$ 112B)	100,000	$\sim$ 292B
Stable	54% ( $\sim$ 76B)	46% ( $\sim$ 64B)	60,000	$\sim$ 140B
1-sqrt-Decay	75% ( $\sim$ 58B)	25% ( $\sim$ 20B)	35,000	$\sim$ 74B
Total	63% ( $\sim$ 314B )	37% ( $\sim$ 196B )	195,000	$\sim$ 408B

Appendix

Table I.29: Language proportions and token count per training stage.

Dataset	Subset	Size (Tokens)	Repetition
GigaVerbo-v2	Education Score $\geq$ 3	90B	2 $\times$
FineWeb-Edu	Education Score $\geq$ 3	88B	1 $\times$
FineMath	Education Score $\geq$ 3	24B	1 $\times$
Portuguese Subtotal		180B (61%)
English Subtotal		112B (39%)
Stage 1 Total		292B

Appendix

Table I.30: Stage 1 (Warmup+Stable) data mixture: 100,000 steps,

\sim

292B tokens.

Dataset	Subset	Size (Tokens)	Repetition
GigaVerbo-v2	Education Score $\geq$ 4	28B	2 $\times$
GigaVerbo-v2 Synth	All	10B	2 $\times$
FineWeb-Edu	Education Score $\geq$ 4	14B	1 $\times$
Cosmopedia v2	All	30B	1 $\times$
FineMath	Education Score $\geq$ 4	8B	1 $\times$
Big Reasoning Traces	All	2B	1 $\times$

Appendix

Table I.31: Stage 2 (Stable) data mixture: 60,000 steps,

\sim

140B tokens.

Dataset	Subset	Size (Tokens)	Repetition
GigaVerbo-v2	Education Score $\geq$ 4	28B	1 $\times$
GigaVerbo-v2 Synth	All	10B	3 $\times$
FineMath	Education Score $\geq$ 4	8B	1 $\times$
Big Reasoning Traces	All	2B	1 $\times$
Math Meta Reasoning	All	1B	1 $\times$
OpenScience	All	9B	1 $\times$

Appendix

Table I.32: Stage 3 (1-sqrt Decay) data mixture: 35,000 steps,

\sim

74B tokens.

Figure I.14: Gradient norms throughout the complete pretraining run (195,000 steps).

Panel of Figure 65 — Figure I.15: Easy Set evaluations across training checkpoints.

Panel of Figure 66 — Figure I.16: Hard Set evaluations across training checkpoints.

	ARC-C	CALAME	Global-PIQA	HellaSwag	Lambada
Qwen3-0.6B	36.92	45.95	54	40.33	41.78
Tucano2-0.6B	37.01	57.61	79	47.74	39.45
Qwen2.5-0.5B	28.46	44.89	44	37.7	39.08
Curió-1.1B	30.43	59.25	75	49.45	46.69
Tucano-2b4	30.43	50.34	73	48.85	32.39
Curió-Edu-1B	32.22	54.91	69	46.3	42.93

Appendix

Table I.33: Easy Set evaluations.

	BLUEX	ENEM	OAB	Belebele	MMLU
Qwen3-0.6B	42.98	49.48	40.46	65	43.54
Tucano2-0.6B	21.14	23.58	23.28	26.22	27.18
Qwen2.5-0.5B	32.55	38.91	35.9	49.56	41.17
Curió-1.1B	21.56	21.06	23.1	22.89	26.35
Tucano-2b4	25.45	21.62	26.74	25.89	26.24
Curió-Edu-1B	23.5	19.87	25.01	26.22	25.43

Appendix

Table I.34: Hard Set evaluations.

	ARC	CAL	PIQA	HS	LAM	BLX	ENEM	OAB	BEL	MMLU
Qwen3-0.6B-Base	36.92	45.95	54	40.32	41.78	42.97	49.47	40.45	65.00	43.53
+ transplantation	28.11	20.52	54	34.80	25.16	39.77	42.12	37.08	59.11	40.03
Qwen3-1.7B-Base	47.17	53.56	67	52.55	50.80	57.16	65.22	45.78	77.88	55.48
+ transplantation	36.83	26.39	63	41.43	26.12	42.00	58.78	44.28	74.22	50.75
Qwen3-4B-Base	54.52	57.94	77	63.19	60.37	69.95	77.60	55.58	87.88	68.59
+ transplantation	42.64	23.84	61	46.92	33.08	51.87	35.96	48.29	81.66	58.93

Appendix

Table J.35: Per-benchmark performance before and after OMP-based tokenizer transplantation. Easy Set: ARC Challenge (ARC), CALAME (CAL), Global PIQA (PIQA), HellaSwag (HS), LAMBADA (LAM). Hard Set: BLUEX (BLX), ENEM, OAB, BELEBELE (BEL), MMLU.

Training Configuration	0.5B	1.5B	3.7B
Parallelism Strategy	DDP	DDP	FSDP2 (ZeRO-2)
Activation Checkpointing	False	False	True
Total Batch Size	1,048,576 tokens	1,048,576 tokens	1,048,576 tokens
Micro Batch Size	8 samples	4 samples	16 samples
Accumulation Steps	4	8	2
Context Length	4,096 tokens	4,096 tokens	4,096 tokens

Appendix

Table J.36: Hyperparameter settings for continual pretraining.

Dataset	0.5B	1.5B	3.7B
GigaVerbo-v2 (subset 3)	—	12B ( $\times$ 1)	—
GigaVerbo-v2 (subset 4)	28B ( $\times$ 1)	28B ( $\times$ 2)	28B ( $\times$ 1)
GigaVerbo-v2 (subset 5)	0.1B ( $\times$ 1)	0.1B ( $\times$ 2)	0.1B ( $\times$ 2)
Total Web Text	28.1B	68.2B	28.2B
GigaVerbo-v2 Synth	10B ( $\times$ 2)	10B ( $\times$ 3)	10B ( $\times$ 2)
Total Synthetic	20B	30B	20B

Appendix

Table J.37: Data mixture composition for continual pretraining. Subset numbers denote education score thresholds from GigaVerbo-v2 .

Panel of Figure 72 — Figure J.17: Per-benchmark comparison: Qwen3-0.6B-Base vs. Tucano2-qwen-0.5B-Base .

	ARC-C	CALAME	G. PIQA	HellaSwag	LAMBADA
Tucano2-qwen-3.7B-Base	57.78	61.08	83	65.32	62.53
Qwen2.5-7B	54.19	58.96	78	67.92	59.52
Qwen3-4B-Base	54.53	57.95	77	63.19	60.37
SmolLM3-3B-Base	51.37	59.15	81	65.57	59.89
Qwen2.5-3B	45.21	58.38	75	59.44	57.17
Tucano2-qwen-1.5B-Base	48.21	59.06	77	56.25	54.20

Appendix

Table J.38: Easy Set evaluations.

	BLUEX	ENEM	OAB	BELEBELE	MMLU
Tucano2-qwen-3.7B-Base	66.20	77.54	58.45	83.67	65.40
Qwen2.5-7B	65.92	75.02	55.03	89.67	68.55
Qwen3-4B-Base	69.96	77.61	55.58	87.89	68.59
SmolLM3-3B-Base	54.52	61.37	45.51	77.67	56.19
Qwen2.5-3B	58.28	67.32	50.34	83.22	59.79
Tucano2-qwen-1.5B-Base	55.91	68.72	48.29	74.00	54.04

Appendix

Table J.39: Hard Set evaluations.

	Params (B)	PT Tokens (B)	CPT Tokens (B)	Total Tokens (B)	PT FLOPs	CPT FLOPs	Total FLOPs	NPM
Tucano2-qwen-3.7B-Base	3.7	36,000	50	36,050	8.64e+23	1.11e+21	8.65e+23	59.21
Qwen2.5-7B	7.0	18,000	—	18,000	7.56e+23	—	7.56e+23	57.97
Qwen3-4B-Base	4.0	36,000	—	36,000	8.64e+23	—	8.64e+23	57.86
SmolLM3-3B-Base	3.0	11,200	—	11,200	2.02e+23	—	2.02e+23	50.25
Qwen2.5-3B	3.0	18,000	—	18,000	3.24e+23	—	3.24e+23	50.15
Tucano2-qwen-1.5B-Base	1.5	36,000	100	36,100	3.67e+23	9.0e+20	3.68e+23	47.89

Appendix

Table J.40: Performance vs. compute breakdown for all evaluated models. PT = Pretraining, CPT = Continual Pretraining.

Training Configuration	Classification Model	Generation Model
Base model	Qwen3-4B	Qwen3-4B
Epochs	2	3
Batch size	64	64
Maximum context length	6,032 tokens	6,032 tokens
Optimizer	AdamW (cosine schedule)	AdamW (cosine schedule)
Warmup steps	100	100

Appendix

Table K.41: Training configurations for Instruct-Completion Quality Annotators.

	Task	F1 Macro	F1 ( $\geq$ 3 threshold)
Classification model	Quality scoring	0.80	0.98
Conditional generation model	Quality explanation	0.72	0.88

Appendix

Table K.42: Performance metrics for Instruct-Completion Quality Annotators.

Task Type	Description
Code Generation	Instruction-guided code writing and synthesis
Function Calling	Tool calling and structured command invocation
General Instruction	Diverse tasks across multiple domains
Mathematical Reasoning	Problem-solving with step-by-step solutions
Reasoning	Explicit reasoning traces ( <think> $\ldots$ </think> )
Retrieval-Augmented Gen.	Context-aware question answering

Appendix

Table L.43: Task taxonomy.

Panel of Figure 79 — Figure L.19: SFT dataset statistics after filtering.

Subset	Examples	Chosen Tok.	Rejected Tok.	Total Tok.
Harmless (reasoning)	9,641	6,105,673	4,571,353	10,677,026
Harmless (no reasoning)	10,521	4,711,020	4,057,440	8,768,460
Harmful (reasoning)	4,008	1,462,284	3,242,908	4,705,192
Harmful (no reasoning)	4,267	809,240	2,795,035	3,604,275
Total	28,437	14,278,217	13,666,736	27,945,053

Appendix

Table L.44: Preference dataset composition.

Panel of Figure 81 — Figure L.20: Preference dataset statistics.

Training Configuration	0.5B	1.5B	3.7B
Parallelism Strategy	DDP	DDP	FSDP2 (ZeRO-2)
Activation Checkpointing	False	True	True
Total Batch Size	524,288 tokens	524,288 tokens	524,288 tokens
Micro Batch Size	4 / 4 samples	4 / 4 samples	4 / 1 samples
Accumulation Steps	4 / 4	4 / 4	4 / 32
Context Length	4,096 tokens	4,096 tokens	4,096 tokens

Appendix

Table L.45: Hyperparameter settings for Instruct variants (SFT / APO).

Training Configuration	0.5B	1.5B	3.7B
Parallelism Strategy	DDP	DDP	FSDP2 (ZeRO-2)
Activation Checkpointing	False	True	True
Total Batch Size	524,288 tokens	524,288 tokens	524,288 tokens
Micro Batch Size	4 / 4 samples	4 / 4 samples	4 / 1 samples
Accumulation Steps	4 / 4	4 / 4	4 / 32
Context Length	4,096 tokens	4,096 tokens	4,096 tokens

Appendix

Table L.46: Hyperparameter settings for Think variants (SFT / APO).

Task Category	Tokens (approx.)	%
General Instruction Following	700M	80.1
Structured Output Generation	35M	4.0
Math and CoT	27M	3.1
Function Calling	17.5M	2.0
Preference Data (Chosen)	14M	1.6
Translation	5.7M	0.7

Appendix

Table L.47: Instruct SFT data mixture.

Subset	Samples	%
Harmless (no reasoning)	10,521	37.0
Harmless (reasoning stripped)	9,641	33.9
Harmful (no reasoning)	4,267	15.0
Harmful (reasoning stripped)	4,008	14.1
Total	28,437

Appendix

Table L.48: Instruct APO data mixture.

Task Category	Tokens (approx.)	%
Reasoning	34M	100

Appendix

Table L.49: Think SFT data mixture.

Subset	Samples	%
Harmless (with reasoning)	9,641	70.6
Harmful (with reasoning)	4,008	29.4
Total	13,649

Appendix

Table L.50: Think APO data mixture.

	BLUEX	ENEM	OAB	ARC	BELEBELE	MMLU	IFEval	GSM8K	HumanEval
Tucano2-qwen-3.7B-Instruct	64.53	72.92	54.31	60.34	85.22	64.64	41.67	53.81	47.56
Jurema-7B	63.42	70.96	64.97	52.56	88.44	49.91	47.00	30.29	75.61
Qwen2.5-3B-Inst.	56.88	68.65	46.79	41.71	84.00	58.22	63.33	51.90	70.73
Qwen3-4B	63.28	72.15	50.30	43.08	83.67	26.93	79.33	39.88	86.59
Gemma-3-Gaia-4b-it	50.90	64.52	43.46	54.70	78.89	51.49	70.33	51.29	64.02
SmolLM3-3B	53.55	64.73	41.00	52.74	78.67	53.23	69.67	44.44	68.29

Appendix

Table L.51: Individual benchmark scores for Instruct variants and baselines.

Figure L.21: Per-benchmark comparison for 0.5B-Instruct.

Figure L.22: Per-benchmark comparison for 1.5B-Instruct.

	BLUEX	ENEM	OAB	ARC	BELEBELE	MMLU	IFEval	GSM8K
Tucano2-qwen-3.7B-Think	63.00	71.52	53.76	55.38	83.56	61.18	31.67	56.70
SmolLM3-3B	48.82	60.60	42.19	52.56	77.78	52.82	64.67	72.15
Qwen3-4B	78.30	85.23	47.97	39.66	23.00	31.53	84.00	77.02
Qwen3-1.7B	68.29	72.50	39.32	35.64	22.89	22.85	68.33	71.69
Tucano2-qwen-1.5B-Think	39.22	39.89	34.26	42.82	67.67	43.30	33.67	22.83
Qwen3-0.6B	43.53	48.71	34.21	29.91	22.89	22.85	58.33	58.37

Appendix

Table L.52: Individual benchmark scores for Think variants and baselines.

Figure L.23: Per-benchmark comparison for 0.5B-Think.

Figure L.24: Per-benchmark comparison for 1.5B-Think.

Model	Instruct	Think	$\Delta$
Qwen3-4B	86.59	50.61	$-$ 35%
SmolLM3-3B	68.29	33.54	$-$ 34%
Qwen3-1.7B	64.02	38.41	$-$ 25%
Qwen3-0.6B	39.02	17.68	$-$ 21%
Average			$-$ 29%

Appendix

Table L.53: HumanEval scores: Instruct vs. Think mode.

Metric	Qwen3-4B	3.7B-Inst.	3.7B-Think
RULER Score (aggregate)	0.966	0.795	0.817
NIAH multi-key (line retrieval)	0.886	0.814	0.834
NIAH multi-key (KV retrieval)	1.000	0.800	0.744
NIAH multi-key	0.998	0.844	0.870
NIAH multi-query	0.886	0.768	0.797
NIAH multi-value	1.000	0.769	0.778

Appendix

Table L.54: RULER-PT detailed results at 1,024 tokens.

Metric	Qwen3-4B	3.7B-Inst.	3.7B-Think
RULER Score (aggregate)	0.984	0.710	0.765
NIAH multi-key (line retrieval)	1.000	0.720	0.820
NIAH multi-key (KV retrieval)	0.998	0.696	0.590
NIAH multi-key	0.998	0.860	0.868
NIAH multi-query	1.000	0.712	0.741
NIAH multi-value	1.000	0.595	0.759

Appendix

Table L.55: RULER-PT detailed results at 2,048 tokens.

Metric	Qwen3-4B	3.7B-Inst.	3.7B-Think
RULER Score (aggregate)	0.979	0.686	0.707
NIAH multi-key (line retrieval)	1.000	0.684	0.770
NIAH multi-key (KV retrieval)	0.996	0.610	0.472
NIAH multi-key	0.996	0.782	0.782
NIAH multi-query	0.999	0.677	0.695
NIAH multi-value	0.999	0.522	0.653

Appendix

Table L.56: RULER-PT detailed results at 4,096 tokens.

Portuguese Language Models

See all 18 papers

Mar 27, 2026

AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese

Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.

Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos +19

NOVA School of Science and Technology · NOVA LINCS · Instituto de Telecomunicações +2

Portuguese Language Models LLM Training Low-Resource Language Processing LLM Evaluation Multilingual Language Models Language Model Bias Evaluation

Ask PDF

Aug 31, 2026

Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

Brazilian Portuguese remains under-served by open language models, and the few that exist are difficult to reproduce and are often compared without measures of uncertainty. We release Manacá-1B, an open decoder-only model of 1.72 billion parameters trained from scratch for Brazilian Portuguese with a fully containerized, reproducible pipeline. The pretraining is stable, with zero skipped or NaN steps and self-recovering loss spikes, and we release its full log and dynamics. We evaluate the model against nine open baselines on four Portuguese benchmarks under a single harness. Every comparison reports a standard error and a paired significance test, and the harness is validated against previously published numbers. On last-word prediction Manacá-1B is the strongest model below the 7B scale, exceeding both Tucano-1b1 and Tucano-2b4 on LAMBADA-PT with large paired margins; it is competitive on commonsense completion and near chance on multiple-choice reasoning, as are all small base models. Along the way we document a concrete evaluation pitfall: converting a SentencePiece tokenizer with case-folding normalization to the HuggingFace fast format silently drops the normalizer, routing every capitalized token to byte-fallback and depressing scores in a way that is invisible in aggregate metrics. The uncorrected tokenizer lowered LAMBADA-PT accuracy from 45.3 to 25.0; we quantify the effect and provide a one-line fix that reproduces the training tokenizer exactly. Code, raw training and evaluation logs, per-example prediction vectors, the model weights, and the corrected tokenizer are released so that every number in this paper can be recomputed.

Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado Porto

Laboratório Nacional de Computação Científica (LNCC), Petrópolis, Brazil

Language Model Pretraining Portuguese Language Models LLM Evaluation Open-Weight Language Models Language Modeling

Ask PDF

Mar 13, 2026

CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.

João Silva, Luís Gomes, António Branco

University of Lisbon NLX—Grupo de Fala e Linguagem Natural, Departmento de Informática Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal

LLM Evaluation Portuguese Language Models Cross-Cultural Language Model Evaluation LLM Safety Benchmarks LLM Alignment

Ask PDF

Abstract

Portuguese Language Models

Explore similar work

AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese

Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility