The Constitution Speaks Nheengatu: An Open MT System and Reproducible Corpus Pipeline for the Amazonian Língua Geral
Authors: Fabrício Carraro, Rodolfo Joel Zevallos, Rodrigo Gonçalves de Souza, Caio Henrique Faustino da Silva, John E. Ortega
Organizations: Universidad Internacional Menéndez Pelayo · Universitat Pompeu Fabra · Tribunal Regional Federal da 1a Região · PUC-Rio · Northeastern University
Nheengatu (yrl), the Amazonian Língua Geral, has 6--10 thousand speakers and is co-official in São Gabriel da Cachoeira (Brazil), yet remains poorly served by evaluated Portuguese--Nheengatu MT resources. We present the largest parallel corpus assembled for the language (14.5k Portuguese--Nheengatu pairs), anchored by the 2023 official translation of the Brazilian Federal Constitution; the first MT evaluation set built from transcribed spontaneous speech; and, to our knowledge, the first open-weight MT models dedicated to bidirectional Portuguese--Nheengatu translation. Three findings organize the paper. First, translation into Nheengatu on held-out constitutional articles plateaus on development data regardless of training length or model size, and much of the remaining error sits in conventional legal terms that the written sources do not attest consistently; a terminology success rate over a frozen lexicon shows that glossary mechanisms at training time and lexical constraints at decoding each raise term realization, the latter at a cost in repetitive output that chrF++ barely registers. Second, a corpus expansion that adds transcribed conversation raises speech chrF++ by 24--27 points while leaving written domains unchanged, a register gap that written-only benchmarks cannot see. Third, a larger NLLB model helps translation into Portuguese but not into Nheengatu on legal or speech text. For Portuguese-to-Nheengatu translation, our final systems reach 31.7--32.8 chrF++ on article-disjoint constitutional text, 64.1--64.5 on the near-domain set (62.1 after removing items with target-side content overlap), and 45.2--47.5 on speech.
Figures & tables
Figure 1: The Upper Rio Negro. Dark: São Gabriel da Cachoeira, where Nheengatu has been co-official since 2002. Boundaries: IBGE; base map: Natural Earth.
source
pairs
yrl w.
w/s
register
Constitution 2023
2,031
42,262
20.8
legal
Avila 2021 (dict.)
2,745
24,437
8.9
dictionary ex.
UD treebank
2,484
19,270
7.8
mixed
LEETRA (4 books)
526
8,284
15.7
school
Melgueiro 2022
1,870
8,128
4.3
dictionary ex.
Gurgel (compil.)
1,562
7,538
4.8
teaching
Table 1: Corpus composition, ordered by yrl words. “yrl w.” is running words on the Nheengatu side; “w/s” is the mean number of words per aligned segment; speech units are pause-delimited turns rather than sentences. Religious text is 0.2% of the corpus (23 pairs), where for most of the world’s languages scripture is the only large digital resource; our anchor is instead a contemporary legal text translated under editorial review.
legal (const)
near-dom.
pt → yrl
yrl → pt
pt → yrl
yrl → pt
600M (3 seeds)
31.25
35.45
64.12
56.12
± 0.44
± 0.31
± 0.38
± 0.41
1.3B (1 seed)
32.03
37.76
66.37
58.22
Table 2: Phase-1 evaluation (v1 series): chrF++ for the 600M (mean ± sd over three seeds) and the 1.3B (one run). The 1.3B differences are descriptive: a single run has no seed spread of its own, so the 600M’s ± sd does not standardize them.
training mechanism
unhinted
hinted
none (baseline)
51.5
not run
token pairs only
55.1
not run
annotation only
51.0
63.6
annotation + token pairs
54.5
66.2 ± 2.7
Table 3: TSR on legal development data (198 term occurrences): training mechanism crossed with decode-time hints (v2, 600M). We did not evaluate hints on models trained without annotation; their response to unseen markup would also test robustness to the input format. Single-seed values, except the hinted combination (three-seed mean).
eval
v2 model
v3 model
Δ
speech pt → yrl
23.06 †
46.77
+23.7
speech yrl → pt
22.82 †
49.81
+27.0
legal pt → yrl
29.90 ± 0.17
29.98 ± 0.13
+0.1
near-dom. pt → yrl
60.70 ± 1.01
59.99 ± 0.64
−0.7
Table 4: Effect of the v2 → v3 expansion (3,408 pairs, 1,156 of them speech) on development chrF++. † The v2 model saw no speech in training, so the speech rows measure the whole expansion, not the speech subset alone; those two rows are single runs without terminology mechanisms, while the written rows are three-seed means of the shared configuration. The expansion also raises unhinted TSR by 3.0 pp, an effect not attributed to any single source.
legal
near-domain
spontaneous speech
system
pt → yrl
yrl → pt
pt → yrl
yrl → pt
pt → yrl
yrl → pt
flagship
31.70
37.39
64.11
57.10
45.24
49.67
+ hints
31.13
—
64.42
—
—
—
avg
32.78
38.14
64.54 †
58.76
47.45
49.37
avg, v1 refs
32.74
38.15
62.55
56.92
—
—
Table 5: Phase-2 final test, decoded once. For pt → yrl, paired bootstrap intervals do not resolve the difference between our two systems on legal text (95% CI [−0.06,+2.26] ) or near-domain text ( [−0.48,+1.44] ); the speech interval excludes zero ( [+0.46,+4.06] ). The same holds against Phase 1 on identical v1 references ( 32.74 vs. 32.03 , [−0.37,+1.79] , p=0.20 ). The 0.71 gap against Phase 1 on v1 references is not resolved by the paired-bootstrap interval on this 218-segment set. † Removing the identified target-side content-overlap items lowers this to 62.09; we report both, and the Limitations give the overlap measurement.
system
language
test set
→ Indig.
Indig. →
ANLP-25 base
Guarani
XNLI
35.7
35.9
ANLP-25 base
Quechua
XNLI
31.9
37.2
ANLP-25 base
Nahuatl
XNLI
22.4
26.4
ANLP-25 base
Otomí
XNLI
12.8
20.8
ours 1.3B
Nheengatu
legal (article-disjoint)
32.8
38.1
ours 1.3B
Nheengatu
speech
47.5
49.4
Table 6: Context from AmericasNLP, not a ranking: no cell is a like-for-like comparison. All values are chrF++ from fine-tuned NLLB-200-distilled-1.3B models with an extended embedding matrix; our rows are the avg system of Table 5 . The AmericasNLP pairs use Spanish and translated XNLI test sets; ours use Portuguese and single-step translations (field translation on the speech side, see Limitations).
Figure 2: Terminology realization versus chrF++ on hinted legal development data. Free beam-4 gives 30.22 chrF++ at 62.6 TSR; hard constraints with a wide beam and five hints reach 83.3 TSR at 29.37. Two settings are dominated in both dimensions at once: soft reranking (29.41/64.1) and hard constraints at beam-4 (27.43/75.8), where narrow constrained search substantially reduces reference similarity and a wider beam recovers part of the loss, as the dynamic-beam-allocation literature predicts Post and Vilar (2018) . The decoder realizes 164 of 165 requested surface-form insertions, or 99.4%. This measures surface inclusion rather than semantic, morphological, or syntactic correctness. The wide-beam gain is hint-conditional: the argmax drops only when hints are present.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
cell
seq2seq 600M
dec.-only @1k
dec.-only late
Δ late
3⋅ SE
legal pt → yrl
29.57 ± 0.22
25.89 ± 0.67
27.67 ± 0.57
+1.90
1.06
legal yrl → pt
34.83 ± 0.29
28.14 ± 0.68
33.11 ± 1.51
+1.72
2.67
near-dom. pt → yrl
60.87 ± 0.49
52.11 ± 0.82
62.37 ± 0.54
−1.50
1.26
near-dom. yrl → pt
56.68 ± 0.23
48.94 ± 0.51
59.91 ± 0.05
−3.23
0.41
speech pt → yrl
46.23 ± 0.48
39.96 ± 1.11
44.74 ± 0.90
+1.49
1.77
speech yrl → pt
50.95 ± 1.05
47.16 ± 1.88
47.88 ± 1.60
+3.08
3.31
Appendix
Table 7: The architecture control under both checkpoint selections (development sets, chrF++, mean ± sd over three seeds). “@1k” is the loss-selected checkpoint; “late” is steps 11–12k, scored after submission as a proxy for chrF++ selection. Δ is seq2seq minus late decoder-only (positive favours the seq2seq); 3⋅ SE is three standard errors of the difference, the pre-specified threshold for that comparison, and a cell is conclusive when ∣Δ∣≥3⋅ SE and ∣Δ∣≥1.0 : legal pt → yrl (seq2seq) and both near-domain cells (decoder-only).
set
dir.
chrF++
chrF
spBLEU
flagship
legal
pt → yrl
31.70
35.95
12.83
legal
yrl → pt
37.39
39.56
19.90
near-dom.
pt → yrl
64.11
65.99
54.65
near-dom.
yrl → pt
57.10
57.86
41.68
speech
pt → yrl
45.24
49.77
27.86
Appendix
Table 8: Final test under three metrics. Plain chrF is approximately 0.8–4.5 points above chrF++ in these results; chrF++ additionally incorporates word n -grams. spBLEU scores are lower on these sets. The ordering of the two systems within each set and direction is the same under all three metrics; comparisons across directions need not have the same ordering.
ok
minor
suspect
defective
Speech (200)
183
11
2
4
Dictionary ex. (240)
208
16
—
16
Appendix
Table 9: Audit outcomes; the approval rates quoted in § 3 are the ok column.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.