Multilingual translation requires a model to preserve meaning while producing text in a requested language. We present HexaBee-4B, a translator adapted from Gemma 3 4B IT through three stages: Wikipedia continued pretraining, joint multilingual answering and answer-text translation, and bidirectional translation specialization. The intermediate stage connects two tasks through shared multilingual answers: answering a question in the requested language and translating an answer into that language. It comprises 19.46 million answering records, including 5.76 million cross-language question-answer pairs, and 10.70 million answer-text translations. The final stage uses 658,274 forward and reverse synthetic translation examples. On 36,572 synthetic test examples spanning all 30 directions among Arabic, Chinese, English, Hebrew, Persian, and Spanish, HexaBee-4B achieves direction-macro chrF++ of 73.75, compared with 57.11 for Gemma 3 4B IT and 66.83 for Gemma 3 12B IT. BLEU, TER, COMET, and a matched model-judged subset support the same overall ranking. These results demonstrate improved translation across the six-language set, with chrF++ gains in every evaluated direction over both baselines.
Figures & tables
Figure 1: The complete adaptation recipe. Phase 1 applies next-token language modeling to Wikipedia text. Phase 2 continues the same adapter with multilingual answering and answer-text translation. After merging, phase 3 trains a fresh adapter on bidirectional translations and produces the final merged HexaBee-4B model. The base is Gemma 3 4B IT. Phase 1 counts processed training examples; phases 2–3 count SFT records. Examples derived from one source can occur in multiple tasks.
Figure 2: Illustrative English-to-Spanish supervision. The tasks have different inputs and instructions but can share the same target answer. This constructed example explains the method; it is not a quoted training record.
Figure 3: Data and supervision flow in phases 2 and 3. Gemma 3 27B IT expands the Wikipedia-derived inventory into multilingual question and answer variants. Answering and answer-text translation supply complementary phase-2 inputs, with system instructions specifying the requested response. A separate DeepSeek-generated corpus supplies the final bidirectional translation stage. The full checkpoint sequence, including phase 1, is shown in Figure 1 .
Phase-2 supervision
Records
Same-language answering
13,698,526
Cross-language answering
5,759,806
Answer-text translation
10,695,072
Total
30,153,404
Table 1: Retained phase-2 supervision. Same/cross refers to question and answer language. Response-style and direction breakdowns are given in the appendix.
System
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
4B IT
40.70
58.91
57.11
51.40
85.88
80.32
81.43
12B IT
50.54
68.70
66.83
38.89
88.20
91.73
91.97
HexaBee-4B
62.03
75.08
73.75
30.05
89.86
92.92
94.10
Table 2: Full-test direction-macro automatic metrics ( N=36,572 ) and holistic judge means on 300 matched items. COMET is scaled by 100; TER is lower-is-better. Bold marks the best displayed score and underlining the second distinct score. Each system uses its recorded inference prompt.
Figure 4: Translation quality by target language. Each point averages five incoming directions on the full test partition. HexaBee-4B improves both direction-macro chrF++ and COMET for each target language. The focused axis ranges are shown explicitly; marker shapes distinguish systems as well as colors.
Baseline
Judge view
Gain
95% CI
4B IT
Reference
+12.60
[9.96,15.31]
4B IT
No reference
+12.67
[10.13,15.34]
12B IT
Reference
+1.19
[0.34,2.08]
12B IT
No reference
+2.13
[1.29,3.13]
Table 3: Paired holistic-score differences, HexaBee-4B minus baseline, on the matched 300-item subset.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Raw text rows
English
6,406,711
Persian
802,549
Arabic
1,216,238
Chinese
1,354,407
Egyptian Arabic
1,615,156
Total across distinct source files
11,395,061
Appendix
Table 4: Available phase-1 Wikipedia sources before training preprocessing. The registered mixture repeats the Persian source four times.
Subcorpus
Complete QA
Moderate QA
Formal QA
Informal QA
Translation
Total
General QA
3,952,557
208,364
330,194
284,650
2,481,368
7,257,133
Standard QA
7,240,185
300,875
1,540,222
411,033
5,554,592
15,046,907
Title QA
3,835,308
0
646,570
708,374
2,659,112
7,849,364
Appendix
Table 5: All selected phase-2 data by question format and task, with source partitions pooled. A zero denotes an absent task export.
s\t
fa
en
ar
zh
he
es
fa
2,025,720
764,747
202,573
101,287
19,744
1,139
en
1,418,003
2,025,730
101,287
462,076
39,489
0
ar
405,143
202,573
2,025,730
20,258
4,936
0
zh
202,572
101,287
40,514
2,025,730
4,114
822
he
49,362
29,616
9,873
4,936
987,223
0
es
107,249
76,607
15,322
15,322
4,936
1,532,130
Appendix
Table 6: Complete QA records by input language (rows) and response language (columns). Diagonal cells are same-language answering; translation excludes identity directions.
s\t
fa
en
ar
zh
he
es
fa
4,545,437
931,711
202,573
101,287
19,744
1,139
en
2,513,228
2,582,276
101,287
462,076
39,489
0
ar
460,798
202,573
2,025,730
20,258
4,936
0
zh
222,051
101,287
40,514
2,025,730
4,114
822
he
57,710
29,616
9,873
4,936
987,223
0
es
115,597
76,607
15,322
15,322
4,936
1,532,130
Appendix
Table 7: All QA response variants (complete, moderate, formal, informal) records by input language (rows) and response language (columns). Diagonal cells are same-language answering; translation excludes identity directions.
s\t
fa
en
ar
zh
he
es
fa
0
1,418,003
506,429
506,429
246,806
383,029
en
1,418,003
0
506,433
506,433
246,806
383,033
ar
506,429
506,433
0
162,059
78,978
122,571
zh
506,429
506,433
162,059
0
78,978
122,571
he
246,806
246,806
78,978
78,978
0
78,978
es
383,029
383,033
122,571
122,571
78,978
0
Appendix
Table 8: Answer-text MT records by input language (rows) and response language (columns). Diagonal cells are same-language answering; translation excludes identity directions.
Answer-language
Completeness
Prompt language
Rows
Unique
Rows
Unique
fa
1302
1110
1076
836
en
1302
944
1000
759
ar
1060
817
1032
792
zh
976
755
1024
784
he
914
691
1080
840
Appendix
Table 9: Representative prompt pools before record sampling. Counts use the pool whose target-language label matches its prompt language; completeness text does not itself specify the answer language. Unique counts deduplicate exact sentence strings. Repeated rows remain in the sampler. A dash indicates no such pool.
Figure 5: Retained phase-2 records flowing from the three pooled question formats to answering and translation tasks. Ribbon widths encode record counts. Translation and several answering variants are jointly selected; source-row reuse across these tasks means the plotted counts are not independent documents.
Figure 6: Realized question/answer language coverage. Left: complete-answer QA. Right: all answering variants. Color uses a logarithmic count scale. Gold outlines mark the two Spanish-target cells whose complete-answer directives incorrectly request English (1,961 records). Zero cells indicate absent exported combinations.
Figure 7: Training lineage. The translation stage starts a new adapter on the merged cross-lingual model. Phase 2 combines system-directed answering with translations of multilingual answer texts; phase 3 specializes the merged model on independent synthetic translations.
Figure 8: Recorded response-token training loss during final translation adaptation, with a centered rolling median over up to 25 logged values for visual guidance. This is an optimization diagnostic, not a translation-quality or generalization metric.
Setting
Wikipedia adaptation
Cross-lingual SFT
Translation SFT
Training examples
3,399,664
30,153,102
658,267
Internal validation examples
Not established
302
7
Epochs / optimizer steps
1 / 13,280
1 / 6,544
1 / 3,429
CUDA processes
4
6
2
Per-device batch / accumulation
16 / 4
12 / 64
12 / 8
Effective batch
256
4,608
192
Appendix
Table 10: Executed settings from training logs and generated model cards. Counts have stage-specific semantics and should not be added as unique documents. Current nearby YAMLs may describe other runs; logged values take precedence.
Figure 9: All eight judge dimensions, averaged over the same 300 identities. Both views compare the same three systems; they remain judgments from one evaluator and are not independent human assessments.
Figure 10: Full-test chrF++ gains of HexaBee-4B over both public comparison systems across all 30 source–target directions.
Target
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
All
4B IT
36,572
40.70
58.91
57.11
51.40
85.88
80.32
81.43
12B IT
36,572
50.54
68.70
66.83
38.89
88.20
91.73
91.97
HexaBee-4B
36,572
62.03
75.08
73.75
30.05
89.86
92.92
94.10
ar
4B IT
6,120
34.09
62.25
59.33
56.99
86.64
82.56
81.60
12B IT
6,120
43.14
70.43
67.69
44.60
88.52
89.12
89.78
HexaBee-4B
6,120
56.17
77.09
75.10
33.62
90.17
92.72
93.60
Appendix
Table 11: Overall and target-language comparisons. Automatic metrics are direction-macro averages on the full test set; holistic judge means use 300 matched items overall or 50 per target. COMET is scaled by 100; TER is lower-is-better. Bold marks the best displayed score and underlining the second distinct displayed score within each group; ties share rank. The adapted model is tinted for easy comparison. These recorded systems use different inference prompts.
Direction
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
chrF++ 10
TER 10↓
COMET 10
zh→ar
4B IT
1229
23.68
51.49
48.28
80.63
84.07
72.60
71.60
37.49
100.59
80.35
12B IT
1229
32.53
64.44
60.91
61.20
87.91
91.60
92.50
62.38
53.76
87.21
HexaBee-4B
1229
42.46
69.06
66.34
57.30
89.15
90.40
93.00
71.36
38.78
88.68
en→ar
4B IT
1213
37.50
66.77
63.77
51.70
87.99
86.50
85.20
65.44
44.06
88.04
12B IT
1213
42.30
68.56
66.09
47.70
87.47
81.10
82.20
57.25
48.09
85.10
HexaBee-4B
1213
55.60
76.61
74.66
35.72
90.06
93.40
94.10
76.86
27.55
90.48
Appendix
Table 12: All systems translating into Arabic. Columns through COMET use the full direction-specific test set ( N ). The last 5 columns use the ten shared judge-subset items per direction. Bold/underline mark first/second within the same direction and metric; TER is lower-is-better.
Direction
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
chrF++ 10
TER 10↓
COMET 10
ar→zh
4B IT
1201
36.56
32.17
30.93
58.04
83.81
73.60
73.90
22.92
62.38
81.89
12B IT
1201
42.36
39.84
37.13
50.06
86.94
91.80
91.90
30.13
51.75
87.06
HexaBee-4B
1201
57.09
51.82
49.12
35.01
89.69
93.70
94.20
40.72
35.73
89.56
en→zh
4B IT
1276
30.42
27.90
27.56
69.42
83.44
82.30
82.80
29.76
49.49
83.68
12B IT
1276
32.77
31.48
29.94
67.91
85.07
91.80
93.50
30.05
44.73
87.02
HexaBee-4B
1276
50.33
45.82
45.30
45.39
90.02
93.00
93.60
44.30
31.80
89.95
Appendix
Table 13: All systems translating into Chinese. Columns through COMET use the full direction-specific test set ( N ). The last 5 columns use the ten shared judge-subset items per direction. Bold/underline mark first/second within the same direction and metric; TER is lower-is-better.
Direction
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
chrF++ 10
TER 10↓
COMET 10
ar→en
4B IT
1200
59.91
75.60
74.32
29.38
86.98
82.00
83.30
64.16
41.19
85.14
12B IT
1200
70.24
86.13
84.97
21.03
88.74
92.40
94.80
81.37
23.10
88.22
HexaBee-4B
1200
76.36
88.32
87.58
16.02
89.58
90.10
94.60
85.46
16.95
89.33
zh→en
4B IT
1213
38.89
64.46
62.03
59.53
86.20
64.40
65.90
53.20
73.92
81.00
12B IT
1213
47.49
75.48
72.87
40.65
87.93
95.70
94.90
73.06
39.17
88.17
HexaBee-4B
1213
58.71
79.80
77.87
32.67
89.16
95.20
95.60
78.02
30.94
89.30
Appendix
Table 14: All systems translating into English. Columns through COMET use the full direction-specific test set ( N ). The last 5 columns use the ten shared judge-subset items per direction. Bold/underline mark first/second within the same direction and metric; TER is lower-is-better.
Direction
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
chrF++ 10
TER 10↓
COMET 10
ar→he
4B IT
1255
33.86
55.36
53.52
55.20
85.90
70.40
66.30
48.01
52.39
84.15
12B IT
1255
48.39
68.91
67.01
37.34
88.91
91.70
93.20
70.71
30.88
88.60
HexaBee-4B
1255
61.41
76.53
75.50
28.87
91.03
93.00
94.70
78.93
20.92
90.49
zh→he
4B IT
1232
19.81
39.55
38.42
88.40
82.50
85.00
87.00
53.38
61.20
85.05
12B IT
1232
27.88
48.10
47.27
78.13
85.34
90.30
88.60
58.72
53.58
86.56
HexaBee-4B
1232
35.82
51.34
51.36
90.09
86.74
93.70
94.70
63.72
46.14
87.24
Appendix
Table 15: All systems translating into Hebrew. Columns through COMET use the full direction-specific test set ( N ). The last 5 columns use the ten shared judge-subset items per direction. Bold/underline mark first/second within the same direction and metric; TER is lower-is-better.
Direction
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
chrF++ 10
TER 10↓
COMET 10
ar→fa
4B IT
1228
32.96
57.46
55.29
56.93
84.60
73.20
75.10
58.50
48.14
84.77
12B IT
1228
45.58
71.82
69.85
40.84
88.30
89.80
92.60
70.16
34.38
89.77
HexaBee-4B
1228
62.29
80.41
79.05
23.95
90.46
89.50
95.10
80.22
21.59
91.29
zh→fa
4B IT
1208
19.17
41.95
40.29
90.09
81.51
51.60
52.70
32.82
98.86
79.60
12B IT
1208
32.94
63.68
61.27
55.33
87.85
89.40
91.50
63.25
49.32
89.01
HexaBee-4B
1208
45.05
69.11
67.38
55.17
89.04
91.00
93.10
69.82
36.28
90.41
Appendix
Table 16: All systems translating into Persian. Columns through COMET use the full direction-specific test set ( N ). The last 5 columns use the ten shared judge-subset items per direction. Bold/underline mark first/second within the same direction and metric; TER is lower-is-better.
Direction
System
N
BLEU
chrF
chrF++
TER ↓
COMET
Ref. J.
No-ref. J.
chrF++ 10
TER 10↓
COMET 10
ar→es
4B IT
1257
59.24
74.94
73.59
30.24
85.60
88.20
91.10
82.38
20.28
87.14
12B IT
1257
68.19
85.36
84.08
20.85
87.93
94.10
96.30
86.38
14.91
88.39
HexaBee-4B
1257
77.23
88.55
87.73
14.41
89.37
91.80
94.20
88.63
12.20
88.66
zh→es
4B IT
1173
32.40
55.60
53.72
68.49
83.08
81.90
86.20
63.96
53.96
86.22
12B IT
1173
48.63
75.81
73.37
44.50
87.27
93.60
92.90
75.41
35.64
88.58
HexaBee-4B
1173
58.16
80.03
78.11
31.49
89.03
94.10
94.90
81.32
24.58
89.61
Appendix
Table 17: All systems translating into Spanish. Columns through COMET use the full direction-specific test set ( N ). The last 5 columns use the ten shared judge-subset items per direction. Bold/underline mark first/second within the same direction and metric; TER is lower-is-better.
Reference-assisted judge
Reference-free judge
Dir.
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
zh→ar
4B IT
72.0
66.5
73.8
72.1
71.8
73.9
68.8
73.5
74.1
69.3
74.0
72.7
72.7
74.8
71.5
75.8
12B IT
94.8
87.5
94.2
92.8
93.8
94.6
90.5
93.0
95.1
92.0
91.8
92.6
92.8
93.6
91.6
94.0
HexaBee-4B
92.4
89.3
95.2
93.3
93.9
94.8
91.3
96.0
95.0
92.1
94.3
93.8
94.5
95.0
92.9
95.0
en→ar
4B IT
91.0
80.2
90.2
91.0
90.1
89.2
85.5
91.5
88.0
79.7
88.5
88.6
86.6
89.0
85.0
91.5
12B IT
83.3
76.0
94.3
94.5
82.8
84.5
81.3
95.0
94.0
88.5
93.4
92.2
93.2
84.0
91.2
94.5
Appendix
Table 18: Every judge dimension for translations into Arabic, ten shared items per direction. Sem.: semantic accuracy; Term.: terminology; Flu.: fluency; Coh.: cohesion; Ctx.: contextual appropriateness; Lang.: target-language adherence; Fmt.: formatting. Scores are model judgments on 0–100 scales, not validated accuracy percentages. Bold/underline mark first/second within each direction and view; ties at displayed precision share rank.
Reference-assisted judge
Reference-free judge
Dir.
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
ar→zh
4B IT
75.8
72.4
74.9
73.7
74.5
75.1
72.1
74.5
76.7
72.9
75.3
74.4
74.8
75.3
72.4
74.0
12B IT
95.6
88.5
94.1
93.8
94.7
93.6
90.6
91.5
95.1
91.8
94.4
92.4
93.3
92.4
90.8
91.5
HexaBee-4B
95.9
92.1
96.5
95.3
95.6
96.3
92.5
96.4
96.0
94.0
95.4
94.1
94.5
95.7
92.5
95.0
en→zh
4B IT
84.8
80.5
80.7
81.1
83.1
82.5
80.0
92.5
86.0
81.7
90.9
93.5
84.2
82.7
81.2
94.5
12B IT
95.4
90.4
91.0
92.4
93.5
92.3
88.8
90.5
96.0
93.7
93.2
93.4
94.3
93.9
92.2
93.5
Appendix
Table 19: Every judge dimension for translations into Chinese, ten shared items per direction. Sem.: semantic accuracy; Term.: terminology; Flu.: fluency; Coh.: cohesion; Ctx.: contextual appropriateness; Lang.: target-language adherence; Fmt.: formatting. Scores are model judgments on 0–100 scales, not validated accuracy percentages. Bold/underline mark first/second within each direction and view; ties at displayed precision share rank.
Reference-assisted judge
Reference-free judge
Dir.
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
ar→en
4B IT
83.3
75.0
85.0
84.0
83.7
85.1
80.6
86.5
86.2
82.8
84.4
84.3
83.7
85.2
81.3
85.5
12B IT
95.1
89.0
95.5
94.5
94.5
95.5
91.2
93.0
96.3
94.4
95.5
95.3
95.4
95.7
93.7
94.5
HexaBee-4B
91.2
87.4
96.9
96.9
96.3
97.1
94.1
97.0
96.6
94.7
95.2
94.5
95.4
96.2
93.5
95.1
zh→en
4B IT
65.1
61.5
64.1
65.8
64.9
63.8
62.0
75.5
68.0
65.0
75.0
75.4
65.8
66.6
65.3
76.0
12B IT
97.9
95.3
95.9
96.9
96.7
96.1
94.4
95.2
97.1
96.4
94.8
94.8
95.0
96.2
93.1
94.5
Appendix
Table 20: Every judge dimension for translations into English, ten shared items per direction. Sem.: semantic accuracy; Term.: terminology; Flu.: fluency; Coh.: cohesion; Ctx.: contextual appropriateness; Lang.: target-language adherence; Fmt.: formatting. Scores are model judgments on 0–100 scales, not validated accuracy percentages. Bold/underline mark first/second within each direction and view; ties at displayed precision share rank.
Reference-assisted judge
Reference-free judge
Dir.
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
ar→he
4B IT
72.5
64.0
72.0
72.5
72.0
72.0
68.5
83.0
69.4
62.5
69.5
69.9
69.2
70.0
66.6
69.5
12B IT
96.0
86.3
94.5
94.4
95.0
95.0
90.0
94.5
96.3
92.9
95.2
93.9
94.1
94.5
92.3
92.1
HexaBee-4B
96.3
88.5
95.6
96.2
96.2
95.6
93.6
97.5
97.6
94.8
95.8
94.9
96.5
95.9
93.8
96.5
zh→he
4B IT
89.0
80.8
87.7
88.7
88.7
87.4
86.0
88.0
90.4
84.8
87.5
87.9
89.0
89.1
86.5
90.0
12B IT
93.3
86.8
92.9
91.8
92.3
92.9
90.1
91.2
92.7
87.2
89.0
91.2
88.7
89.5
86.9
91.0
Appendix
Table 21: Every judge dimension for translations into Hebrew, ten shared items per direction. Sem.: semantic accuracy; Term.: terminology; Flu.: fluency; Coh.: cohesion; Ctx.: contextual appropriateness; Lang.: target-language adherence; Fmt.: formatting. Scores are model judgments on 0–100 scales, not validated accuracy percentages. Bold/underline mark first/second within each direction and view; ties at displayed precision share rank.
Reference-assisted judge
Reference-free judge
Dir.
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
ar→fa
4B IT
77.0
68.8
73.4
74.8
75.5
75.3
71.0
78.0
77.6
74.1
76.0
76.3
76.2
76.6
74.3
75.5
12B IT
93.2
85.4
94.1
93.0
91.1
94.1
89.3
94.5
95.4
91.0
95.0
93.5
94.1
94.8
91.8
94.0
HexaBee-4B
92.2
85.7
96.4
94.5
94.6
96.4
92.0
97.7
97.0
95.2
96.5
96.4
96.8
96.1
95.3
96.5
zh→fa
4B IT
51.5
45.0
53.0
51.0
51.5
51.5
49.0
64.5
55.0
49.5
55.0
55.0
55.5
54.0
53.0
56.5
12B IT
91.4
84.8
93.3
92.4
92.3
93.0
90.0
93.0
94.3
89.8
92.3
93.1
93.6
93.6
90.8
92.5
Appendix
Table 22: Every judge dimension for translations into Persian, ten shared items per direction. Sem.: semantic accuracy; Term.: terminology; Flu.: fluency; Coh.: cohesion; Ctx.: contextual appropriateness; Lang.: target-language adherence; Fmt.: formatting. Scores are model judgments on 0–100 scales, not validated accuracy percentages. Bold/underline mark first/second within each direction and view; ties at displayed precision share rank.
Reference-assisted judge
Reference-free judge
Dir.
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
ar→es
4B IT
91.8
80.7
94.1
92.2
92.3
93.0
89.1
92.5
94.4
87.8
94.3
94.0
93.4
93.5
92.5
95.8
12B IT
97.4
89.5
98.0
97.1
96.9
97.3
93.9
93.0
98.5
96.5
96.6
96.7
97.5
96.6
95.9
96.0
HexaBee-4B
95.0
85.5
96.8
95.8
96.1
96.1
93.8
96.8
96.2
91.6
97.1
95.7
95.3
97.1
94.3
96.1
zh→es
4B IT
86.1
78.0
85.0
84.7
85.4
84.9
81.7
83.5
87.6
87.3
85.6
86.4
86.7
86.4
84.6
85.5
12B IT
96.4
90.6
95.7
95.1
95.5
95.5
92.0
93.5
95.8
93.3
92.9
93.6
93.2
93.3
91.5
93.6
Appendix
Table 23: Every judge dimension for translations into Spanish, ten shared items per direction. Sem.: semantic accuracy; Term.: terminology; Flu.: fluency; Coh.: cohesion; Ctx.: contextual appropriateness; Lang.: target-language adherence; Fmt.: formatting. Scores are model judgments on 0–100 scales, not validated accuracy percentages. Bold/underline mark first/second within each direction and view; ties at displayed precision share rank.
Reference-assisted judge
Reference-free judge
Target
System
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
Sem.
Term.
Flu.
Coh.
Ctx.
Lang.
Style
Fmt.
ar
4B IT
85.1
77.1
85.5
85.3
84.3
84.9
81.6
87.0
84.4
78.3
83.6
84.0
83.2
84.0
81.5
85.5
12B IT
92.8
84.3
94.5
93.9
91.5
92.0
89.0
92.9
94.3
90.6
92.2
92.3
92.9
90.9
91.2
93.1
HexaBee-4B
95.7
89.8
96.6
95.4
94.8
96.1
92.6
96.4
95.5
93.4
94.4
94.3
94.5
95.0
92.3
94.6
zh
4B IT
78.1
71.2
76.2
76.4
76.8
76.8
73.7
77.8
82.5
77.0
83.3
83.2
79.5
77.8
77.5
83.4
12B IT
92.9
88.0
91.3
91.3
91.8
91.4
88.2
90.5
93.5
90.5
92.0
91.4
92.2
92.7
90.2
91.1
Appendix
Table 24: All eight judge dimensions in both views, aggregated by target language, with 50 shared examples per target. All directions receive equal weight.
Split
System
N
Src.
Cand.
Ref.
Full
4B IT
36,572
7757
8240
7789
Full
12B IT
36,572
7757
8334
7789
Full
HexaBee-4B
36,572
7757
7743
7789
Judge
4B IT
300
51
52
52
Judge
12B IT
300
51
53
52
Judge
HexaBee-4B
300
51
51
52
Appendix
Table 25: Texts exceeding the 512-token COMET encoder limit. Src., Cand., and Ref. denote source, candidate, and reference. Counts can overlap within an example.
Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.
Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (\r{ho} = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.
Nowadays, large multilingual translation models demonstrate impressive translation capabilities in the machine translation benchmarks. This raises a practical question to the developers: is it worth developing translation models specialized for a particular language pair if you only need to support that language pair? To give an anecdotal answer to this question, we develop a family of small language models (0.8B, 1.4B, 3.3B, and 7B parameters) specialized for Japanese-English bidirectional translation. We employ a two-stage supervised fine-tuning approach followed by Multi-Objective GRPO (Ichihara et al. 2025) to train models on synthetically generated parallel corpora. We evaluate our models on WMT and real-world translation benchmarks across business, legal, medical, financial, and patent domains. While multilingual models achieve strong performance on WMT benchmarks, our compact models outperform them on real-world benchmarks, suggesting the practical utility of developing specialized translation models even in the era of large multilingual models.
Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction following benchmarks ignore the cross-lingual nature of translation. We introduce \bench, a benchmark for multilingual translation instruction following covering seven languages, with 4,506 single-constraint and 2,838 multi-constraint items spanning six constraint dimensions and five compositional patterns with instructions issued in all seven languages. Constraints are split into a gating subset verified by deterministic checkers and a continuous subset scored by a rubric-based LLM judge, combined under a multiplicative rule that resists reward hacking. Evaluating 15 models reveals systematic gaps that prior protocols miss: Instruction following scales with size more sharply than translation quality, glossary and structured-format constraints dominate the difficulty gradient, and general instruction following rankings correlate only weakly with translation behavior. Our benchmark are available at https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench.