The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction

Hyunsik Kim

cs.CLOct 15, 2026

The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction

Authors: Hyunsik Kim

Organizations: Samsung Research

Abstract

Many hierarchical byte-level language models group bytes into patches using a learned boundary predictor whose view of the next codepoint depends on encoding and lookahead. In UTF-8, codepoints of many scripts take three bytes: 8 of 24 bits mark length and continuation, and the first byte carries only 4 high codepoint bits. After the same left context, a mark extending a grapheme cluster and a letter starting the next can thus share their first byte. Universal Byte-Level Encoding (UBE) keeps one- and two-byte UTF-8 encodings but writes these codepoints as two UTF-16 bytes, low byte first, so its first symbol carries 8 low codepoint bits that can distinguish such continuations. We train small causal predictors on matched FineWeb-2 text and evaluate on FLORES-200 under UTF-8, UBE, and UBE's same-length big-endian variant. At one-symbol lookahead, balanced grapheme-boundary error at codepoint boundaries across 17 languages in 13 Brahmic scripts is 8.76% with UTF-8, 8.62% with the big-endian variant, and 0.06% with UBE. When the target instead retains a parent tokenizer's subword cuts at grapheme boundaries, UBE lowers the mean error relative to UTF-8 at the same lookahead by 72.91% in Korean, 35.29% in Chinese, and 67.32-99.55% in eight more scripts. In the seven of them whose main Unicode block fits on one 256-codepoint page, the big-endian variant stays within 3.0% of UTF-8, so the gain follows byte order, not length. In Korean and Chinese, UBE's mean error falls below UTF-8's final mean error after only 25% and 37.5% of training. Which byte comes first thus affects how much lookahead a byte-level boundary predictor needs.

Figures & tables

Panel of Figure 1 — Figure 1: Boundary errors at one-symbol lookahead ( $k=1$ ) and with the whole next scalar visible ( $k=3$ for UTF-8; $k=2$ for UBE and UBE-BE). (a) Every-cluster target on the 17-language FLORES-200 panel: balanced error at scalar ends, the mean of the missed-boundary and false-cut rates; within-scalar cuts are reported separately (Section 4 ). (b, c) Retained parent-tokenizer cuts in Korean and Chinese, where every evaluated cluster is one scalar: errors per 1,000 scalars, including within-scalar false cuts. Bars are means over 2 seeds in (a) and 3 in (b, c); dots show seeds.

Figure 2: How the first symbol can expose the distinction needed for segmentation. (a) Bit layout of a scalar that needs three UTF-8 bytes ( x : codepoint bit) under UTF-8 and UTF-16; UBE writes the low byte first and UBE-BE the high byte first. (b) After the same Devanagari base, vowel sign I extends the cluster and letter MA starts a new one (outlined: target clusters; dotted circle: isolated mark). Both scalars share the high byte 09 , so their first symbols coincide under UTF-8 and UBE-BE and differ only under UBE (P5, Section 3 ). Byte values are hexadecimal; one-symbol lookahead sees only the solid first symbol.

Panel of Figure 3 — Figure 3: Three mechanisms of tokenizer inheritance and two sequence lengths. Blue solid arrows trace input encoding, green dash-dotted arrows vocabulary features, and orange dashed arrows boundary supervision. Cluster alignment changes the supervision target; encoding choice changes the local symbol stream. Transferring the vocabulary features to UBE requires a separate alignment policy.

Encoding	$k=1$	$k=2$	$k=3$
UTF-8	8.76	6.37	0.20
UBE	0.06	0.16	0.19
UBE-BE	8.62	0.18	0.20

Table 1: Balanced error (%, two-seed means) at FLORES-200 devtest scalar ends with

k

symbols of lookahead. UBE-BE has UBE’s routing and stream length but reverses the byte order within each UTF-16 code unit. Within-scalar errors are reported separately in the text and Appendix D .

		Every cluster			Retained parent cuts
Encoding	$k$	Korean	Chinese	Mean	Korean	Chinese	Mean
UTF-8	1	0.000	0.000	0.000	54.438	159.946	107.192
UBE	1	0.035	0.046	0.041	14.745	103.496	59.121
UBE-BE	1	0.000	0.000	0.000	25.357	121.239	73.298
UTF-8	3	0.000	0.008	0.004	11.147	103.242	57.195
UBE	2	0.096	0.023	0.060	10.556	89.276	49.916

Table 2: Korean/Chinese extension: false cuts plus missed target cuts per 1,000 Unicode scalars, averaged over 3 seeds. False cuts include positions inside scalars; the text denominator is shared across encodings. “Mean” weights the two languages equally. Both targets and all lookahead conditions are shown. Lower is better.

	$N$	$A_{\rm in}$	$B_{\rm in}$	Split %
Qwen3 parent	3,370,517	2,003,437	1,217,605	99.89
Bwen-8B	3,359,694	1,995,374	1,214,734	99.49
Bwen-8B †	1,752,896	573,822	1,578	66.18

Table 4: Boundary audit totals over all 22 languages.

N

counts text units,

A_{\rm in}

cuts strictly inside a grapheme cluster, and

B_{\rm in}

cuts strictly inside a Unicode scalar; “Split” is the percentage of multi-scalar clusters that receive at least one cut. † Without the parent-vocabulary embedding, no retraining.

Global model: patch positions per codepoint
	Panel	English	$\times$ English
Parent target	1.2768	0.2093	6.10
Bwen-8B, realized	1.2743	0.2057	6.19
Retained-parent target	0.4705	0.2093	2.25
Local modules: byte symbols per codepoint
UTF-8 bytes	2.7125	1.0010	2.71

Table 5: Positions per NFC codepoint, pooled over the 17-language panel, with English as a reference. Global computation uses patch positions; local computation uses symbol positions. “Retained-parent target” counts the units remaining after deletion of the parent’s cluster-internal cuts.

Panel of Figure 4 — Figure 4: Text positions per NFC codepoint on full devtest, excluding BOS. Parent-target and realized Bwen markers nearly coincide. The retained-parent target deletes cluster-internal parent cuts. Vertical English references are nearly coincident; the two target densities are identical in English.

Model	Items changed	%	Flag share (%)	Bytes/patch
Bwen-8B	458/850	53.9	13.19	2.127
Bolmo-1B	328/850	38.6	9.76	1.746
Bolmo-7B	326/850	38.4	10.46	1.746

Table 6: Instability in patch counts under single-scalar, span-preserving edits, measured with standalone forwards on 50 items per language. Flag share is measured on the original text. Bytes/patch uses all sentence bytes and counts the forced BOS flag in the denominator. The dolma2 conversions share an item bank; Bwen uses different substitutions and prefixes. Bwen’s row uses the first 50 of its 200 items per language (all 3,400 items: 52.5%, Section 5.4 ); flag share pools path cost over items.

Pair	$\Delta R$	95% interval	Excl. 0
Qwen3 $\to$ Bwen-8B	+0.0177	[+0.0074, +0.0283]	7/17
OLMo-2 $\to$ Bolmo-1B	+0.0064	[−0.0080, +0.0214]	2/17

Table 7: Pooled margin-ratio differences, conversion minus parent. Intervals are paired percentile bootstrap intervals over shared FLORES source sentences, pointwise and without multiplicity correction. “Excl. 0” counts languages whose own interval excludes zero.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

		Parent (Qwen3)				Bwen-8B
Language	$K$	$N$	$A_{\rm in}$	$B_{\rm in}$	$S$	$N$	$A_{\rm in}$	$B_{\rm in}$	$S$	Shared %
Amharic	87,132	111,737	40,880	40,880	0	111,600	40,752	40,752	0	99.87
Assamese	83,536	148,763	93,534	54,661	38,873	144,474	92,098	54,662	37,436	97.07
Bengali	81,693	138,814	87,846	46,927	40,919	138,133	87,858	46,948	40,910	99.47
English ∗	131,966	27,621	0	0	0	27,149	0	0	0	97.71
Gujarati	80,578	186,119	125,560	79,712	45,848	186,117	125,559	79,711	45,848	99.99

Appendix

Table 8: Parent-tokenizer and predicted Bwen boundaries on full devtest. Grapheme clusters are the units of the fragmentation audit; they need not coincide with linguistic syllables.

Language	$N$	$A_{\rm in}$	$S$	$A_{\rm in}$ /cut	Coarsening	Multi split %
Amharic	34,434	0	0	0.0000	3.31	—
Assamese	103,183	37,498	37,498	0.3670	1.45	88.07
Bengali	107,247	41,026	41,026	0.3862	1.30	88.31
English ∗	49,290	0	0	0.0000	0.55	—
Gujarati	103,873	45,243	45,243	0.4398	1.80	98.61
Hindi	108,523	44,480	44,480	0.4137	1.12	98.53

Appendix

Table 9: Parent-vocabulary embedding removal without retraining. Coarsening is the ratio of parent to ablated interior-cut counts; values below one indicate more cuts in the ablated predictor than in the parent segmentation.

A_{\mathrm{in}}

/cut is the fraction of interior cuts that fall strictly inside a cluster. A dash marks an undefined split percentage because no multi-scalar clusters occur.

Language	Scalars $M$	Parent $N$	UTF-8	UBE	Parent	Bwen	Retained	UBE saving %
			Symbols		Positions per codepoint
Assamese	127,594	148,763	340,036	233,823	1.1659	1.1323	0.4328	31.2
Bengali	130,391	138,814	348,729	239,566	1.0646	1.0594	0.3909	31.3
Gujarati	126,428	186,119	331,068	228,751	1.4721	1.4721	0.4790	30.9
Hindi	131,180	121,957	337,439	234,314	0.9297	0.9287	0.3941	30.6
Kannada	138,027	191,127	375,341	256,686	1.3847	1.3845	0.5456	31.6

Appendix

Table 10: Sequence-length accounting for the full panel and reference languages. Scalars

M

are NFC codepoints, and Parent

N

counts parent tokens. Symbol columns give UTF-8 byte and UBE symbol counts. Positions per codepoint distinguish parent-target density, realized Bwen density, and retained-parent target density (“Retained”). UBE saving is the percentage reduction in symbol count.

Script	Vowel sign	Consonant	UTF-8 first (sign / base)	UTF-16LE first (sign / base)
Devanagari	U+093F	U+092E	E0 / E0	3F / 2E
Bengali	U+09BF	U+09AE	E0 / E0	BF / AE
Oriya	U+0B3F	U+0B2E	E0 / E0	3F / 2E
Sinhala	U+0DD2	U+0DB8	E0 / E0	D2 / B8
Thai	U+0E34	U+0E21	E0 / E0	34 / 21
Myanmar	U+102D	U+1019	E1 / E1	2D / 19

Appendix

Table 11: Hexadecimal continuation scalars and their first byte values under UTF-8 and UTF-16LE. The UTF-16LE values occupy UBE’s separate UTF-16 symbol alphabet.

Figure 5: Encoding and segmentation of Hindi lekin (‘but’), tokenized in isolation with a leading space. Qwen3 splits 16 UTF-8 bytes into 6 tokens: 3 cuts fall inside a scalar (red) and 2 between scalars within a cluster (orange); none falls at a cluster boundary. Deleting these cuts leaves 1 target unit; cutting at every cluster boundary gives 4 units, including the space. UBE uses 11 symbols (white: UTF-8; violet: UTF-16LE), drawn at the UTF-8 byte width. Inset: after identical left context, the same next UTF-8 byte hides whether the next scalar extends the cluster, whereas UBE’s first symbol distinguishes the two (P5). The word was chosen post hoc from the first 300 Hindi FLORES-200 devtest sentences and is illustrative, not representative.

Encoding	$k$	Balanced 0 / 1	Missed	False cut	Within scalar	Pairs both right
		Error rate (%)
UTF-8	1	8.765 / 8.747	9.504	8.008	0.000	0 / 0
UTF-8	2	6.314 / 6.434	5.114	7.634	0.000	4,676 / 4,949
UTF-8	3	0.217 / 0.179	0.049	0.347	0.000	17,204 / 17,204
UBE	1	0.021 / 0.101	0.100	0.021	0.194	17,204 / 17,204
UBE	2	0.128 / 0.187	0.024	0.290	0.054	17,204 / 17,204

Appendix

Table 12: Probe errors (%) and same-context pairs answered correctly, out of 17,204 per seed. “Balanced 0 / 1” gives scalar-end balanced error separately by seed; the other error columns are two-seed means. Missed boundaries and false cuts use only scalar ends, while “Within scalar” uses within-scalar sites. Pair counts are also seed 0 / 1.

Language	Script	UTF-8	UBE	UBE-BE	UBE seeds 0 / 1
		Balanced error at $k=1$ (%)
Assamese	Bengali	8.294	0.002	7.834	0.002 / 0.002
Bengali	Bengali	8.060	0.005	7.350	0.005 / 0.005
Burmese	Myanmar	10.556	0.004	10.485	0.005 / 0.003
Gujarati	Gujarati	9.231	0.000	8.906	0.000 / 0.000
Hindi	Devanagari	8.504	0.003	8.055	0.003 / 0.004

Appendix

Table 13: One-symbol balanced error (%) at scalar ends by panel language, two-seed means; the last column lists UBE’s two seeds.

Contrast	Stated direction	Estimate (pp)	95% interval (pp)
UBE $k{=}1$ $-$ UTF-8 $k{=}1$	negative	−8.695	[−8.757, −8.633]
UTF-8 $k{=}3$ $-$ UTF-8 $k{=}1$	negative	−8.558	[−8.620, −8.496]
UBE $k{=}1$ $-$ UBE-BE $k{=}1$	negative	−8.555	[−8.617, −8.493]
UBE-BE $k{=}2$ $-$ UBE-BE $k{=}1$	negative	−8.438	[−8.499, −8.378]
UBE $k{=}1$ $-$ UTF-8 $k{=}3$	none stated	−0.137	[−0.148, −0.127]

Appendix

Table 14: Probe contrasts in balanced error (percentage points), differences of two-seed means.

		Balanced error (%)
Encoding	$k$	All scalar ends	Next to a joiner	Other scalar ends
UTF-8	1	8.7557	50.00	8.5994
UTF-8	2	6.3742	50.15	6.2133
UTF-8	3	0.1980	61.22	0.0019
UBE	1	0.0609	3.66	0.0492
UBE	2	0.1573	40.62	0.0123

Appendix

Table 15: Post hoc: balanced error (%, two-seed means) at all scalar ends, at the scalar ends next to a zero-width joiner or non-joiner, and elsewhere. Each column uses its own class denominators.

Language	Scalars	UTF-8	UBE	Saved %	Parent	Retained	Split %
Korean	65,965	157,799	111,887	29.10	45,020	41,075	4.74
Chinese (Simplified)	43,248	120,803	82,081	32.05	27,701	27,391	0.61

Appendix

Table 16: Korean/Chinese structural accounting on NFC devtest. “Retained” counts target units after deleting incompatible Qwen3 cuts. “Split” is the percentage of clusters split by Qwen3, before deletion. Stream savings compare UBE symbols with UTF-8 bytes; UBE-BE has the same length as UBE.

Target	Baseline	Korean	Chinese
Every cluster	Scalar decoder	0.00	0.00
Every cluster	No cuts	984.66	976.60
Retained parent cuts	Scalar decoder	377.32	366.65
Retained parent cuts	No cuts	607.34	609.95

Appendix

Table 17: Deterministic baselines in errors per 1,000 Unicode scalars. Both respect scalar interiors.

Every cluster
Encoding	$k$	Korean: seeds 0 / 1 / 2	Chinese: seeds 0 / 1 / 2
UTF-8	1	0.000 / 0.000 / 0.000	0.000 / 0.000 / 0.000
UBE	1	0.000 / 0.030 / 0.076	0.023 / 0.000 / 0.116
UBE-BE	1	0.000 / 0.000 / 0.000	0.000 / 0.000 / 0.000
UTF-8	3	0.000 / 0.000 / 0.000	0.023 / 0.000 / 0.000
UBE	2	0.258 / 0.015 / 0.015	0.069 / 0.000 / 0.000

Appendix

Table 18: Errors per 1,000 Unicode scalars for every trained seed, including within-scalar cuts.

Every cluster
	Korean		Chinese
Contrast	Difference	95% interval	Difference	95% interval
UBE $k=1$ $-$ UTF-8 $k=1$	+0.035	[+0.000, +0.095]	+0.046	[+0.015, +0.086]
UBE $k=1$ $-$ UBE-BE $k=1$	+0.035	[+0.000, +0.095]	+0.046	[+0.015, +0.086]
UBE $k=2$ $-$ UTF-8 $k=3$	+0.096	[+0.005, +0.229]	+0.015	[+0.000, +0.039]
UBE $k=2$ $-$ UBE-BE $k=2$	+0.096	[+0.005, +0.229]	+0.023	[+0.000, +0.062]

Appendix

Table 19: UBE-minus-control contrasts in errors per 1,000 Unicode scalars; negative values favor UBE. Intervals resample parallel source sentences and condition on the three seeds.

Every cluster
Encoding	$k$	Language	Within / 1,000	Missed %	False cut %	F1 %	Split %
UTF-8	1	Korean	0.000	0.000	—	100.000	0.000
UTF-8	1	Chinese	0.000	0.000	—	100.000	0.000
UBE	1	Korean	0.025	0.001	—	99.998	0.003
UBE	1	Chinese	0.031	0.002	—	99.998	0.003
UBE-BE	1	Korean	0.000	0.000	—	100.000	0.000

Appendix

Table 20: Supporting metrics, averaged over the three seeds. Within-scalar cuts use the common text denominator; missed boundaries and scalar-end false cuts use their own class denominators. A dash means the negative scalar-end class is absent. F1 includes within-scalar false cuts; “Split” counts clusters with at least one internal cut.

Language	Position	Scalar gaps	Target cuts %
Korean	Before whitespace	14,418	100.00
Korean	After whitespace	14,418	2.53
Korean	Between whitespace	4	75.00
Korean	Neither side	36,113	69.99
Chinese	Before whitespace	1,057	100.00
Chinese	After whitespace	1,057	36.23

Appendix

Table 21: Whitespace partition of the retained-parent target. Counts refer to interior scalar gaps; target-cut percentages use the row’s gap count. A dash marks an absent stratum.

Korean
Encoding	$k$	Before	After	Between	Neither side
UTF-8	1	0.000	0.000	25.000	9.941
UBE	1	0.000	0.005	25.000	2.688
UBE-BE	1	0.000	0.000	33.333	4.627
UTF-8	3	0.009	0.000	75.000	2.023
UBE	2	0.000	0.000	25.000	1.920

Appendix

Table 22: Post hoc scalar-gap error percentages by whitespace context. “Before,” “After,” and “Between” refer to whitespace; “Neither side” has no whitespace on either side. All encoding/lookahead conditions are included, with 3-seed means. These percentages have different denominators from the primary errors per 1,000 scalars.

	One symbol ( $k=1$ )			Whole next scalar
Language	UBE $-$ UTF-8	UBE $-$ UBE-BE	UBE-BE $-$ UTF-8	UBE $-$ UTF-8	UBE $-$ UBE-BE
Japanese	−79.94	−71.05	−8.89	−2.53	−0.17
	[−82.55, −77.43]	[−73.64, −68.52]	[−10.33, −7.42]	[−3.37, −1.66]	[−0.89, +0.60]
Thai	−56.37	−54.73	−1.64	−0.67	−0.22
	[−58.10, −54.64]	[−56.42, −53.01]	[−2.18, −1.08]	[−0.98, −0.34]	[−0.51, +0.08]
Lao	−67.00	−65.83	−1.18	+0.18	−0.02

Appendix

Table 23: Paired contrasts for the eight-script extension: differences in errors per 1,000 scalars between three-seed means, each with its 95% source-sentence bootstrap interval in brackets below. Negative values favor the first encoding. The whole-next-scalar columns compare UBE at

k=2

with UTF-8 at

k=3

and with UBE-BE at

k=2

Korean and Chinese (Section 4.1 )
Language	Encoding	250	500	750	1,000	1,250	1,500	1,750	2,000
Korean	UTF-8	80.1	64.1	62.0	57.9	58.4	54.8	54.2	54.4
Korean	UBE-BE	60.6	37.8	31.2	29.3	27.0	26.4	25.7	25.4
Korean	UBE	62.1	30.2	21.6	18.5	16.0	15.6	15.1	14.7
Chinese	UTF-8	235.7	210.3	192.6	180.4	173.0	164.9	162.2	159.9
Chinese	UBE-BE	215.8	181.7	158.6	143.6	135.9	127.7	123.4	121.2

Appendix

Table 24: Learning curves at one-symbol lookahead (

k=1

): retained-parent errors per 1,000 scalars after each number of updates, averaged over three seeds. The final column equals the

k=1

values in Tables 3 and 3 .

Language	base	mark	base	mark	parent	conv.	$\Delta R$	95% interval	$\Delta I$	95% interval
	Parent margin		Conv. margin		Ratio $R$
Assamese	25.88	22.49	19.57	16.35	0.8689	0.8354	−0.0335	[−0.0865, +0.0197]	−0.0123	[−0.0527, +0.0261]
Bengali	29.69	25.90	25.41	22.35	0.8722	0.8796	+0.0074	[−0.0358, +0.0506]	−0.0083	[−0.0350, +0.0175]
Gujarati	28.52	22.37	24.41	19.96	0.7845	0.8179	+0.0334	[+0.0059, +0.0636]	−0.0260	[−0.0475, −0.0049]
Hindi	30.26	24.51	27.92	22.99	0.8098	0.8232	+0.0135	[−0.0147, +0.0415]	−0.0126	[−0.0317, +0.0056]
Kannada	30.51	24.70	24.70	20.78	0.8095	0.8412	+0.0317	[−0.0067, +0.0715]	−0.0184	[−0.0460, +0.0099]

Appendix

Table 25: Qwen3-8B-Base and Bwen-8B, 200 items per language. Paired, pointwise percentile intervals; no multiplicity correction.

Language	base	mark	base	mark	parent	conv.	$\Delta R$	95% interval	$\Delta I$	95% interval
	Parent margin		Conv. margin		Ratio $R$
Assamese	17.01	16.62	11.98	11.35	0.9772	0.9471	−0.0301	[−0.1002, +0.0424]	+0.0358	[−0.0160, +0.0888]
Bengali	21.17	19.94	15.78	15.39	0.9421	0.9753	+0.0331	[−0.0190, +0.0872]	−0.0282	[−0.0613, +0.0045]
Gujarati	20.17	16.66	13.52	11.86	0.8257	0.8775	+0.0517	[−0.0022, +0.1114]	−0.0578	[−0.1023, −0.0131]
Hindi	24.57	22.18	21.42	19.01	0.9029	0.8873	−0.0156	[−0.0558, +0.0252]	−0.0129	[−0.0462, +0.0189]
Kannada	22.32	18.32	15.94	12.69	0.8205	0.7965	−0.0240	[−0.0655, +0.0180]	+0.0315	[−0.0018, +0.0688]

Appendix

Table 26: OLMo-2-0425-1B and Bolmo-1B, 200 items per language. This bank uses the OLMo parent tokenizer convention. Its items and prefixes need not equal those of the first pair.

Omitted language	$\Delta R$	$\Delta I$
Assamese	+0.0205	−0.0206
Bengali	+0.0183	−0.0208
Gujarati	+0.0169	−0.0197
Hindi	+0.0185	−0.0206
Kannada	+0.0167	−0.0202
Khmer	+0.0200	−0.0219

Appendix

Table 27: Leave-one-language-out point estimates for pair one.

Language	Bwen	1B	7B	1B=7B	Bwen	1B	7B	Bwen	1B	7B
	Instability (%)			of 50	Flag share (%)			Bytes per patch
Assamese	72	64	64	50	8.97	13.07	13.18	2.352	2.057	2.053
Bengali	74	82	82	50	7.50	14.20	14.16	2.523	2.201	2.200
Gujarati	34	0	0	50	12.48	1.21	1.36	1.764	1.590	1.591
Hindi	68	80	80	50	15.64	17.82	18.49	2.771	2.629	2.630
Kannada	42	0	0	50	13.37	1.37	2.00	1.950	1.566	1.564

Appendix

Table 28: Per-language instability, agreement of the dolma2 pair’s instability indicators (1B=7B, out of 50 items), flag share of path cost, and bytes per patch. Bwen uses a different item bank, precluding a paired magnitude comparison with the dolma2 models.

Model	Bytes/patch, instability	Bytes/patch, flag share	Flag share, instability
Bwen-8B	+0.620	−0.369	−0.091
Bolmo-1B	+0.757	+0.511	+0.894
Bolmo-7B	+0.764	+0.504	+0.883

Appendix

Table 29: Unweighted Pearson associations over language aggregates. They do not identify a mechanism or propagate uncertainty in the aggregates.

Language	Parent	Bwen path	Gap	Flag term	Gap $-$ flag	Flag share %
	Bits per scored byte
Assamese	0.7307	0.9526	0.2219	0.0855	0.1364	8.97
Bengali	0.5550	0.6505	0.0955	0.0488	0.0467	7.50
Gujarati	0.6361	0.7689	0.1328	0.0959	0.0369	12.48
Hindi	0.5150	0.5798	0.0648	0.0907	−0.0259	15.64
Kannada	0.5734	0.7487	0.1753	0.1001	0.0752	13.37

Appendix

Table 30: Bwen flag decomposition on the matched original-sentence subset. Parent, path, gap, flag, and remainder columns are bits per scored byte; the final column is flag share. Negative remainders do not establish a marginal-likelihood advantage.

Model	Variants	Max $\|\Delta\|$ (bits)	Mean $\|\Delta\|$ (bits)	Paths moved
Bwen-8B	306	12.32	1.54	10
Bolmo-1B	2,550	41.28	1.83	11
Bolmo-7B	2,550	20.88	1.06	29

Appendix

Table 31: Absolute score changes and boundary-path changes when the same original/mark/base variants are scored together instead of alone. Sample sizes differ across checkpoints.

Language	Inflation %	Added bits	First-token bits	Ratio
Assamese	44.7	146.3	17.3	8.4
Bengali	61.2	159.1	18.2	8.7
Gujarati	76.6	194.3	19.8	9.8
Hindi	83.9	179.0	19.8	9.0
Kannada	78.7	199.2	18.2	11.0
Khmer	88.4	325.2	20.1	16.2

Appendix

Table 32: Qwen prefix sensitivity. Language rows average per-sentence inflation fractions and costs; the ratio column divides summed extra cost by summed first-token cost. The panel percentage in Section 5.6 is a ratio of pooled costs, not a mean of the language percentages.

Tokenization in Language Models

See all 46 papers

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities
Oct 1, 2026Hyunsik Kim, Youngmoon JungMultilingual Language Models Token Efficiency
Samsung Research
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Byte-Level Language Model LLM Reliability
Google LLC / Mountain View, CA, USA · Institute of Science Tokyo / Tokyo, Japan · Mohamed bin Zayed University of Artificial Intelligence / Abu Dhabi, United Arab Emirates
Disentangling Language Modeling and Boundaries
Aug 4, 2026Mykola HaltiukTokenization in Language Models Byte-Level Language Model
Faculty of Computer Science AGH University of Krakow Krakow, Poland

Oct 1, 2026J/K move · Enter open · S save

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

Hyunsik Kim, Youngmoon Jung

Samsung Research

Multilingual Language Models Token Efficiency Subword Tokenization Byte-Pair Encoding

Open paper AskPDF arXiv

Byte-level byte-pair encoding (BBPE) tokenizers are attractive for multilingual large language models (LLMs) because they cover all Unicode text. In UTF-8-based BBPE, however, many scripts start from a higher fallback cost than English: when no learned merges can be applied, a multibyte character requires multiple byte-derived symbols. We call this worst-case pre-merge cost the encoding floor. A higher floor can increase token counts and per-request cost and shrink usable context. Changing the text encoding can reduce this gap, but a single global encoding can make already-efficient English spans more expensive in mixed-script text. We propose Universal Byte-Level Encoding (UBE), a dual-alphabet tokenizer that keeps 1-2-byte UTF-8 characters on the UTF-8 path while routing 3-4-byte UTF-8 characters through UTF-16. This lowers the encoding floor for 3-byte Basic Multilingual Plane (BMP) characters in scripts with high token premiums (token counts relative to English) without raising it for already-efficient spans in mixed-script text. UBE changes only the byte representation presented to byte-pair encoding (BPE); the merge rule remains standard, and exact decoding is preserved. UBE also composes with alternative boundary policies and morphology-based representations. In a Unicode 17 audit, UBE exactly round-trips all Unicode scalar values and all inputs in the official normalization, grapheme-break, and emoji test suites. Across intrinsic evaluations, UBE lowers dispersion in English-normalized token-count ratios, reducing cross-lingual token-budget disparity. In multilingual language model (LM) experiments, UBE matches BBPE's LM quality. In the main multilingual settings, UBE reduces token counts most for high-premium scripts and slightly lowers English token counts, yielding more usable context under fixed token budgets and faster prompt processing in content-matched benchmarks.

Abstract

Tokenization in Language Models

Explore similar work

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

Disentangling Language Modeling and Boundaries