Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this issue by processing raw UTF-8 characters, yet they create a granularity mismatch for word-level tasks in non-Latin scripts. Hierarchical byte-level architectures address this mismatch by grouping bytes into word-aligned chunks. However, these architectures require massive training data and suffer from representational misalignment when paired with frozen subword-based language models. In this paper, we propose an adapted hierarchical network framework that bridges this modality gap without extensive training. Our method initializes byte embeddings directly from the subword representations of a frozen base model. We apply a chunk alignment loss to project dynamically grouped byte chunks toward precomputed subword targets, and interleave lightweight part-of-speech (POS) supervision to guide boundary detection. Experiments across six languages demonstrate that our tokenizer-free approach improves performance for word-level morphological tasks, yielding up to a 13.3% improvement on POS tagging.
Figures & tables
Qwen3 Subword
Gemma3 Subword
H-Net (Qwen3)
H-Net (Gemma3)
Language
Words
#SW
SW/W
Split (%)
#SW
SW/W
Split (%)
C/W
1:1 (%)
C/W
1:1 (%)
Hindi
281,057
1,155,744
4.11
93.7
445,511
1.59
42.1
1.24
92.5
0.98
90.4
Bhojpuri
6,665
29,491
4.43
97.7
11,708
1.76
52.9
1.17
91.0
0.95
79.6
Marathi
2,997
12,917
4.31
81.6
5,354
1.79
56.0
1.42
82.5
0.94
89.0
Magahi
7,702
37,740
4.90
97.2
13,401
1.74
52.5
1.17
87.8
0.97
91.5
Sanskrit
1,843
10,542
5.72
98.9
4,533
2.46
75.9
1.96
87.3
1.00
99.6
Table 1: Tokenizer fragmentation and H-Net chunk alignment statistics across six Indic languages. #SW stands for the total number of subwords. SW/W and C/W denote the subword-to-word and chunk-to-word ratios, respectively. Split (%) indicates the percentage of words fragmented into two or more subword tokens, while 1:1 (%) represents the percentage of chunks covering exactly one word.
Figure 1: Overview of the proposed H-Net architecture for low-resource languages.
Language
Script
Qwen3-1.7B
Qwen3-4B
Gemma3-4B
Gemma3-12B
Sub
CPT
H-Net
Sub
CPT
H-Net
Sub
CPT
H-Net
Sub
CPT
H-Net
①
Part-of-Speech Tagging (Accuracy %)
Bhojpuri
Devanagari
43.8
48.1
54.4
47.9
49.3
57.9
59.2
56.9
56.9
55.2
56.4
56.8
Marathi
Devanagari
39.1
35.3
46.3
44.0
36.8
46.0
55.6
44.5
59.2
53.9
41.7
58.4
Magahi
Devanagari
41.0
40.9
53.1
45.3
43.2
46.5
55.3
51.9
53.1
49.8
50.4
52.3
Sanskrit
Devanagari
21.6
30.2
30.6
25.7
35.1
33.2
21.9
32.8
30.6
30.6
31.1
32.7
Table 2: Zero-shot performance across three downstream tasks. Bold indicates the best result among Sub, CPT, and H-Net per task and model family.
Figure 2: BPB during H-Net pretraining under random ( red ) vs backbone ( blue ) byte embedding initialization.
Figure 3: Training dynamics with random ( red ) vs backbone ( blue ) byte embedding initialization. Left : average boundary ratio over 20,000 steps oscillates at 0.43–0.49. Right : average chunk length.
Pretraining
Byte-level
Chunk-level
Without POS
32.5
43.1
With POS
32.8
59.5
Table 3: Hindi POS accuracy (%) with and without task-guided pretraining (Qwen3-1.7B backbone).
Language
Full model
w/o align loss
C/W
1:1%
C/W
1:1%
Hindi
1.24
92.5
0.05
5.5
Bhojpuri
1.17
91.0
0.06
3.4
Marathi
1.42
82.5
0.13
0.0
Magahi
1.17
87.8
0.07
0.0
Sanskrit
1.96
87.3
0.13
0.4
Table 4: Chunk-word alignment statistics for the full H-Net model vs without chunk alignment loss ( λalign=0 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Hyperparameter
Value
H-Net pretrain
Optimiser
AdamW
Learning rate
5×10−5
Weight decay
0.01
Max grad norm
0.5
LR schedule
Cosine + 5% warmup
Batch size
8 sequences
Appendix
Table 5: Hyperparameters for H-Net pretraining, CPT baseline, and downstream probe training.
Task
Model
Sub
H-Net
POS
Qwen3-1.7B
65.3
59.5
Qwen3-4B
71.6
72.1
Gemma3-4B
94.3
87.4
NER
Qwen3-1.7B
24.6
29.9
Qwen3-4B
25.2
34.6
Gemma3-4B
52.2
55.4
Appendix
Table 6: In-language Hindi evaluation on the held-out UD Hindi test set. Bold indicates the best result per row.