Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.
Figures & tables
Figure 1: Overview of the MBP architecture. An input byte sequence is first processed by the model, and the MBP head uses the LCA mask ( Figure 2 ) to predict multiple future bytes in parallel from a single head. Colored bands denote learned segment boundaries. See Appendix A for a detailed description of several parts in this figure.
Figure 2: The Latent Causal Attention ( LCA ) mask. Each row corresponds to a query byte and each column to a key byte. A query at position i in segment si may attend to itself, to earlier positions within its own segment, and to all bytes in the immediately preceding segment si−1 . Attention to later positions in the same segment and to segments further in the past is masked out. Colored bands denote segment boundaries and match those in Figure 1 . This mask preserves autoregressive causality at the segment level while enabling all bytes in a group to be predicted in parallel from a single decoder head.
Figure 3: Performance vs. throughput across four downstream tasks. Metrics are instruction-loose accuracy on IfEval, F1 on CoQA, ROUGE on CNN/DailyMail, and COMET on Spanish–English translation. LCA ( ⋆ ) lies on the frontier for three of four tasks on the Pareto front, which represents the method with the best tradeoff.
Model
IfEval
CoQA
DailySum
es-en
fr-en
MLP-MBP
61.91
63.45
58.08
62.67
65.01
LCA -MBP
45.95
46.62
48.89
51.68
49.74
Table 1: Byte acceptance rates of MLP and LCA multi-byte prediction heads across five downstream tasks. Acceptance rate measures the fraction of predicted bytes accepted. While MLP achieves higher raw acceptance, indicating that the model is highly confident about its prediction, it performs worse than LCA -MBP on the Pareto front.
Figure 4: Ablation on the acceptance threshold, averaged across DailySum, es-en, and fr-en. Each star marks LCA evaluated at a different value of τ . Lowering τ admits more speculative tokens, increasing throughput and acceptance rate.
Figure 5: Performance vs. Candidates ( n at τ=0.75 .) Marker color encodes throughput (darker = faster; full values in Table 3 ). COMET scores (es-en, fr-en) scaled × 100.
Figure 6: Throughput vs. candidates n under speculative decoding acceptance. Solid lines show LCA throughput growing monotonically with n on all three tasks while downstream performance is maintained ( Table 4 in Appendix B ).
Figure 7: Distribution of accepted speculative bytes over 100 decoding steps. LCA accepts a mean of 3.05 of the 6 speculative candidates per step, and 15% of steps accept the full window, indicating that LCA ’s multi-byte predictions are frequently coherent across an entire segment rather than accepted one byte at a time.
Figure 8: Throughput vs. performance across four inference strategies. We compare LCA ’s next-byte head only, LCA ’s self-verified multi-byte prediction, LCA with FxT verifying the MBP head, and the FxT baseline. External verification with FxT lets LCA match or exceed FxT’s performance while running 2.1-2.3 × faster across all three tasks with n=7 . See acceptance rate in Table 5 of Appendix B .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Encoder
LM Module
Decoder NB
Decoder MB
Module
NT
MB
LlamaByte
22
0
0
0
SpaceByte
2
16
4
0
FxT
2
16
4
0
MLP-MBP
2
16
2
3
LCA -MBP
2
16
2
2
Appendix
Table 2: Architectural configurations used across model variants. Unless otherwise stated, all models use the same hidden dimension, training hyperparameters, and optimizer settings. For MLP-MBP trained with compression rate 5 , the MB decoder uses 5 layers instead of 3 .
Figure 9: Performance vs. throughput on French–English translation. LCA ( ⋆ ) lies on the Pareto front.
n
DailySum
es-en
fr-en
3
210.42
285.75
278.21
4
205.42
283.47
278.13
5
194.50
254.57
263.99
6
181.24
234.55
252.31
7
164.18
221.09
230.99
Appendix
Table 3: Throughput (bytes/sec) of LCA -MBP across speculative-candidate counts n∈{3,4,5,6,7} at acceptance threshold τ=0.75 . Throughput decreases monotonically with n on all three tasks, as larger speculative windows incur more rejected candidates per step.
Model
IfEval
CoQA
DailySum
es-en
fr-en
Avg
Performance
MLP-MBP
46.54
20.85
32.73
0.8178
0.8064
–
LCA-MBP
44.97
22.11
38.09
0.8147
0.7977
–
Acceptance rate (%)
MLP-MBP
71.80
90.15
90.79
91.82
91.82
87.28
LCA-MBP
51.17
73.93
81.22
79.85
79.85
73.20
Appendix
Table 4: Performance and acceptance rate under speculative decoding acceptance. Performance is constant across the number of speculative candidates n ; acceptance rates are reported at n=3 .
Model
DailySum
es-en
fr-en
FxT
100.00
100.00
100.00
LCA NBP
100.00
100.00
100.00
LCA Self-verify
46.36
45.85
45.85
LCA FxT-verify
47.97
50.77
52.04
Appendix
Table 5: Token acceptance rates (%) across three tasks under self and external (FxT) verification using n=7 candidate bytes.