Long-context DNA models are limited by token-mixing cost and by how compression allocates representational budget across the genome. Existing approaches operate close to base-pair resolution, apply fixed downsampling, or learn content-dependent chunks without an explicit genomic budget, making long-context pretraining expensive and difficult to control. We introduce GeneZip, a region-aware DNA compression framework that combines H-Net-style dynamic routing with a Region-Aware Ratio (RAR) objective and bounded routing. GeneZip uses static gene-structure annotations during compression training to specify region-wise base-pairs-per-token (BPT) targets; at inference time, it compresses raw unseen DNA without annotations. GeneZip provides three main benefits. First, it is effective: GeneZip variants achieve the best validation PPL among encoder-based compressors, with GeneZip-70M operating at 137.6 BPT, and across four reproducible DNALongBench tasks--contact map prediction, eQTL prediction, enhancer-target gene prediction, and transcription-initiation signal prediction--GeneZip obtains the best average rank among compared sequence models. Second, it is redundancy-aware: a post-hoc RepeatMasker/TRF analysis shows that, without repeat supervision, GeneZip assigns higher local BPT to TE-derived interspersed repeats and tandem repeats, two major classes of repetitive DNA sequence redundancy. Third, it is efficient: by reducing the effective token-mixing length, GeneZip enables longer-context and larger-capacity pretraining, including 128K-context and 636M-parameter variants on a single A100 80GB GPU, and fine-tunes the eQTL task 50.4x faster than JanusDNA (50 vs. 2520 minutes). These results establish GeneZip as an effective, redundancy-aware, and efficient compression interface for long-context DNA modeling.
Figures & tables
Figure 1: GeneZip overview. GeneZip compresses long-context DNA into a much shorter latent token sequence via hierarchical dynamic routing. During training, the Region-Aware Ratio (RAR) loss learns a region-adaptive token budget, and bounded routing enforces global token-count constraints for stable training. The compressed tokens are then processed by an arbitrary token-mixing backbone for downstream long-range tasks. After training, GeneZip compresses unseen raw DNA sequences without requiring annotations or repeat masks at inference time.
Figure 2: Why bounded routing is necessary. (a) In stage 1 compression, the router can be highly unstable early on and occasionally selects an excessively large number of tokens, which risks memory blow-up; a ceiling constraint prevents such pathological spikes. (b) In stage 2 compression, the router may collapse to selecting too few tokens (often near 1) at the beginning, which can induce training spikes and slow down optimization; a floor constraint enforces a minimum token budget for stable learning. (c) The floor constraint yields smoother and more favorable perplexity trajectories compared to training without it.
Figure 3: Inference efficiency on long-context inputs. End-to-end inference latency as a function of input sequence length (256 bp to 1M bp). GeneZip (70M/636M) remains consistently fast across the full range, while baseline long-context models exhibit sharply increasing latency as sequence length grows. The inset zooms into the short-to-mid regime (1K–131K) to highlight differences at smaller contexts.
Model
BPT
Overall ↓
Prom. ↓
CDS ↓
UTR ↓
Exon ↓
Intron ↓
NIG ↓
Backbone / fine-grained upper bounds
Mamba2-84M
1.0
2.4135
2.0494
2.4136
2.5608
2.6346
2.5076
2.4826
H-Net-3BPT-70M
3.0
2.4115
2.0694
2.4815
2.4306
2.8113
2.5462
2.4357
Uniform / hierarchical compression baselines
U-Net-70M
128.0
2.8414
3.1335
3.4665
3.2389
3.0586
2.8415
2.6474
H-Net-128BPT-70M
122.2
2.7930
2.8940
3.4657
3.0241
3.0277
2.8270
2.6375
Table 1: Region-wise autoregressive perplexity (PPL) on the validation dataset. We omit dig (distal-intergenic) because it is weakly constrained and often heavily repeated, making PPL less informative. Bold indicates the best (lowest) PPL among encoder-compression methods (U-Net, H-Net, GeneZip). Underline indicates the best PPL of small model trained at 12.8K length.
Figure 4: Redundancy-aware compression across TE-derived and tandem-repeat sequence. Repeat positions are compared with matched non-repeat positions in the same full-window context. BPT ratio above 1 means stronger local compression; PPL details are in Table 9 . GeneZip converts repeat predictability into higher local compression, especially for LINE, human SINE, and TR, whereas H-Net often assigns equal or lower BPT to the same repetitive subsets.
Model
Time (min)
CMP SCC
eQTL AUROC
ETGP AUPRC
TISP PCC
Avg. rank
DNALongBench-official
HyenaDNA [ 4 ]
210
0.115
0.514
0.249
0.132
7.5
Caduceus-Ph [ 5 ]
435
0.133
0.566
0.380
0.109
5.5
Caduceus-PS [ 5 ]
891
0.127
0.538
0.376
0.108
6.8
External-pretraining
JanusDNA [ 6 ]
2520
0.159
0.840
0.438
0.002
3.5
Table 2: All-task summary on DNALongBench [ 3 ] . Time column reports the fine-tuning time of language models measured on the eQTL Artery_Tibial task. GeneZip-70M reports task-wise validation-selected RAR priors, with the fixed transcript-balanced prior reported in Table 10 . Avg. rank is computed over the four task metrics, with lower being better. Best task-metric results within Leakage-controlled-pretraining are in bold ; best results across all rows are underlined .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
12.8K pre-train
128K pre-train
Dataset
gencode_human_12.8k
gencode_human_128k
Region loss weight α
0.03
0.03
Max length (bp)
12,800
128,000
Per-device batch size
32
4
Grad. accumulation
8
8
Update steps
30,600
24,414
Appendix
Table 3: Pre-training hyperparameters for the GeneZip models.
Model
Time
eQTL Datasets and Average (AUROC)
(min)
Artery
Adipose
Fibro
Muscle
Nerve
Skin (NSE)
Skin (SE)
Thyroid
Blood
Average
Task-specific baselines
Expert model [ 3 ]
-
0.741
0.736
0.639
0.621
0.683
0.710
0.700
0.612
0.689
0.681
CNN [ 3 ]
-
0.576
0.551
0.547
0.502
0.516
0.499
0.499
0.487
0.577
0.528
DNALongBench-official
HyenaDNA [ 3 ]
210
0.479
0.513
0.584
0.487
0.511
0.471
0.544
0.529
0.512
0.514
Appendix
Table 4: DNALongBench eQTL tasks (AUROC; higher is better). The best result in each dataset and average column is bolded , the second best is underlined . Training time reports the fine-tuning time of language models measured on the Artery_Tibial task.
Model
HFF
H1hESC
GM12878
IMR90
HCT116
Avg
SCC ↑
Corr ↑
SCC ↑
Corr ↑
SCC ↑
Corr ↑
SCC ↑
Corr ↑
SCC ↑
Corr ↑
SCC ↑
Corr ↑
Task-specific baselines
CNN-one-hot
0.0291
0.0571
0.0010
0.0297
0.0086
0.0321
0.0065
0.0302
0.0190
0.0203
0.0128
0.0339
External-pretraining
JanusDNA
0.1604
0.2125
0.1589
0.2099
0.1396
0.1872
0.1575
0.2066
0.1788
0.1993
0.1590
0.2031
NTv3-100M-pre
0.1433
0.1951
0.1371
0.1873
0.1009
0.1548
0.1344
0.1832
0.1646
0.1825
0.1361
0.1806
Appendix
Table 5: Contact map prediction on DNALongBench [ 3 ] . We report stratum-adjusted correlation coefficient (SCC) and Pearson correlation (Corr) across five cell lines, along with the average. Best results are in bold ; second best are underlined .
Model
FC
EC
ER
GC
PC
Avg
Task-specific baselines
Expert Model [ 3 ]
0.808
0.710
0.749
0.624
0.774
0.733
CNN [ 3 ]
0.029
0.038
0.043
0.037
0.066
0.042
DNALongBench-official
HyenaDNA [ 3 ]
0.138
0.124
0.118
0.112
0.168
0.132
Caduceus-Ph [ 3 ]
0.114
0.088
0.088
0.097
0.154
0.109
Appendix
Table 6: Transcription initiation signal prediction (TISP) on DNALongBench [ 3 ] . We report Pearson correlation (PCC) for FANTOM CAGE (FC), ENCODE CAGE (EC), ENCODE RAMPAGE (ER), GRO-cap (GC), and PRO-cap (PC), along with the average. Task-specific and DNALongBench-official baselines are copied from Cheng et al. [3] ; all other language-model rows follow the official TISP evaluation protocol with chromosome-level post-hoc evaluation. Best overall scores are in bold ; best non-expert scores are underlined .
Model
AUROC
AUPRC
Task-specific baselines
Expert Model [ 3 ]
0.926
0.407
CNN [ 3 ]
0.797
0.310
DNALongBench-official
HyenaDNA [ 3 ]
0.828
0.249
Caduceus-Ph [ 3 ]
0.826
0.380
Appendix
Table 7: AUROC and AUPRC for the enhancer-target gene prediction (ETGP) task. Higher is better. Best scores are in bold .
Species
Model
Observed BPT
BPT-ratio
PPL ↓
MicroErr ↓
Intergenic enrich
Promoter enrich
Genic enrich
Human
H-Net-BPT128
124.4 ± 4.4
0.972 ± 0.034
2.8699
-
1.001 ± 0.009
0.996 ± 0.138
0.970 ± 0.232
GeneZip-12.8K-intergenic-focused
113.7 ± 16.1
1.397 ± 0.326
2.8618
0.4969
1.034 ± 0.038
0.668 ± 0.236
0.680 ± 0.247
GeneZip-12.8K
189.2 ± 80.7
1.060 ± 0.240
2.8730
0.3372
0.922 ± 0.086
1.902 ± 1.041
1.593 ± 0.835
GeneZip-128K
198.8 ± 96.0
1.106 ± 0.263
2.8417
0.2676
0.891 ± 0.127
2.424 ± 1.872
1.581 ± 1.001
Mouse
H-Net-BPT128
120.7 ± 5.2
0.943 ± 0.040
3.5559
-
1.005 ± 0.011
0.978 ± 0.178
0.912 ± 0.226
GeneZip-12.8K-intergenic-focused
142.2 ± 18.4
1.538 ± 0.420
3.5489
0.4739
1.034 ± 0.041
0.621 ± 0.263
0.674 ± 0.253
Appendix
Table 8: Compressor analysis of encoder-only checkpoints on full-chromosome held-out blocks. Human uses contiguous windows from chr8/chr9/chr10 ; mouse uses contiguous windows from chr11/chr19/chr10 . BPT-ratio is reported as a diagnostic of global token budget; PPL and MicroErr best values are bold within each species. Region enrichment values are interpreted relative to each row’s region-allocation prior.
Species
Repeat subset
GeneZip repeat / bg / ratio
H-Net repeat / bg / ratio
GeneZip BPT ratio
H-Net BPT ratio
Human
DNA transposon
2.67 / 2.85 / 0.94
2.68 / 2.85 / 0.94
1.07
0.97
LINE
2.17 / 2.95 / 0.74
2.17 / 2.95 / 0.74
1.17
0.92
LTR
2.28 / 2.87 / 0.80
2.27 / 2.87 / 0.80
1.13
0.95
SINE
1.69 / 2.94 / 0.58
1.69 / 2.94 / 0.58
2.67
1.24
TR
1.97 / 2.87 / 0.69
2.02 / 2.86 / 0.71
1.56
0.79
Mouse
DNA transposon
2.97 / 3.58 / 0.83
2.98 / 3.57 / 0.83
2.00
1.18
Appendix
Table 9: Repeat-subset predictability details for the Figure 4 analysis. Each cell reports mean repeat-position PPL / matched-background PPL / PPL ratio, averaged per window in the original full-window context. PPL ratio below 1 means the named repeat subset is easier to predict than its matched background.
Figure 5: Case studies of region-adaptive compression. Region labels are induced from GENCODE gene models [ 8 ] (promoter/CDS/UTR/exon/intron/near-intergenic (NIG)), and the curves plot per-base boundary probability for H-Net [ 10 ] and GeneZip. We highlight two loci on chr9: a promoter-dense transcript locus (chr9:136,980,237–137,030,237; top) and an intron-dominated gene locus (chr9:37,530,000–37,580,000; bottom). Across both loci, GeneZip concentrates boundary mass in promoters and transcribed features, while suppressing diffuse boundaries in long intronic or intergenic spans.
Setting
μr
Biological allocation prior
eQTL
ETGP
CMP
TISP
AUROC
AUROC/AUPRC
SCC/Corr
PCC
Uniform RAR
(1,1,1,1,1,1,1)
no region-specific biological prior
0.8183
0.7931 / 0.2297
0.1182 / 0.1607
0.2349
Promoter-distal regulatory
(1,16,8,8,2,2,4)
promoter, intron, and near-gene intergenic resolution
0.7967
0.8204 / 0.4330
0.1324 / 0.1788
0.2785
Cis-regulatory focused
(1,16,2,4,2,2,4)
promoter, UTR, intron, and near-gene intergenic resolution
promoter/CDS resolution with moderate UTR/exon resolution
0.7558
0.8160 / 0.3334
0.1332 / 0.1791
0.2741
Appendix
Table 10: Region-ratio ablation over the RAR multipliers μr . The region order is (promoter, CDS, UTR, exon, intron, NIG, DIG). Smaller μr gives higher resolution in that region, whereas larger μr gives stronger compression. eQTL reports average AUROC across tissues, ETGP reports AUROC/AUPRC, CMP reports average SCC/Corr across five cell lines, and TISP reports average PCC across five transcription-initiation assays.
Model
Context
eQTL
ETGP
CMP
TISP
AUROC
AUROC/AUPRC
SCC/Corr
PCC
GeneZip-70M
12.8K
0.7558
0.8160 / 0.3334
0.1332 / 0.1791
0.2741
GeneZip-70M-128K
128K
0.7837
0.8354 / 0.4559
0.1360 / 0.1823
0.2993
Appendix
Table 11: Auxiliary context-length ablation on DNALongBench under the default transcript-balanced GeneZip prior. Both rows use 70M GeneZip checkpoints and matched downstream fine-tuning/evaluation protocols. eQTL reports average AUROC across nine tissues, ETGP reports AUROC/AUPRC, CMP reports average SCC/Corr across five cell lines, and TISP reports average PCC across five transcription-initiation assays.
Dataset
Split
Total BP (M)
N BP (M)
Promoter
CDS
UTR
Exon
Intron
NIG
DIG
GENCODE-Human-12.8k
train
12160.00
0.16
1815.54 (14.93%)
324.84 (2.67%)
296.16 (2.44%)
190.66 (1.57%)
6758.93 (55.58%)
2636.61 (21.68%)
137.26 (1.13%)
valid
640.00
0.01
96.56 (15.09%)
17.08 (2.67%)
15.51 (2.42%)
9.89 (1.55%)
355.94 (55.61%)
138.24 (21.60%)
6.79 (1.06%)
GENCODE-Human-128k
train
12672.00
0.63
1047.20 (8.26%)
223.82 (1.77%)
259.47 (2.05%)
174.46 (1.38%)
7546.70 (59.55%)
3266.07 (25.77%)
154.28 (1.22%)
valid
128.00
0.01
10.52 (8.22%)
2.23 (1.74%)
2.72 (2.12%)
1.79 (1.40%)
77.64 (60.65%)
32.21 (25.17%)
0.90 (0.70%)
Held-out (chr8, 9, chr10)
417.34
17.52
22.44 (5.38%)
3.42 (0.82%)
4.81 (1.15%)
5.11 (1.22%)
243.77 (58.41%)
117.50 (28.15%)
20.31 (4.87%)
Appendix
Table 12: Statistics of the GENCODE-derived pretraining corpora used in this work. Total BP is reported in Mbp. Region columns report Mbp with within-split percentages in parentheses under our 7-region taxonomy: promoter, coding sequence (CDS), UTR, exon, intron, near-intergenic (NIG), and distal-intergenic (DIG). N BP reports ambiguous bases ( N ) in Mbp. The test split corresponds to contiguous, non-sampled windows from held-out chromosomes ( chr8 , chr9 , chr10 ), while train / valid are sampled windows.
Model
BPT
eQTL AUROC
ETGP AUPRC
BPE
1.3947
0.7544
0.2540
MxDNA
3.0283
OOM
OOM
GeneZip
169.6499
0.8200
0.4464
Appendix
Table 13: Comparison with tokenization baselines on long-context downstream windows. BPT is evaluated on the eQTL Artery Tibial test set and computed as total bp divided by total tokens, using effective stage 2 router tokens for GeneZip. Downstream metrics follow Table 2 .
Long-context compression is essential for reducing the cost and latency of large language model inference. However, existing methods can fragment important evidence, require additional training or alignment, and often depend on the target model for effective compression. We introduce TopoCompress, a training-free and model-agnostic framework that compresses long contexts by selecting coherent semantic spans. TopoCompress first scores each span using dense and lexical query relevance together with semantic acceleration. It then constructs a hybrid graph that connects spans based on semantic similarity and sequential adjacency, and propagates the query-guided relevance scores over the graph. Across five long-context tasks-HotpotQA, 2WikiMQA, MuSiQue, Qasper, and MultiFieldQA-en-TopoCompress consistently outperforms strong compression baselines. Notably, TopoCompress achieves performance comparable to the strongest baseline while using a 4x smaller compression budget, and provides a 1.41x smaller compression time over the fastest baseline.
Dynamic vocabularies built on-the-fly via Lempel-Ziv-Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity. We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model. Our analysis reveals four key issues: (1) LZW's compression algorithm creates an implicit conflict between the input embedding and the output embedding, a tension that a single shared embedding cannot resolve; (2) naively applying RoPE to LZW-compressed sequences ignores that merged tokens can span multiple tokens; (3) merging certain tokens (notably digits) disproportionately hurts downstream accuracy; and (4) dynamic vocabularies introduce an inherent segmentation ambiguity that confounds standard evaluation. We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW codec supporting selective merges and multi-view segmentations. Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline at a similar compression ratio on general text. This represents a significant step toward lossless adaptive-vocabulary compression for language models.
Saibo Geng, Andrea Gentilini, Xinxian Ma +3
EPFL · Technical University of Munich · Zuse School ELIZA +1
In this paper, we study whether an off-the-shelf LLM can be adapted into a discrete, variable-length token compressor and decompressor for long-context processing. To this end, we design a self-expressive autoencoding framework that fine-tunes a pretrained LLM with lightweight LoRA adapters to map long texts into compact sequences of learned latent codes, termed Z-tokens, and to decode them back into natural language or task outputs. The resulting representation is content-adaptive: less predictable or information-dense segments can receive more Z-tokens, while redundant regions can be represented more compactly through a budget-aware length regularizer. Our method is evaluated on long-context datasets such as Wikipedia, CNN/DailyMail, HotpotQA, and QuALITY, showing that it preserves reconstruction quality and downstream performance while reducing effective context length, generation-stage memory usage, and end-to-end latency. This simple design supports both direct decoding from compressed contexts and autoregressive generation in the Z-token space, providing a practical interface for efficient long-context inference.