GeneZip: Region-Aware Compression for Long Context DNA Modeling

Jianan Zhao; Xixian Liu; Zhihao Zhan; Xinyu Yuan; Hongyu Guo; Jian Tang

q-bio.GNFeb 19, 2026

GeneZip: Region-Aware Compression for Long Context DNA Modeling

Authors: Jianan Zhao, Xixian Liu, Zhihao Zhan, Xinyu Yuan, Hongyu Guo, Jian Tang

Organizations: Mila - Québec AI Institute · Université de Montréal · National Research Council of Canada · University of Ottawa ·

Abstract

Long-context DNA models are limited by token-mixing cost and by how compression allocates representational budget across the genome. Existing approaches operate close to base-pair resolution, apply fixed downsampling, or learn content-dependent chunks without an explicit genomic budget, making long-context pretraining expensive and difficult to control. We introduce GeneZip, a region-aware DNA compression framework that combines H-Net-style dynamic routing with a Region-Aware Ratio (RAR) objective and bounded routing. GeneZip uses static gene-structure annotations during compression training to specify region-wise base-pairs-per-token (BPT) targets; at inference time, it compresses raw unseen DNA without annotations. GeneZip provides three main benefits. First, it is effective: GeneZip variants achieve the best validation PPL among encoder-based compressors, with GeneZip-70M operating at 137.6 BPT, and across four reproducible DNALongBench tasks--contact map prediction, eQTL prediction, enhancer-target gene prediction, and transcription-initiation signal prediction--GeneZip obtains the best average rank among compared sequence models. Second, it is redundancy-aware: a post-hoc RepeatMasker/TRF analysis shows that, without repeat supervision, GeneZip assigns higher local BPT to TE-derived interspersed repeats and tandem repeats, two major classes of repetitive DNA sequence redundancy. Third, it is efficient: by reducing the effective token-mixing length, GeneZip enables longer-context and larger-capacity pretraining, including 128K-context and 636M-parameter variants on a single A100 80GB GPU, and fine-tunes the eQTL task 50.4x faster than JanusDNA (50 vs. 2520 minutes). These results establish GeneZip as an effective, redundancy-aware, and efficient compression interface for long-context DNA modeling.

Figures & tables

Figure 1: GeneZip overview. GeneZip compresses long-context DNA into a much shorter latent token sequence via hierarchical dynamic routing. During training, the Region-Aware Ratio (RAR) loss learns a region-adaptive token budget, and bounded routing enforces global token-count constraints for stable training. The compressed tokens are then processed by an arbitrary token-mixing backbone for downstream long-range tasks. After training, GeneZip compresses unseen raw DNA sequences without requiring annotations or repeat masks at inference time.

Figure 2: Why bounded routing is necessary. (a) In stage 1 compression, the router can be highly unstable early on and occasionally selects an excessively large number of tokens, which risks memory blow-up; a ceiling constraint prevents such pathological spikes. (b) In stage 2 compression, the router may collapse to selecting too few tokens (often near 1) at the beginning, which can induce training spikes and slow down optimization; a floor constraint enforces a minimum token budget for stable learning. (c) The floor constraint yields smoother and more favorable perplexity trajectories compared to training without it.

Figure 3: Inference efficiency on long-context inputs. End-to-end inference latency as a function of input sequence length (256 bp to 1M bp). GeneZip (70M/636M) remains consistently fast across the full range, while baseline long-context models exhibit sharply increasing latency as sequence length grows. The inset zooms into the short-to-mid regime (1K–131K) to highlight differences at smaller contexts.

Backbone / fine-grained upper bounds
Model	BPT	Overall $\downarrow$	Prom. $\downarrow$	CDS $\downarrow$	UTR $\downarrow$	Exon $\downarrow$	Intron $\downarrow$	NIG $\downarrow$
Mamba2-84M	1.0	2.4135	2.0494	2.4136	2.5608	2.6346	2.5076	2.4826
H-Net-3BPT-70M	3.0	2.4115	2.0694	2.4815	2.4306	2.8113	2.5462	2.4357
Uniform / hierarchical compression baselines
U-Net-70M	128.0	2.8414	3.1335	3.4665	3.2389	3.0586	2.8415	2.6474
H-Net-128BPT-70M	122.2	2.7930	2.8940	3.4657	3.0241	3.0277	2.8270	2.6375

Table 1: Region-wise autoregressive perplexity (PPL) on the validation dataset. We omit dig (distal-intergenic) because it is weakly constrained and often heavily repeated, making PPL less informative. Bold indicates the best (lowest) PPL among encoder-compression methods (U-Net, H-Net, GeneZip). Underline indicates the best PPL of small model trained at 12.8K length.

Figure 4: Redundancy-aware compression across TE-derived and tandem-repeat sequence. Repeat positions are compared with matched non-repeat positions in the same full-window context. BPT ratio above

1

means stronger local compression; PPL details are in Table 9 . GeneZip converts repeat predictability into higher local compression, especially for LINE, human SINE, and TR, whereas H-Net often assigns equal or lower BPT to the same repetitive subsets.

DNALongBench-official
Model	Time (min)	CMP SCC	eQTL AUROC	ETGP AUPRC	TISP PCC	Avg. rank
HyenaDNA [ 4 ]	210	0.115	0.514	0.249	0.132	7.5
Caduceus-Ph [ 5 ]	435	0.133	0.566	0.380	0.109	5.5
Caduceus-PS [ 5 ]	891	0.127	0.538	0.376	0.108	6.8
External-pretraining
JanusDNA [ 6 ]	2520	0.159	0.840	0.438	0.002	3.5

Table 2: All-task summary on DNALongBench [ 3 ] . Time column reports the fine-tuning time of language models measured on the eQTL Artery_Tibial task. GeneZip-70M reports task-wise validation-selected RAR priors, with the fixed transcript-balanced prior reported in Table 10 . Avg. rank is computed over the four task metrics, with lower being better. Best task-metric results within Leakage-controlled-pretraining are in bold ; best results across all rows are underlined .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	12.8K pre-train	128K pre-train
Dataset	gencode_human_12.8k	gencode_human_128k
Region loss weight $\alpha$	0.03	0.03
Max length (bp)	$12{,}800$	$128{,}000$
Per-device batch size	$32$	$4$
Grad. accumulation	$8$	$8$
Update steps	$30{,}600$	$24{,}414$

Appendix

Table 3: Pre-training hyperparameters for the GeneZip models.

Task-specific baselines
Model	Time	eQTL Datasets and Average (AUROC)
Model	(min)	Artery	Adipose	Fibro	Muscle	Nerve	Skin (NSE)	Skin (SE)	Thyroid	Blood	Average
Expert model [ 3 ]	-	0.741	0.736	0.639	0.621	0.683	0.710	0.700	0.612	0.689	0.681
CNN [ 3 ]	-	0.576	0.551	0.547	0.502	0.516	0.499	0.499	0.487	0.577	0.528
DNALongBench-official
HyenaDNA [ 3 ]	210	0.479	0.513	0.584	0.487	0.511	0.471	0.544	0.529	0.512	0.514

Appendix

Table 4: DNALongBench eQTL tasks (AUROC; higher is better). The best result in each dataset and average column is bolded , the second best is underlined . Training time reports the fine-tuning time of language models measured on the Artery_Tibial task.

Task-specific baselines
Model	HFF		H1hESC		GM12878		IMR90		HCT116		Avg
Model	SCC $\uparrow$	Corr $\uparrow$	SCC $\uparrow$	Corr $\uparrow$	SCC $\uparrow$	Corr $\uparrow$	SCC $\uparrow$	Corr $\uparrow$	SCC $\uparrow$	Corr $\uparrow$	SCC $\uparrow$	Corr $\uparrow$
CNN-one-hot	0.0291	0.0571	0.0010	0.0297	0.0086	0.0321	0.0065	0.0302	0.0190	0.0203	0.0128	0.0339
External-pretraining
JanusDNA	0.1604	0.2125	0.1589	0.2099	0.1396	0.1872	0.1575	0.2066	0.1788	0.1993	0.1590	0.2031
NTv3-100M-pre	0.1433	0.1951	0.1371	0.1873	0.1009	0.1548	0.1344	0.1832	0.1646	0.1825	0.1361	0.1806

Appendix

Table 5: Contact map prediction on DNALongBench [ 3 ] . We report stratum-adjusted correlation coefficient (SCC) and Pearson correlation (Corr) across five cell lines, along with the average. Best results are in bold ; second best are underlined .

Task-specific baselines
Model	FC	EC	ER	GC	PC	Avg
Expert Model [ 3 ]	0.808	0.710	0.749	0.624	0.774	0.733
CNN [ 3 ]	0.029	0.038	0.043	0.037	0.066	0.042
DNALongBench-official
HyenaDNA [ 3 ]	0.138	0.124	0.118	0.112	0.168	0.132
Caduceus-Ph [ 3 ]	0.114	0.088	0.088	0.097	0.154	0.109

Appendix

Table 6: Transcription initiation signal prediction (TISP) on DNALongBench [ 3 ] . We report Pearson correlation (PCC) for FANTOM CAGE (FC), ENCODE CAGE (EC), ENCODE RAMPAGE (ER), GRO-cap (GC), and PRO-cap (PC), along with the average. Task-specific and DNALongBench-official baselines are copied from Cheng et al. [3] ; all other language-model rows follow the official TISP evaluation protocol with chromosome-level post-hoc evaluation. Best overall scores are in bold ; best non-expert scores are underlined .

Task-specific baselines
Model	AUROC	AUPRC
Expert Model [ 3 ]	0.926	0.407
CNN [ 3 ]	0.797	0.310
DNALongBench-official
HyenaDNA [ 3 ]	0.828	0.249
Caduceus-Ph [ 3 ]	0.826	0.380

Appendix

Table 7: AUROC and AUPRC for the enhancer-target gene prediction (ETGP) task. Higher is better. Best scores are in bold .

Species	Model	Observed BPT	BPT-ratio	PPL $\downarrow$	MicroErr $\downarrow$	Intergenic enrich	Promoter enrich	Genic enrich
Human	H-Net-BPT128	124.4 $\pm$ 4.4	0.972 $\pm$ 0.034	2.8699	-	1.001 $\pm$ 0.009	0.996 $\pm$ 0.138	0.970 $\pm$ 0.232
	GeneZip-12.8K-intergenic-focused	113.7 $\pm$ 16.1	1.397 $\pm$ 0.326	2.8618	0.4969	1.034 $\pm$ 0.038	0.668 $\pm$ 0.236	0.680 $\pm$ 0.247
	GeneZip-12.8K	189.2 $\pm$ 80.7	1.060 $\pm$ 0.240	2.8730	0.3372	0.922 $\pm$ 0.086	1.902 $\pm$ 1.041	1.593 $\pm$ 0.835
	GeneZip-128K	198.8 $\pm$ 96.0	1.106 $\pm$ 0.263	2.8417	0.2676	0.891 $\pm$ 0.127	2.424 $\pm$ 1.872	1.581 $\pm$ 1.001
Mouse	H-Net-BPT128	120.7 $\pm$ 5.2	0.943 $\pm$ 0.040	3.5559	-	1.005 $\pm$ 0.011	0.978 $\pm$ 0.178	0.912 $\pm$ 0.226
	GeneZip-12.8K-intergenic-focused	142.2 $\pm$ 18.4	1.538 $\pm$ 0.420	3.5489	0.4739	1.034 $\pm$ 0.041	0.621 $\pm$ 0.263	0.674 $\pm$ 0.253

Appendix

Table 8: Compressor analysis of encoder-only checkpoints on full-chromosome held-out blocks. Human uses contiguous windows from chr8/chr9/chr10 ; mouse uses contiguous windows from chr11/chr19/chr10 . BPT-ratio is reported as a diagnostic of global token budget; PPL and MicroErr best values are bold within each species. Region enrichment values are interpreted relative to each row’s region-allocation prior.

Species	Repeat subset	GeneZip repeat / bg / ratio	H-Net repeat / bg / ratio	GeneZip BPT ratio	H-Net BPT ratio
Human	DNA transposon	2.67 / 2.85 / 0.94	2.68 / 2.85 / 0.94	1.07	0.97
	LINE	2.17 / 2.95 / 0.74	2.17 / 2.95 / 0.74	1.17	0.92
	LTR	2.28 / 2.87 / 0.80	2.27 / 2.87 / 0.80	1.13	0.95
	SINE	1.69 / 2.94 / 0.58	1.69 / 2.94 / 0.58	2.67	1.24
	TR	1.97 / 2.87 / 0.69	2.02 / 2.86 / 0.71	1.56	0.79
Mouse	DNA transposon	2.97 / 3.58 / 0.83	2.98 / 3.57 / 0.83	2.00	1.18

Appendix

Table 9: Repeat-subset predictability details for the Figure 4 analysis. Each cell reports mean repeat-position PPL / matched-background PPL / PPL ratio, averaged per window in the original full-window context. PPL ratio below

1

means the named repeat subset is easier to predict than its matched background.

Figure 5: Case studies of region-adaptive compression. Region labels are induced from GENCODE gene models [ 8 ] (promoter/CDS/UTR/exon/intron/near-intergenic (NIG)), and the curves plot per-base boundary probability for H-Net [ 10 ] and GeneZip. We highlight two loci on chr9: a promoter-dense transcript locus (chr9:136,980,237–137,030,237; top) and an intron-dominated gene locus (chr9:37,530,000–37,580,000; bottom). Across both loci, GeneZip concentrates boundary mass in promoters and transcribed features, while suppressing diffuse boundaries in long intronic or intergenic spans.

Setting	$\mu_{r}$	Biological allocation prior	eQTL	ETGP	CMP	TISP
			AUROC	AUROC/AUPRC	SCC/Corr	PCC
Uniform RAR	$(1,1,1,1,1,1,1)$	no region-specific biological prior	0.8183	0.7931 / 0.2297	0.1182 / 0.1607	0.2349
Promoter-distal regulatory	$(1,16,8,8,2,2,4)$	promoter, intron, and near-gene intergenic resolution	0.7967	0.8204 / 0.4330	0.1324 / 0.1788	0.2785
Cis-regulatory focused	$(1,16,2,4,2,2,4)$	promoter, UTR, intron, and near-gene intergenic resolution	0.8200	0.8196 / 0.3554	0.1282 / 0.1751	0.2764
Intergenic-focused	$(32,16,8,4,2,4,1)$	intergenic/noncoding resolution; intentionally coarser promoter resolution	0.7863	0.8225 / 0.4464	0.1280 / 0.1735	0.2238
Transcript-balanced	$(1,1,2,2,8,8,16)$	promoter/CDS resolution with moderate UTR/exon resolution	0.7558	0.8160 / 0.3334	0.1332 / 0.1791	0.2741

Appendix

Table 10: Region-ratio ablation over the RAR multipliers

\mu_{r}

. The region order is (promoter, CDS, UTR, exon, intron, NIG, DIG). Smaller

\mu_{r}

gives higher resolution in that region, whereas larger

\mu_{r}

gives stronger compression. eQTL reports average AUROC across tissues, ETGP reports AUROC/AUPRC, CMP reports average SCC/Corr across five cell lines, and TISP reports average PCC across five transcription-initiation assays.

Model	Context	eQTL	ETGP	CMP	TISP
		AUROC	AUROC/AUPRC	SCC/Corr	PCC
GeneZip-70M	12.8K	0.7558	0.8160 / 0.3334	0.1332 / 0.1791	0.2741
GeneZip-70M-128K	128K	0.7837	0.8354 / 0.4559	0.1360 / 0.1823	0.2993

Appendix

Table 11: Auxiliary context-length ablation on DNALongBench under the default transcript-balanced GeneZip prior. Both rows use 70M GeneZip checkpoints and matched downstream fine-tuning/evaluation protocols. eQTL reports average AUROC across nine tissues, ETGP reports AUROC/AUPRC, CMP reports average SCC/Corr across five cell lines, and TISP reports average PCC across five transcription-initiation assays.

Dataset	Split	Total BP (M)	N BP (M)	Promoter	CDS	UTR	Exon	Intron	NIG	DIG
GENCODE-Human-12.8k	train	12160.00	0.16	1815.54 (14.93%)	324.84 (2.67%)	296.16 (2.44%)	190.66 (1.57%)	6758.93 (55.58%)	2636.61 (21.68%)	137.26 (1.13%)
GENCODE-Human-12.8k	valid	640.00	0.01	96.56 (15.09%)	17.08 (2.67%)	15.51 (2.42%)	9.89 (1.55%)	355.94 (55.61%)	138.24 (21.60%)	6.79 (1.06%)
GENCODE-Human-128k	train	12672.00	0.63	1047.20 (8.26%)	223.82 (1.77%)	259.47 (2.05%)	174.46 (1.38%)	7546.70 (59.55%)	3266.07 (25.77%)	154.28 (1.22%)
GENCODE-Human-128k	valid	128.00	0.01	10.52 (8.22%)	2.23 (1.74%)	2.72 (2.12%)	1.79 (1.40%)	77.64 (60.65%)	32.21 (25.17%)	0.90 (0.70%)
Held-out (chr8, 9, chr10)		417.34	17.52	22.44 (5.38%)	3.42 (0.82%)	4.81 (1.15%)	5.11 (1.22%)	243.77 (58.41%)	117.50 (28.15%)	20.31 (4.87%)

Appendix

Table 12: Statistics of the GENCODE-derived pretraining corpora used in this work. Total BP is reported in Mbp. Region columns report Mbp with within-split percentages in parentheses under our 7-region taxonomy: promoter, coding sequence (CDS), UTR, exon, intron, near-intergenic (NIG), and distal-intergenic (DIG). N BP reports ambiguous bases ( N ) in Mbp. The test split corresponds to contiguous, non-sampled windows from held-out chromosomes ( chr8 , chr9 , chr10 ), while train / valid are sampled windows.

Model	BPT	eQTL AUROC	ETGP AUPRC
BPE	1.3947	0.7544	0.2540
MxDNA	3.0283	OOM	OOM
GeneZip	169.6499	0.8200	0.4464

Appendix

Table 13: Comparison with tokenization baselines on long-context downstream windows. BPT is evaluated on the eQTL Artery Tibial test set and computed as total bp divided by total tokens, using effective stage 2 router tokens for GeneZip. Downstream metrics follow Table 2 .

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 122 papers

TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
Aug 31, 2026Daniel Agyei Asante, Yang LiLLM Compression Long-Context QA
1Iowa State University, United States
zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss
Oct 9, 2026Saibo Geng, Andrea Gentilini, Xinxian Ma +3Token Compression LLM Compression
EPFL · Technical University of Munich · Zuse School ELIZA +1
Large Language Model as Token Compressor and Decompressor
Mar 26, 2026Wenbing Li, Yiran Wang, Zikai Song +4LLM Context Compression LLM Compression

Aug 31, 2026J/K move · Enter open · S save

TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories

Daniel Agyei Asante, Yang Li

1Iowa State University, United States

LLM Compression Long-Context QA LLM Inference Acceleration LLM Context Compression

Open paper AskPDF arXiv

Long-context compression is essential for reducing the cost and latency of large language model inference. However, existing methods can fragment important evidence, require additional training or alignment, and often depend on the target model for effective compression. We introduce TopoCompress, a training-free and model-agnostic framework that compresses long contexts by selecting coherent semantic spans. TopoCompress first scores each span using dense and lexical query relevance together with semantic acceleration. It then constructs a hybrid graph that connects spans based on semantic similarity and sequential adjacency, and propagates the query-guided relevance scores over the graph. Across five long-context tasks-HotpotQA, 2WikiMQA, MuSiQue, Qasper, and MultiFieldQA-en-TopoCompress consistently outperforms strong compression baselines. Notably, TopoCompress achieves performance comparable to the strongest baseline while using a 4x smaller compression budget, and provides a 1.41x smaller compression time over the fastest baseline.

Abstract

Model Compression

Explore similar work

TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories

zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss

Large Language Model as Token Compressor and Decompressor