GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

Weicai Long; Yusen Hou; Junning Feng; Houcheng Su; Shuo Yang; Donglin Xie; Yanlin Zhang

q-bio.GNApr 7, 2026

GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

Authors: Weicai Long, Yusen Hou, Junning Feng, Houcheng Su, Shuo Yang, Donglin Xie, Yanlin Zhang

Organizations: Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Peking University

Abstract

Large Language Models (LLMs) are increasingly adopted as conversational assistants in genomics, where they are mainly used to reason over biological knowledge, annotations, and analysis outputs through natural language interfaces. However, existing benchmarks either focus on specialized DNA models trained for sequence prediction or evaluate biological knowledge using text-only questions, leaving the behavior of general-purpose LLMs when directly exposed to raw genome sequences underexplored. We introduce GenomeQA, a benchmark designed to provide a controlled evaluation setting for general-purpose LLMs on sequence-based genome inference tasks. GenomeQA comprises 5,200 samples drawn from multiple biological databases, with sequence lengths ranging from 6 to 1,000 base pairs (bp), spanning six task families: Enhancer and Promoter Identification, Splice Site Identification, Taxonomic Classification, Histone Mark Prediction, Transcription Factor Binding Site Prediction, and TF Motif Prediction. Across six frontier LLMs, we find that models consistently outperform random baselines and can exploit local sequence signals such as GC content and short motifs, while performance degrades on tasks that require more indirect or multi-step inference over sequence patterns. GenomeQA establishes a diagnostic benchmark for studying and improving the use of general-purpose LLMs on raw genomic sequences.

Figures & tables

Figure 1: Overview of GenomeQA. The pipeline consists of Data collection, processing, construction and LLM evaluation.

Task	Source	Seq. Len	Label	BCQ		MCQ
Task	Source	Seq. Len	Label	Counts	Avg. Len	Counts	Avg. Len
Enhancer and Promoter Identification	EPDnew,SCREEN	400	2	500	176	500	70
Splice Site Identification	NT downstream tasks	600	3	500	254	500	94
Taxonomic Classification	NCBI RefSeq	1000	3	500	402	500	146
Histone Mark Prediction	NT downstream tasks	1000	10	500	419	500	176
TFBS Prediction	ENCODE, JASPAR	100	20	500	61	500	31

Table 1: The statistics of GenomeQA suite, where Source indicates the data source, Seq. Len refs to the length of DNA sequence, and Label denotes the number of label sets. Avg. Len represents the average lengths of the questions. We report the number of tokens after tokenization using the Llama-4 tokenizer.

Figure 2: Distribution of Options in BCQ (a) and MCQ (b) of GenomeQA.

Model	Thinking	Enhancer and Promoter Identification	Splice Site Identification	Taxonomic Classification	Histone Mark Prediction	TFBS Prediction	TF Motif Prediction	Avg.
BCQ
Claude-Sonnet-4.5	✔	69.00	53.20	61.60	53.80	56.60	64.00	59.70
GPT-5.1	✔	70.60	54.40	64.60	53.20	55.80	66.00	60.77
Gemini-3-Pro	✔	67.80	50.00	78.80	57.20	59.80	84.00	66.27
Grok-4.1	✔	67.60	47.60	63.80	53.20	54.60	69.00	59.30
Llama-4	✘	55.40	50.00	57.00	52.60	51.60	73.00	56.60

Table 2: Overall results of LLMs on GenomeQA. Thinking denotes whether the model utilizes Chain-of-Thought reasoning. The table reports the classification accuracy (%) for each subtask and Avg. denotes the average accuracy. The best performance in each task is bolded.

Figure 3: Performance comparison of LLMs across six tasks. The scores represent the macro-average accuracy (%) derived from both BCQ and MCQ. The numerical annotations indicate the highest accuracy achieved for each task (dominated by Gemini-3-Pro).

model	Thinking	BCQ	MCQ
GPT-5.1	✔	60.77	52.30
GPT-5.1	✘	58.03	43.97
Qwen3-Max	✔	56.53	45.33
Qwen3-Max	✘	54.57	40.80

Table 3: Performance comparison of LLMs with and without the thinking process on GenomeQA. The table reports the accuracy percentages on BCQ and MCQ.

Model	BCQ		MCQ
Model	w/	w/o	w/	w/o
Claude-Sonnet-4.5	68.07	54.22	63.86	27.11
GPT-5.1	68.07	52.41	63.86	27.11
Gemini-3-Pro	74.70	60.24	67.47	44.58
Grok-4.1	61.45	54.82	58.43	27.11
Llama-4	58.43	50.00	37.95	24.10

Table 4: Impact of implicit target inference on TFBS prediction task. The table reports accuracy percentages where w/ denotes the target (CTCF) is explicitly named, allowing direct pattern recognition and w/o denotes the implied target setting.

Figure 4: Distribution of failed cases.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Examples	Note
BCQ	Does the Human DNA sequence {seq} contain any {target}?	{seq} represents a sequence, {target} is the query label
	Does this Human DNA sequence contain authentic {target}? Sequence: {seq}	Specially used in Splice-Site Prediction
	Consider the chromatin state in K562 cells. Is this Human DNA sequence located in an {state} region? Sequence: {seq}	Specially used in Histone Mark Prediction
	We are looking for a sequence bound by the master regulator of chromatin looping and insulation. Is this sequence a match? {seq}	Specially used in TFBS prediction
MCQ	Which of the following Human DNA sequences contains a {target}?	{target} is the query label
	There is an exception in this group. While three sequences contain {target1}, one actually harbors a {target2}. Can you find it?	{target1} and {target2} represent query labels

Appendix

Table 5: Question Templates in GenomeQA.

Figure 5: Label statistics for all tasks. The charts display the distribution of question types for BCQ and MCQ. (a)-(b): Enhancer and Promoter Identification. (c)-(d): Splice Site Identification. (e)-(f): Taxonomic Classification. (g)-(h): Histone Mark Prediction. (i)-(j): TFBS Prediction. (k)-(l): TF Motif Prediction.

Taxonomy	Species	NCBI Accession
Eukaryote	Home sapiens	GCF_000001405.40
	Mus musculus	GCF_000001635.27
	Sus scrofa	GCF_000003025.6
Prokaryote	Escherichia coli	GCF_000005845.2
	Staphylococcus aureus	GCF_000013425.1
	Bacillus subtilis	GCF_000009045.1

Appendix

Table 6: Species in Taxonomic Classification task.

Chrom. State	Selected Histone Marks
Open	H3K4me3, H3K9ac, H3K27ac
Close	H3K9me3, H3K27me3
Undefined	H3K27me3, H2AFZ, H3K4me1
Undefined	H3K4me2, H3K36me3, H4K20me1

Appendix

Table 7: Histone marks in the Histone Mark Prediction task. Chrom. State indicates the chromatin state.

3D Genome	Selected TFs
Related	CTCF
Unrelated	SP1, GABPA, E2F1, TEAD4, NRF1
Undefined	REST, EGR1, YY1, ZBTB33, FOXA1, JUN, MAX
Undefined	SRF, MEF2A, RELA, STAT1, NR3C1, POU5F1, GATA2

Appendix

Table 8: TFs in TFBS Prediction.

Figure 6: Motifs in TFBS Prediction.

Panel of Figure 7 — Figure 7: The Base Prompt used as a baseline. It contains only basic task descriptions and formatting instructions without domain-specific reasoning guidance.

Panel of Figure 8 — Figure 8: The Optimized Prompt designed for the LLM. It includes explicit role definition, domain constraints, a step-by-step reasoning protocol, and strict output formatting rules.

Model	BCQ (33 samples)
	Enhancer and Promoter Identification		Splice Site Identification		Taxonomic Classification		Histone Mark Prediction		TFBS Prediction		TF Motif Prediction		Avg.		No. Task Wins
	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	No. Task Wins
Claude-Sonnet-4.5	100.00	66.67	44.44	22.22	33.33	100.00	33.33	50.00	66.67	55.56	66.67	66.67	57.41	60.19	3
GPT-5.1	100.00	100.00	88.89	44.44	100.00	66.67	66.67	66.67	55.56	66.67	33.33	100.00	74.07	74.07	4
Gemini3-Pro	66.67	100.00	44.44	55.56	100.00	100.00	50.00	33.33	77.78	77.78	100.00	100.00	73.15	77.78	5
Grok-4.1	100.00	100.00	33.33	66.67	66.67	66.67	33.33	50.00	66.67	66.67	33.33	66.67	55.56	69.44	6

Appendix

Table 9: Ablation study on Binary Choice Questions. Performance comparison between the initial and refined prompts on the 33-sample BCQ development set. Base denotes the Base System Prompt and Opt. denotes the Optimized System Prompt. Avg. represents the average accuracy across all six tasks. No. Task Wins indicates the number of tasks where the optimized prompt achieved equal or higher accuracy than the base prompt. Bold values highlight instances where the optimized prompt resulted in improved or equivalent performance compared to the baseline.

Model	MCQ (57 samples)
	Enhancer and Promoter Identification		Splice Site Identification		Taxonomic Classification		Histone Mark Prediction		TFBS Prediction		TF Motif Prediction		Avg.		No. Task Wins
	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	Base	Opt.	No. Task Wins
Claude-Sonnet-4.5	66.67	77.78	44.44	33.33	77.78	88.89	40.00	66.67	44.44	66.67	83.33	83.33	59.44	69.44	5
GPT-5.1	44.44	77.78	11.11	22.22	66.67	66.67	40.00	53.33	44.44	55.56	100.00	83.33	51.11	59.81	5
Gemini3-Pro	66.67	55.56	22.22	44.44	100.00	88.89	40.00	46.67	22.22	44.44	100.00	100.00	58.52	63.33	4
Grok-4.1	44.44	44.44	33.33	22.22	55.56	44.44	33.33	40.00	77.78	66.67	100.00	100.00	57.41	52.96	3

Appendix

Table 10: Ablation study on Multiple Choice Questions. Performance comparison on the 57-sample MCQ development set. Base denotes the Base System Prompt and Opt. denotes the Optimized System Prompt. Avg. represents the average accuracy across all six tasks. No. Task Wins indicates the number of tasks where the optimized prompt achieved equal or higher accuracy than the base prompt. Bold formatting signifies that the optimized prompt achieved performance parity or improvement over the base prompt.

Task	BCQ	MCQ
Enhancer and Promoter Identification	51.40	25.80
Splice Site Identification	52.00	24.00
Taxonomic Classification	50.80	28.20
Histone Mark Prediction	49.60	23.00
TFBS Prediction	49.20	22.40
TF Motif Prediction	54.00	28.00

Appendix

Table 11: Empirical random baseline accuracy (%) across different tasks in GenomeQA.

Figure 9: Case Study. Examples of failure modes in GenomeQA along with input details and model responses. Green: Ground truth labels, specific input characteristics provided by the benchmark. Red: Incorrect reasoning, or fabricated evidence generated by the model.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts
Oct 8, 2026Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli +5Genomics Bioinformatics
University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark
Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks
Sep 20, 2026Jie Ying, Zhefan Wang, Zihong Chen +11LLM Evaluation On-Policy Distillation
Shanghai Artificial Intelligence Laboratory · Yazhouwan National Laboratory · Shanghai Innovation Institute +2
PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?
Apr 7, 2026Yusen Hou, Weicai Long, Haitao Hu +3Genomics LLM Evaluation
Hong Kong University of Science and Technology (Guangzhou)

Oct 8, 2026J/K move · Enter open · S save

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli, Andreas Bjerregaard+4

University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark

Genomics Bioinformatics Language Modeling Long-Context Language Modeling Knowledge Augmentation for Language Models

Open paper AskPDF arXiv

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100

\times

longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.

Abstract

LLM Evaluation

Explore similar work

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks

PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?