PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?

Yusen Hou; Weicai Long; Haitao Hu; Houcheng Su; Junning Feng; Yanlin Zhang

cs.CLApr 7, 2026

PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?

Authors: Yusen Hou, Weicai Long, Haitao Hu, Houcheng Su, Junning Feng, Yanlin Zhang

Organizations: Hong Kong University of Science and Technology (Guangzhou)

Abstract

Bacteriophages, often referred to as the dark matter of the biosphere, play a critical role in regulating microbial ecosystems and in antibiotic alternatives. Thus, accurate interpretation of their genomes holds significant scientific and practical value. While general-purpose Large Language Models (LLMs) excel at understanding biological texts, their ability to directly interpret raw nucleotide sequences and perform biological reasoning remains underexplored. To address this, we introduce PhageBench, the first benchmark designed to evaluate phage genome understanding by mirroring the workflow of bioinformatics experts. The dataset contains 5,600 high-quality samples covering five core tasks across three stages: Screening, Quality Control, and Phenotype Annotation. Our evaluation of eight LLMs reveals that general-purpose reasoning models significantly outperform random baselines in phage contig identification and host prediction, demonstrating promising potential for genomic understanding. However, they exhibit significant limitations in complex reasoning tasks involving long-range dependencies and fine-grained functional localization. These findings highlight the necessity of developing next-generation models with enhanced reasoning capabilities for biological sequences.

Figures & tables

Figure 1: PhageBench challenges general-purpose LLMs to perform five phage analysis tasks, processing raw nucleotide sequence inputs to generate reasoning-based phenotypic predictions.

Figure 2: The overview of data construction of PhageBench. We filter raw sequences from public repositories, apply task-specific processing rules, and format the resulting samples into standardized instructions for evaluation. For detailed procedures for each task, please refer to Appendix B.1 .

Figure 3: The compositional structure of PhageBench. The sizes of the constructed instruction datasets are labeled below the horizontal lines, and the percentages on the pie charts represent the proportion of data within each major category.

Figure 4: Statistical characteristics of the genomic sequences within the PhageBench dataset. The left panel displays the distribution of sequence sources. The central panel illustrates the probability density of contig lengths. The right panel depicts the distribution of GC content.

Non-reasoning
Model	T1: Phage Identification	T2: Contamination	T3: Completeness	T4: Lifestyle	T5: Host Prediction	Avg.
GPT-4o-mini	50.00	50.08	24.50	48.40	27.75	40.15
LLaMA 4	53.25	49.58	27.10	54.50	28.58	42.60
Qwen3-235b	51.58	49.33	25.40	52.00	28.58	41.38
Reasoning
GPT-OSS-120b	50.08	46.58	24.20	49.50	41.00	42.27

Table 1: Model performance accuracy (%) on PhageBench tasks under zero shot CoT setting. Bold: the best performance. Underlined: the second performance. Avg.: the average across all tasks. T: task.

Figure 5: Performance heatmaps across sequence length bins for Tasks 1–4. Each cell represents the accuracy of a specific model on sequences within a particular length range.

Model	T1: Phage Identification		T2: Contamination		T3: Completeness		T4: Lifestyle		T5: Host Prediction		Average
	0-shot	$\Delta$	0-shot	$\Delta$	0-shot	$\Delta$	0-shot	$\Delta$	0-shot	$\Delta$	0-shot	$\Delta$
GPT-4o-mini	50.00	0.00	50.00	+0.08	27.90	$-$ 3.40	48.20	+0.20	26.75	+1.00	40.57	$-$ 0.42
LLaMA-4	51.75	+1.50	51.33	$-$ 1.75	25.60	+1.50	50.60	+3.90	27.83	+0.75	41.42	+1.18
Qwen3-235b	50.00	+1.58	50.42	$-$ 1.09	25.00	+0.40	50.60	+1.40	28.42	+0.16	40.89	+0.49
GPT-5.2	68.50	+2.25	51.42	+2.75	36.60	+6.50	50.50	+1.00	29.17	+6.08	47.24	+3.71
Gemini-3-flash	70.25	+0.58	56.42	+2.41	30.20	+1.90	58.20	$-$ 0.80	64.58	$-$ 2.08	55.93	+0.40

Table 2: Comparison of model accuracy (%) between zero-shot and zero-shot with CoT modes. Grey cells indicate the

\Delta

Acc_{CoT}-Acc_{0-shot}

, which is the improvement of CoT. Positive values indicate CoT improvement. Bold values denote the best 0-shot performance per task.

Figure 6: A case study of hallucinated reasoning in Lifestyle Classification. (A) While the model correctly identifies integrase as a key marker, it incorrectly asserts its absence in the sequence.(B) The ground truth annotation confirms the presence of integrase.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Question Template
Phage Contig Identification	Is the following nucleotide sequence derived from a bacteriophage genome?
Contamination Classification	This sequence is a phage-associated contig. Does it contain any DNA originating from a non-phage host genome?
Completeness Classification	What is the estimated genome completeness category of the following bacteriophage sequence?
Lifestyle Classification	What is the most likely lifestyle of the bacteriophage represented by the following genome sequence?
Host Prediction	Which host family/genus/order is the following bacteriophage most likely to infect?

Appendix

Table 3: The specific instructions designed for each task in PhageBench.

Figure 7: Detailed class and subclass distributions across the task 1–4. The pie charts demonstrate the strict balancing strategy applied to primary classes and quality tiers ensuring statistical fairness.

Figure 8: Detailed class distributions of task 5. The pie charts demonstrate the strict balancing strategy applied to fine-grained taxonomic categories.

Figure 9: Distribution of correct answer keys across all tasks. The bar charts verify that the correct options (A, B, C, D) are uniformly distributed, confirming the effectiveness of the option shuffling strategy in mitigating position bias.

Panel of Figure 10 — Figure 10: The zero shot with CoT prompt.

Panel of Figure 11 — Figure 11: The zero shot prompt.

Figure 12: Detailed performance analysis of LLMs on PhageBench tasks. (Top) Phage Identification results separated by binary classes and fine-grained biological sources, followed by Contamination Detection across varying contamination ratios; (Bottom) Completeness Estimation across quality tiers, Lifestyle Classification phenotypes, and Host Prediction at three taxonomic resolutions.

Figure 13: Case study of successful Phage Contig Identification.

Figure 14: Case study of failure in Contamination Detection.

Figure 15: Case study of structural hallucination in Completeness Estimation.

Figure 16: Case study of failure in Lifestyle Classification.

Figure 17: Case study of successful Host Prediction.

Genomics

Genomics uses computational and experimental methods to study genomes, genetic variation, gene regulation, and their links to traits and disease.

See all 125 papers

GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding
Apr 7, 2026Weicai Long, Yusen Hou, Junning Feng +4LLM Evaluation Genomics
Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Peking University
ViroBench: Benchmarking Nucleotide Foundation Models on Viral Genomics Tasks
May 25, 2026Dongxin Ye, Fang Hu, Han Hu +6Distribution Shift Benchmark Design
Shanghai Innovation Institute Shanghai, China · University of Electronic Science and Technology of China Chengdu, China · Fudan University Shanghai, China +7
Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks
Sep 20, 2026Jie Ying, Zhefan Wang, Zihong Chen +11LLM Evaluation On-Policy Distillation
Shanghai Artificial Intelligence Laboratory · Yazhouwan National Laboratory · Shanghai Innovation Institute +2

Apr 7, 2026J/K move · Enter open · S save

GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

Weicai Long, Yusen Hou, Junning Feng, Houcheng Su+3

Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · Peking University

LLM Evaluation Genomics Bioinformatics

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly adopted as conversational assistants in genomics, where they are mainly used to reason over biological knowledge, annotations, and analysis outputs through natural language interfaces. However, existing benchmarks either focus on specialized DNA models trained for sequence prediction or evaluate biological knowledge using text-only questions, leaving the behavior of general-purpose LLMs when directly exposed to raw genome sequences underexplored. We introduce GenomeQA, a benchmark designed to provide a controlled evaluation setting for general-purpose LLMs on sequence-based genome inference tasks. GenomeQA comprises 5,200 samples drawn from multiple biological databases, with sequence lengths ranging from 6 to 1,000 base pairs (bp), spanning six task families: Enhancer and Promoter Identification, Splice Site Identification, Taxonomic Classification, Histone Mark Prediction, Transcription Factor Binding Site Prediction, and TF Motif Prediction. Across six frontier LLMs, we find that models consistently outperform random baselines and can exploit local sequence signals such as GC content and short motifs, while performance degrades on tasks that require more indirect or multi-step inference over sequence patterns. GenomeQA establishes a diagnostic benchmark for studying and improving the use of general-purpose LLMs on raw genomic sequences.

Abstract

Genomics

Explore similar work

GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

ViroBench: Benchmarking Nucleotide Foundation Models on Viral Genomics Tasks

Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks