When Do Biological Reasoning Models Use Their Biological Inputs?

Ada Fang; Nikitha Thoduguli; Lukas Fesser; Hanlin Zhang; Sham M. Kakade; Marinka Zitnik

cs.LGOct 1, 2026

When Do Biological Reasoning Models Use Their Biological Inputs?

Authors: Ada Fang, Nikitha Thoduguli, Lukas Fesser, Hanlin Zhang, Sham M. Kakade, Marinka Zitnik

Organizations: Harvard University · Massachusetts Institute of Technology

Abstract

Biological reasoning models use post-training to connect LLMs to biological foundation model representations and biological text. Their benchmark accuracy is taken as evidence that LLMs reason over these inputs. We test this assumption in six biological reasoning models across DNA, protein, and single-cell tasks. We perturb one biological input while holding the query and other inputs fixed, construct evidence conflicts that pair the foundation model representation of one genome, protein, or cell with the text of another, fit linear probes to the representations the language model receives, and analyze reasoning traces against the biological inputs. Evo2 and ESM3 contribute little to BioReason and BioReason-Pro performance on the evaluated tasks. Shuffling the DNA sequence barely changes BioReason disease prediction accuracy, and in evidence conflicts the two models follow the text in 97.9% and 99.7% of cases. Linear probes trained on the Evo2 and ESM3 representations predict the task targets, so these foundation models encode information relevant to the task, but provide limited overall performance improvement to BioReason and BioReason-Pro. In contrast, foundation model inputs contribute to ChatNT, Prot2Text-V2, and CellWhisperer performance, and differentially expressed genes in the gene sentence contribute to Cell2Sentence-Scale performance. Across SFT and RL checkpoints of BioReason-Pro and 42 BioReason checkpoints, increases in accuracy do not imply greater performance contributions from biological inputs. BioReason traces misstate nucleotide changes, while BioReason-Pro traces describe functions omitted from final predictions under evidence conflicts. We find that current post-training strategies do not ensure that foundation model representations contribute to task performance.

Figures & tables

Figure 1: Evaluating biological input usage in biological reasoning models across DNA, proteins and single-cell.

Model	Modality	Checkpoints	Input sources	Task	Evaluation data
BioReason ( Fallahpour et al., 2025 ) checkpoints from ( Fesser et al., 2026 )	Genomic sequence	SFT, RL ; 42 post-training checkpoints	Projected reference and variant Evo2 representations ( $\mathbf{Z}_{\mathrm{Evo2}}$ ); chromosome ( $\mathbf{Z}_{\mathrm{chr}}$ ), gene ( $\mathbf{Z}_{\mathrm{gene}}$ ), KEGG pathway ( $\mathbf{Z}_{\mathrm{pathway}}$ ), and network ( $\mathbf{Z}_{\mathrm{network}}$ ) text	Disease prediction	1,449 KEGG-derived queries over 708 genomes and 37 disease labels; 165 genome-dependent queries; 145 held-out queries for the auxiliary task evaluation
ChatNT ( de Almeida et al., 2025 )	Genomic sequence	–	Nucleotide Transformer representations ( $\mathbf{Z}_{\mathrm{NT}}$ )	Splice donor, splice acceptor, and TATA promoter prediction	4,578 ChatNT test queries from three binary tasks (2,000 splice donors, 2,000 splice acceptors, 578 TATA promoters; balanced, up to 1,000 per class)
BioReason-Pro ( Fallahpour et al., 2026 )	Protein sequence	SFT, RL	ESM3 ( $\mathbf{Z}_{\mathrm{ESM3}}$ ); GO-GPT ( $\mathbf{Z}_{\mathrm{GO\text{-}GPT}}$ ); InterPro ( $\mathbf{Z}_{\mathrm{InterPro}}$ ); organism text ( $\mathbf{Z}_{\mathrm{organism}}$ )	Protein function prediction	14,102 reviewed human proteins; 91-protein temporal holdout; 540 proteins across 140 InterPro families; 96 GO NOT pairs
Prot2Text-V2 ( Fei et al., 2025 )	Protein sequence	–	ESM2 ( $\mathbf{Z}_{\mathrm{ESM2}}$ ); protein name ( $\mathbf{Z}_{\mathrm{name}}$ ); taxon text ( $\mathbf{Z}_{\mathrm{organism}}$ )	Protein function description generation	3,917 proteins from test split; 632 same-taxon pairs (1,264 evidence conflicts between name and sequence)
Cell2Sentence-Scale ( Levine et al., 2024 )	Single-cell transcriptome	2B, 27B	Expression-ranked gene sentence ( $\mathbf{Z}_{\mathrm{cell}}$ )	Cell type annotation and rationale generation	Five atlases; 4,846 cells for annotation; 2,000 cells for rationale analysis
CellWhisperer ( Schaefer et al., 2025 )	Single-cell transcriptome	–	CellWhisperer transcriptome representation ( $\mathbf{Z}_{\mathrm{CW}}$ ); expression-ranked top- $k$ gene list prefixes ( $\mathbf{Z}_{\mathrm{cell}}^{(k)}$ ), when supplied	Cell type prediction and evidence conflict	Same 4,846 C2S-Scale cells (five atlases); 330 same-tissue pairs, with correct prediction from representation only (660 conflicts)

Table 1: Models, checkpoints, input sources, tasks, and evaluation data. Underlined checkpoints are used unless otherwise specified, – refers only one released checkpoint. BioReason checkpoints are from Fesser et al. (2026) , as the original study did not release checkpoints.

Figure 2: Reasoning model performance with representations of intact and shuffled sequences. a BioReason disease prediction accuracy across text conditions. b ChatNT accuracy on three native DNA tasks. c BioReason-Pro $F_{\max}$ across text conditions. d Prot2Text-V2 BioBERT BERTScore F1 with name and taxon text removed or provided. e C2S-Scale 27B accuracy with intact, reordered, or resampled gene sentences. f CellWhisperer $1-\text{perplexity quantile}$ before and after permuting counts across genes within each cell. Error bars are 95% CIs.

Figure 3: Evidence conflicts from unmodified queries. a BioReason: $\mathbf{Z}_{\mathrm{Evo2}}$ from $A$ , gene/pathways from $A$ or $B$ . b BioReason-Pro: $\mathbf{Z}_{\mathrm{ESM3}}$ from $A$ , GO-GPT/InterPro from $A$ or $B$ . c Prot2Text-V2: $\mathbf{Z}_{\mathrm{ESM2}}$ from $A$ , protein name from B, with taxon fixed. d C2S-Scale 27B: $\mathbf{Z}_{\mathrm{cell}}$ from $A$ , cell type description from $B$ . e CellWhisperer: $\mathbf{Z}_{\mathrm{CW}}$ from $A$ , top-1,000 gene text from $B$ . Bars indicate preference for A, B, or neither. Error bars are 95% CIs.

Figure 6: Biological representations contribute similarly little to SFT and RL checkpoint performance. BioReason accuracy with original or perturbed $\mathbf{Z}_{\mathrm{Evo2}}$ across a biological text conditions and b for 42 post-training checkpoints on the 165 genome-dependent queries. c BioReason-Pro $F_{\max}$ with original or perturbed $\mathbf{Z}_{\mathrm{ESM3}}$ across biological text conditions.

Figure 7: Analysis of reasoning traces and rationales. The reference-to-variant base pair BioReason states in its trace on the 111 held-out substitution queries for the a RL and b SFT checkpoint. c Example BioReason-Pro reasoning trace for an evidence conflict. d Proportion of the genes in the C2S-Scale 27B rationale that are ribosomal protein genes and DEGs. Dots are the five single-cell atlases and error bars are 95% CIs.

Figure 8: Auxiliary sequence supervision for BioReason. a Disease prediction accuracy and b edited base pair accuracy on 145 held-out pathway-network split queries with

\mathbf{Z}_{\mathrm{Evo2}}

derived from original or shuffled DNA. Error bars the standard error over three training seeds.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Atlas	Accession	Cells	Types	Options	Label source
immune1	CELLxGENE 62ef75e4 ; E-MTAB-11536	30,423	35	34	Cell Ontology names
immune2	CELLxGENE e9360edf ; GSE271896	33,923	38	38	Cell Ontology names
immune3	CELLxGENE cc431242 ; GSE299043	30,593	31	31	Cell Ontology names
pancreas	GSE84133 (Baron et al. 2016)	5,641	14	14	study shorthand
lung	GSE280502 (Kim et al. 2024)	6,160	7	7	study shorthand

Appendix

Table S1: The five single-cell atlases. Shorthand labels are study names such as alpha , beta , Basal and Ciliated .

Label transition, intact $\rightarrow$ shuffled	RL	SFT
correct $\rightarrow$ correct	$1{,}192$ (82.3)	$1{,}094$ (75.5)
correct $\rightarrow$ incorrect	$35$ (2.4)	$99$ (6.8)
incorrect $\rightarrow$ correct	$36$ (2.5)	$86$ (5.9)
incorrect $\rightarrow$ incorrect, same label	$156$ (10.8)	$126$ (8.7)
incorrect $\rightarrow$ incorrect, other label	$30$ (2.1)	$44$ (3.0)
Accuracy, intact	0.847	0.823

Appendix

Table S2: BioReason prediction changes under DNA shuffling. Each of the 1,449 disease prediction queries falls in exactly one cell, by whether the label the scorer maps the generation to is correct with the input intact and after both Evo2 windows are shuffled. Counts, with per cent of 1,449 in brackets. Net is the arm’s accuracy minus the intact accuracy, which equals (incorrect

\rightarrow

correct

-

correct

\rightarrow

incorrect)

/1{,}449

. Re-decoding the identical prompt changes 0 of 1,449 labels at both checkpoints.

Quantity	Value
Proteins whose emitted GO set changes	$6{,}593$ (46.8%)
Proteins whose $F_{1}$ improves	$1{,}575$ (11.2%)
Proteins whose $F_{1}$ worsens	$1{,}527$ (10.8%)
Proteins whose $F_{1}$ is unchanged	$10{,}984$ (78.0%)
Mean Jaccard of the emitted GO term sets	$0.889$

Appendix

Table S3: Proteins whose BioReason-Pro output changes under ESM3 shuffling. The GO terms emitted from the intact input are compared with those emitted from the shuffled input, with the GO-GPT and InterPro text retained in both. The first row and the Jaccard are over all 14,102 proteins; the

F_{1}

rows are over the 14,086 with a scorable aspect, where

F_{1}

is the per-protein information-accretion-weighted

F_{1}

against that protein’s GOA annotation set propagated over the ontology.

Model	Condition (figure label)	Fields
BioReason		Chromosome	Network	Gene list	Gene in query
	all text ( wt )	✓	✓	✓	✓
	pathway fields, symbols masked ( no_gene )	✓	M	M (descriptions kept)	M
	query naming the gene ( no_pathway )	$\times$	$\times$	$\times$	✓
	query, gene symbol masked ( no_textkey )	$\times$	$\times$	$\times$	M
BioReason-Pro		GO-GPT	InterPro	Organism

Appendix

Table S4: Text fields present in each text condition. ✓ present,

\times

removed, M masked to GENE .

Figure S1: Evidence conflicts for BioReason-Pro (RL) on GO NOT pairs.

\mathbf{Z}_{\mathrm{ESM3}}

from

A

, GO-GPT/InterPro from

A

B

, where one protein of each pair has an experimental annotation to a GO term and the other an experimental NOT annotation to the same term. 106 conflicts from 53 pairs. Bars indicate preference for

A

B

, or neither. Error bars are 95% CIs.

Figure S2: Post-training changes BioReason performance without consistently increasing the performance contribution from $\mathbf{Z}_{\mathrm{Evo2}}$ . a–c Accuracy on the genome-dependent subset and all 1,449 queries as SFT LoRA rank, RL LoRA rank, or RL duration is varied.

Figure S3: Genes referenced in C2S-Scale rationales, and how that set changes when DEGs are removed from the cell sentence. a Composition of referenced genes across five atlases. b DEG precision among the first

k

referenced genes. c Jaccard similarity to the unmodified set of referenced genes after removing the strongest DEGs or non-DEGs with similar expression from

\mathbf{Z}_{\mathrm{cell}}

. Error bars are 95% CIs.

Query	Genes	Ribo. (%)	DEG prec. (%)	Prec.@5 (%)	Acc.
original query	66.5	77.9	11.2 (16.2)	23.2 (16.1)	0.40
Request for marker genes	62.5	72.6	12.0 (16.4)	22.3 (16.3)	0.41
Request for differentially expressed genes	48.4	71.3	13.2 (16.3)	20.6 (16.4)	0.42
Request for marker genes with exclusions	61.6	69.9	13.3 (16.3)	21.8 (16.3)	0.42
Same as above, instruction before the sentence	65.8	74.9	11.6 (16.3)	23.9 (16.1)	0.40
Annotated cell type given, request for marker genes only	65.5	69.8	13.3 (16.2)	20.9 (16.1)	–

Appendix

Table S5: Variation on the rationale query and DEG precision of C2S-Scale 27B. Referenced genes per generation, ribosomal protein share, DEG precision against the annotated type (random draw in parentheses), precision among the first five referenced genes (Prec.@5), and cell type accuracy, pooled over 200 cells (40 per atlas). Accuracy is undefined when the type is supplied in the prompt.

		Annotated type		Predicted type
Cells	$N$	Prec.@5	Prec.@all	Prec.@5	Prec.@all
Original prompt
All	2,000	21.6 (16.1)	10.6 (16.2)	–	–
Correct prediction	748	35.2 (19.9)	16.9 (20.0)	35.4 (20.6)	17.4 (20.7)
Incorrect prediction	1,252	13.4 (13.8)	0 6.9 (13.9)	–	–
predicted type in atlas	452	13.7 ( 0 9.4)	0 5.0 ( 0 9.5)	24.2 ( 0 9.9)	0 8.1 (10.1)

Appendix

Table S6: DEG precision of C2S-Scale 27B rationales by prediction correctness. Precision among the first five referenced genes (Prec.@5) and among all referenced genes (Prec.@all), in %, with the precision of a uniform random draw of the same number of genes from the cell sentence in parentheses. Annotated and predicted columns use the DEGs of the annotated and the predicted cell type. Pooled over five atlases. – marks rows that include cells whose label names no atlas type.

	Changed (%)		Accuracy		Cells
Atlas	Label	Type	w/o rationale	w/ rationale	Correct to incorrect	Incorrect to correct
immune1	21.5	9.8	0.71	0.61	49	7
immune2	62.5	44.0	0.12	0.08	27	8
immune3	16.0	9.8	0.53	0.44	37	3
pancreas	18.0	9.8	0.65	0.60	22	2
lung	34.5	18.0	0.26	0.34	5	37

Appendix

Table S7: Change in the C2S-Scale 27B predicted label when the rationale is requested. Share of cells whose label string or resolved Cell Ontology type differs between the two prompts, cell type accuracy without and with the rationale request, and the number of cells whose prediction changes from correct to incorrect and from incorrect to correct. 400 cells per atlas.

Held-out property	Distinct values	Also in training	Queries with the value in training
Prompt text	12	0	0 / 145
Network definition	9	0	0 / 145
Gene named in the query	8	7	139 / 145
Genome (both windows)	100	90	135 / 145
Disease label	8	8	145 / 145

Appendix

Table S8: Overlap between the 145 held-out queries and the 1,159 training queries.

	Disease		Edited base pair		Conflict
Training target, window	Intact	$\Delta$	Intact	$\Delta$	Donor’s pair
No auxiliary, 2,048 bp	0.901 (0.026)	0.002 (0.022)	0.080 (0.002)	0.005 (0.002)	0.071 (0.005)
Auxiliary, 257 bp	0.970 (0.009)	0.021 (0.007)	0.671 (0.041)	0.549 (0.032)	0.467 (0.072)
Auxiliary, 2,048 bp	0.984 (0.005)	0.028 (0.007)	0.531 (0.115)	0.432 (0.123)	0.361 (0.139)
Text-only reference	0.917		0.103		–

Appendix

Table S9: Accuracy on the 145 held-out queries, as the mean over three training seeds and the standard error of that mean.

\Delta

is intact minus shuffled. The conflict column is the share of queries whose written base pair equals the donor’s.

Training target, window	Unseen genomes	Seen genomes
No auxiliary, 2,048 bp	0.042	0.083
Auxiliary, 257 bp	0.148	0.691
Auxiliary, 2,048 bp	0.106	0.606
Most frequent training pair (T $>$ C)	0.111	0.101

Appendix

Table S10: Accuracy on the edited base pair for held-out and validation queries whose genome is absent from (72 queries) or present in (218 queries) the training set, as the mean over three training seeds.

Figure S4: Predicted reference-to-variant base pairs for the 257 bp auxiliary condition. a Original DNA, b shuffled DNA, and c an evidence conflict that replaces both windows with those of another held-out query. Rows are the query’s true pair in a and b and the donor’s pair in c . Each row is normalized by its own total, so a diagonal cell is the accuracy for that pair. The three training seeds are pooled, giving $145\times 3=435$ generations per matrix.

Explore similar work

How Post-Training Shapes Biological Reasoning Models
Jun 15, 2026Lukas Fesser, Hanlin Zhang, Michelle M. Li +5Large Reasoning Models Post-Training
Harvard University · Google DeepMind · Google Research
AMix-2: Establishing Protein as a Native Modality in Large Language Models
May 29, 2026Keyue Qiu, Yixin Wu, Lihao Wang +19Protein Language Modeling
1Shanghai Artificial Intelligence Laboratory · Institute for AI Industry Research (AIR), Tsinghua University
Plausibility Is Not Prediction: Contrastive Evidence for LLM-Based Cellular Perturbation Reasoning
May 31, 2026Xinyu Yuan, Xixian Liu, Jianan Zhao +3Single-Cell Perturbation Prediction Transcriptomics
1Mila - Québec AI Institute · University of Montréal · University of Ottawa +3

Jun 15, 2026, cs.LGJ/K move · Enter open · S save

How Post-Training Shapes Biological Reasoning Models

Lukas Fesser, Hanlin Zhang, Michelle M. Li, Eric Wang+4

Harvard University · Google DeepMind · Google Research

Large Reasoning Models Post-Training Improved Generalization Transcriptomics

Open paper AskPDF arXiv

Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are built through post-training, yet how each stage shapes reasoning and generalization remains poorly understood. We study when post-training improves performance and when it induces over-specialization. Across genomics, transcriptomics, and proteins, we train and evaluate more than 100 biological reasoning models under controlled variation in backbone, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL), measuring both in-domain (ID) and out-of-domain (OOD) performance. We find that each post-training stage reshapes generalization in a distinct way rather than contributing uniform gains. CPT improves downstream performance by aligning models with biological language. SFT consistently increases ID performance but causes OOD performance to peak early and decline as models fit the training distribution. RL, when applied to strong SFT checkpoints with aligned rewards, improves OOD performance and partially recovers generalization. These results show that biological reasoning does not improve monotonically with additional supervision or compute. Instead, performance depends on how training stages are composed. Under fixed post-training budgets, the strongest ID-OOD trade-off comes from brief SFT, larger RL allocations, and asymmetric adaptation capacity across stages.

Abstract

Explore similar work

How Post-Training Shapes Biological Reasoning Models

AMix-2: Establishing Protein as a Native Modality in Large Language Models

Plausibility Is Not Prediction: Contrastive Evidence for LLM-Based Cellular Perturbation Reasoning