Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

Shen Zhou Hong; Alex Kleinman; Alyssa Mathiowetz; Adam Howes; Julian Cohen; Suveer Ganta; Alex Letizia; Dora Liao

cs.CYFeb 18, 2026

Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

Authors: Shen Zhou Hong, Alex Kleinman, Alyssa Mathiowetz, Adam Howes, Julian Cohen, Suveer Ganta, Alex Letizia, Dora Liao,

Organizations: Active Site, Cambridge, MA 02142, United States · Independent · Model Evaluation and Threat Research, Inc., Covina, CA 91723, United States

Abstract

Large language models (LLMs) perform strongly on biological benchmarks, raising concerns that they may help novice actors acquire dual-use laboratory skills. Yet, whether this translates to improved human performance in the physical laboratory remains unclear. To address this, we conducted a pre-registered, investigator-blinded, randomized controlled trial (June-August 2025; n = 153) evaluating whether LLMs improve novice performance in tasks that collectively model a viral reverse genetics workflow. We observed no significant difference in the primary endpoint of workflow completion (5.2% LLM vs. 6.6% Internet; P = 0.759), nor in the success rate of individual tasks. However, the LLM arm had numerically higher success rates in four of the five tasks, most notably for the cell culture task (68.8% LLM vs. 55.3% Internet; P = 0.059). Post-hoc Bayesian modeling of pooled data estimates an approximate 1.4-fold increase (95% CrI 0.74-2.62) in success for a "typical" reverse genetics task under LLM assistance. Ordinal regression modelling suggests that participants in the LLM arm were more likely to progress through intermediate steps across all tasks (posterior probability of a positive effect: 81%-96%). Overall, mid-2025 LLMs did not substantially increase novice completion of complex laboratory procedures but were associated with a modest performance benefit. These results reveal a gap between in silico benchmarks and real-world utility, underscoring the need for physical-world validation of AI biosecurity assessments as model capabilities and user proficiency evolve.

Figures & tables

Figure 1: Trial Design. Schematic of the 8-week in-person study. Participants (n = 153) completed safety and LLM training prior to randomization and the start of laboratory work (Session 1). Participants completed a workflow consisting of a foundational skill assessment (Pre-task 1) followed by the core reverse genetics sequence (Tasks 2–4) and RNA quantification (Task 5). Baseline surveys were collected at Session 5; outcome measures (task completion) and tool utilization (chat logs, search history) were recorded continuously throughout the 39 sessions.

Panel of Figure 2 — Figure 2: (a) Baseline participant characteristics. Stacked bar charts display the distribution of education level, academic field, and prior biology experience across the full cohort. (b) CONSORT flow diagram illustrating participant allocation, attrition, and analysis sets. Of the 153 randomized participants (Full Analysis Set (FAS)), 128 (84%) met the attendance criteria ( $\geq 35$ sessions) for inclusion in the Per-Protocol Set (PPS).

Figure 3: Task Success Rates and Pooled Effect Estimates. Forest plot displaying success rates expressed as Risk Ratios (RR = LLM/INT). Black markers represent observed RRs with 95% confidence intervals (CI) calculated using the Koopman score method; P values are derived from one-sided Fisher’s exact tests. Purple markers represent posterior estimates from a hierarchical Bayesian logistic regression model, displaying posterior means, 95% credible intervals (CrI), and the posterior probability of a positive effect (

\Pr(RR)>1

). Shaded regions depict full posterior densities. Out-of-sample: Posterior distribution of the predicted RR for a hypothetical, out-of-sample reverse genetics task. The vertical dashed line at

x=1

indicates no effect.

Figure 4: Time-to-Completion per Task. Kaplan-Meier cumulative incidence curves illustrating the probability of successful completion over the study duration for the indicated task or outcome. The Internet arm is shown in pink (

n=76

) and the LLM arm in blue (

n=77

). Shaded regions denote 95% confidence intervals. Vertical dashed lines indicate the scheduled release date for each specific task, prior to which no attempts were permitted. Pink and blue values indicate numbers at risk, per arm.

Panel of Figure 5 — Figure 5: Progress through procedural steps is higher in the LLM arm. (a) Stepwise survival curves showing participant attrition at each procedural substep for Tasks 2–5. Solid lines represent observed completion rates for the Internet (pink) and LLM (blue) arms. (b) Bayesian ordinal regression estimates of LLM effects on progression. The Odds Ratio (OR) quantifies the increased likelihood of an LLM-arm participant reaching a more advanced procedural stage compared to an Internet-arm participant. Shaded regions depict full posterior densities; points and error bars indicate posterior means and 95% credible intervals (CrI).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Distribution of LLM usage. Bar chart displaying the total number of tokens generated by each participant in the LLM arm (n = 75), ordered by cumulative usage intensity. Two randomized participants (2/77) withdrew within 48 hours of study start without accessing the tools and are excluded from this visualization.

Panel of Figure 7 — Figure 7: Total Token Usage by LLM Family. Stacked segments represent the proportion of tokens generated by each model family: OpenAI (red), Anthropic (green), and Google DeepMind (blue). The distribution highlights substantial usage of advanced frontier models, with Gemini 2.5 (32.6%), Claude (25.7%), and OpenAI models (41.7% combined) accounting for the total volume. GPT-5 usage was introduced on week 5 of the study.

Figure 8: Sensitivity analysis of Bayesian model specifications. Posterior distributions of the Risk Ratio (RR) comparing an independent task-level model (pink), a hierarchical task-level model (orange), an independent participant-level model (green), and the primary hierarchical participant-level model (purple). Colored markers represent posterior estimates from a hierarchical Bayesian logistic regression model, displaying posterior means, 95% credible intervals (CrI), and the posterior probability of a positive effect (

\Pr(RR)>1

). Shaded regions depict full posterior densities.

Panel of Figure 9 — Figure 9: Participant progression through procedural steps of each task. A-D. Sankey diagrams illustrating participant flow through sequential milestones for Task 2 (a), Task 3 (b), Task 4 (c), and Task 5 (d). Ribbons represent the Internet arm (pink, n = 76) and LLM arm (blue, n = 77), with stream width proportional to the number of participants successfully completing each stage. Gray paths diverging downwards denote attrition (failure to complete the step or discontinuation of the task). Numbers within nodes indicate the count of participants who succeeded at each step. Detailed definitions of specific procedural steps are provided in Table 16 .

Figure 10: LLM usage patterns and participant perceptions. a , Frequency distribution of daily LLM prompt volume per participant (

n=77

LLM arm). Prompt counts were derived from automated scripts quantifying “user” messages. b , Frequency distribution of cumulative image uploads to LLM interfaces. Note: Analysis is restricted to OpenAI and Gemini platforms as Claude data lacked file export capabilities. c , Comparison of total internet search queries. Participants in the Internet arm executed a higher volume of traditional searches compared to the LLM arm. textbfd, Active engagement time with digital resources. Box plots of total hours spent interacting with LLM interfaces (left) and most commonly used websites (right) (**

P<0.001

, two-way ANOVA with multiple comparisons). textbfe, Resources most frequently cited as “helpful” in exit surveys. f , Longitudinal assessment of self-predicted performance. Bar chart of participants’ estimates of confidence in succeeding at tasks, speed ( “Time to Success” ), and efficiency ( “Attempts to Success” ) at week 2 (Baseline) and week 8 (study exit). Data represent mean

\pm

s.d. (

P<0.05

, two-way ANOVA with Šídák’s multiple comparisons test)

Panel of Figure 11 — Figure 11: Time to correct reagent submission for molecular cloning (Task 3). (a) Kaplan-Meier cumulative incidence curves showing the time to first reagent request submission for the Internet arm (pink) and LLM arm (blue). Shaded regions represent 95% confidence intervals (CI). (b) , Kaplan-Meier cumulative incidence curves showing the time to first correct reagent request submission. Shaded regions represent 95% confidence intervals (CI)

Internet Resource	Inclusion	Rationale for
(Examples)		Inclusion/Exclusion
Websites (Wikipedia, YouTube, ThermoFisher)	Yes	These resources represent standard and commonly accessible sources of online biology and laboratory knowledge, similar to what novices may access in a real-world setting. Image search engines may be used by participants to perform reverse lookups of unfamiliar reagents or equipment. Participants may utilize preprint servers, open-access databases, and open-access textbooks.
Search engines (Google, Bing, Duckduckgo)	Yes
Image search engines (Google Images, TinEye)	Yes
Public Databases (PubMed, Arxiv, Bioarxiv)	Yes
Educational Platforms (Coursera, Khanacademy Biology)	Yes

Appendix

Table 1: Definition of Access to the Internet

LLM-Based Resource	Inclusion	Rationale for
(Examples)		Inclusion/Exclusion
ChatGPT Family of Models , by OpenAI, inclusive but not limited to ChatGPT o3.	Yes	The three top-performing models on biology benchmarks such as ProtocolQA, Virology Capabilities Test (VCT) were selected for inclusion into the study. By using LLMs with proven capabilities in biology, the study standardizes the quality of AI assistance provided to participants
Claude Family of Models , by Anthropic, inclusive but not limited to Claude 4 Opus.	Yes
Gemini Family of Models , by Google DeepMind, inclusive but not limited to Gemini 2.5 Pro.	Yes
Other LLMs outside the above specification (e.g., Grok, Github Copilot, Reddit Answers)	No	In order to reduce variations in the strength of treatment offered, participants will be blocked from accessing LLMs outside of the above specification.

Appendix

Table 2: Definition of Access to LLMs

	Full Analysis Set (FAS)			Per Protocol Set (PPS)
	Internet	LLM	Total	Internet	LLM	Total
Randomized	76	77	153	64	64	128

Appendix

Table 3: Enrollment and Randomization

A. Full Analysis Set (FAS)	Internet	LLM	Balance Test
Age (years)	24.0 (11.5)	23.1 (8.6)	$t(133.81)=0.52,P=0.605,d=0.085$
Sex assigned at birth
Female	42 (55.3%)	41 (53.2%)	$P=0.837,V=0.047$
Male	31 (40.8%)	34 (44.2%)
Other	0 (0.0%)	0 (0.0%)
Missing	3 (3.9%)	2 (2.6%)

Appendix

Table 4: Demographics & Background

A. Full Analysis Set (FAS)	Internet	LLM	Balance Test
Educational attainment
Less than high school	0 (0.0%)	0 (0.0%)	$P=0.256,V=0.231$
High school diploma or GED	11 (14.5%)	11 (14.3%)
Some college ( $<1$ year)	0 (0.0%)	1 (1.3%)
Some college ( $\geq 1$ year), no degree	49 (64.5%)	45 (58.4%)
Associate degree	5 (6.6%)	2 (2.6%)

Appendix

Table 5: Baseline Educational Characteristics

A. Full Analysis Set (FAS)	Internet	LLM	Balance Test
BIO-EXP total (0–30)
Mean (SD)	4.4 (4.7)	5.0 (4.6)	$t(145.45)=-0.68,P=0.497,d=-0.112$
Median (IQR)	3.0 (7.0)	4.0 (6.5)
Pipetting subscore (0–6)
Mean (SD)	2.3 (2.2)	2.6 (2.1)	$t(145.55)=-0.73,P=0.467,d=-0.120$
Median (IQR)	2.0 (4.0)	2.0 (4.0)

Appendix

Table 6: Prior Biology Experience

A. Full Analysis Set (FAS)	Internet	LLM	Balance Test
LLM Tech Acceptance (TAM)
TAM overall			$t(145.98)=0.46,P=0.648,d=0.075$
Mean (SD)	3.9 (0.6)	3.8 (0.7)
Median (IQR)	3.9 (0.9)	3.9 (0.8)
Perceived Usefulness			$t(145.01)=0.80,P=0.424,d=0.132$
Mean (SD)	4.0 (0.8)	3.8 (0.9)

Appendix

Table 7: Self-Reported Measures

B. Per Protocol Set (PPS)	Internet	LLM	Balance Test
LLM Tech Acceptance (TAM)
TAM overall			$t(125.68)=0.15,P=0.879,d=0.027$
Mean (SD)	3.9 (0.7)	3.8 (0.7)
Median (IQR)	3.9 (1.0)	3.9 (0.9)
Perceived Usefulness			$t(124.30)=0.51,P=0.610,d=0.090$
Mean (SD)	4.0 (0.8)	3.9 (0.9)

Appendix

Table 19

	Effect Estimates (LLM Intervention)
Outcomes	Risk Diff. (95% CI)	Risk Ratio (95% CI)	Fisher’s Exact $P$
A. Full Analysis Set (FAS)
Primary Outcome (Tasks 2–4)	-1.4% (-9.9%, 6.9%)	0.790 (0.237, 2.624)	0.759
Pre-task 1. Micropipetting	4.2% (-8.6%, 16.9%)	1.054 (0.894, 1.249)	0.329
Task 2. Cell Culture	13.6% (-1.8%, 28.1%)	1.246 (0.971, 1.619)	0.059
Task 3. Molecular Cloning	-2.8% (-14.9, 9.2%)	0.846 (0.423, 1.685)	0.752

Appendix

Table 8: Primary and Secondary Outcomes

	Estimate ( $\bm{\pm}$ SE)
Model	elpd_loo	p_loo
A. Full Analysis Set (FAS)
Individual: hierarchical	-309.0 ( $\pm$ 15.6)	94.4 ( $\pm$ 5.5)
Individual: independent	-310.8 ( $\pm$ 15.9)	97.9 ( $\pm$ 5.7)
Task: hierarchical	-362.7 ( $\pm$ 15.9)	7.2 ( $\pm$ 0.5)
Task: independent	-364.9 ( $\pm$ 16.3)	10.0 ( $\pm$ 0.7)

Appendix

Table 9: LOO Summary

		RMST (95% CI)		Comparisons
Task	$\bm{\tau}$	INT	LLM	Diff. (95% CI)	$\bm{P}$
Pre-Task 1 Micropipetting	53.17	14.53 (9.54, 19.53)	11.28 (6.49, 16.07)	-3.25 (-10.17, 3.67)	0.36
Task 2 Cell Culture	53.18	39.30 (35.82, 42.78)	33.29 (29.65, 36.92)	-6.02 (-11.05, -0.98)	0.02*
Task 3 Molecular Cloning	44.38	41.80 (40.43, 43.16)	42.21 (40.86, 43.56)	0.41 (-1.51, 2.33)	0.68
Task 4 Virus Production	39.38	37.02 (35.38, 38.67)	35.61 (33.59, 37.63)	-1.41 (-4.01, 1.19)	0.29
Task 5 RNA Quantification	23.38	22.86 (22.32, 23.40)	22.09 (21.12, 23.06)	-0.77 (-1.88, 0.34)	0.18

Appendix

Table 10: RMST Analysis for Time-to-Success

	Median Time (95% CI)		Log-Rank Test
Task	INT	LLM	$\bm{P}$ (df=1, 1-sided)
Pre-Task 1 Micropipetting	3.07 (2.14, 7.06)	2.04 (1.14, 2.11)	0.016*
Task 2 Cell Culture	45.19 (35.11, N/A)	31.12 (21.11, 42.18)	0.012*
Task 3 Molecular Cloning	N/A	N/A	0.32
Task 4 Virus Production	N/A	N/A	0.13
Task 5 RNA Quantification	N/A	N/A	0.18

Appendix

Table 11: Median Time & Log-Rank Analysis

	Full Analysis Set (FAS)		Per Protocol Set (PPS)
Task	Internet	LLM	Internet	LLM
Pre-task 1. Micropipetting	8 [9]	5 [4]	8 [8]	4.5 [4]
Task 2. Cell Culture	4 [6]	3 [3]	4 [4]	3 [3]
Task 3. Molecular Cloning	2 [2]	2 [2]	2 [2]	2 [1.5]
Task 4. Virus Production	2 [3]	2 [2]	2 [2]	2 [2]
Task 5. RNA Quantification	2 [1]	2 [2]	2 [0]	2 [1.5]

Appendix

Table 12: Median Number of Attempts to Succeed per Task

	INT n/N	LLM n/N	Risk Diff.	Risk Ratio	Fisher’s
Task	(95% CI)	(95% CI)	(95% CI)	(95% CI)	Exact $\bm{P}$
A. Full Analysis Set (FAS)
Task 2	64.5% (53.3, 74.3%)	75.3% (64.6, 83.6%)	10.85% (-0.04, 0.25%)	1.17 (0.95, 1.46)	0.099
Task 3	23.7% (15.5, 34.4%)	28.6% (19.7, 39.5%)	4.89% (-0.09, 0.19%)	1.21 (0.71, 2.06)	0.307
Task 4	13.2% (7.3, 22.6%)	13.0% (7.2, 22.3%)	-0.17% (-0.11, 0.11%)	0.99 (0.44, 2.19)	0.607
Task 5	9.2% (4.5, 17.8%)	18.2% (11.2, 28.2%)	8.97% (-0.02, 0.20%)	1.97 (0.87, 4.55)	0.084

Appendix

Table 13: Per-task Milestone Achievement Rates

	Internet	LLM	Total N*
Raven’s 2 Standard score — mean (SD)	115.5 (12.8)	114.4 (12.2)	127

Appendix

Table 14: Non-Verbal Reasoning

Covariate (Full Analysis Set Data)	Main Effect OR (95% CI), P	Interaction ROR* (95% CI), P	LRT**	Stratified Effect: Internet, OR (95% CI), P	Stratified Effect: LLM
Prior Biology Experience	1.02 (0.72–1.43), P = 0.930	0.68 (0.34–1.36), P = 0.274	$P=0.272,\chi^{2}=1.21$	1.21 (0.75–1.95), P = 0.429	0.82 (0.50–1.36), P = 0.448
Self-Reported Prior LLM Experience	1.05 (0.75–1.48), P = 0.777	0.57 (0.28–1.16), P = 0.123	$P=0.119,\chi^{2}=2.43$	1.35 (0.84–2.17), P = 0.213	0.78 (0.46–1.31), P = 0.342
Non-Verbal Reasoning	1.91 (1.24–2.94), P = 0.003**	0.94 (0.39–2.27), P = 0.898	$P=0.898,\chi^{2}=0.02$	1.95 (1.11–3.44), P = 0.020*	1.84 (0.94–3.62), P = 0.075
Number of Tokens (LLM Only)	1.39 (0.71–2.73), P = 0.335	n/a	n/a	n/a	n/a
Number of Uploaded Images (LLM Only)	0.95 (0.59–1.52), P = 0.818	n/a	n/a	n/a	n/a
YouTube Searches	1.45 (0.96–2.18), P = 0.074	0.66 (0.27–1.65), P = 0.376	$P=0.380,\chi^{2}=0.77$	1.62 (0.99–2.67), P = 0.057	1.08 (0.52–2.30), P = 0.849

Appendix

Table 15: Subgroup Analyses

Step	Description of Milestone / Success Criteria
Task 2. Cell Culture
0	Never thawed
1	Self-reported task start
2	Thawed, but only got score 0 / invalid (no successful P1)
3	Successful P1 ( $\geq$ 70\text{,}\mathrm{%}$$ viability), but not P2 (milestone)
4	Successful P2 ( $\geq$ 85\text{,}\mathrm{%}$$ viability), but not P3

Appendix

Table 16: Procedural Steps for Tasks 2–5

	Internet		LLM
	Week 1	Week 8	Week 1	Week 8
A. Full Analysis Set (FAS)
Mean (SD)
Mental	5.2 (1.8)	5.2 (1.9)	5.3 (1.6)	5.0 (2.3)
Physical	3.4 (1.9)	3.8 (2.0)	3.3 (1.7)	3.3 (2.0)
Temporal	3.7 (1.6)	4.1 (1.9)	3.6 (1.6)	4.0 (2.2)

Appendix

Table 17: NASA-TLX

Description of Variant Classes

The following variant classes are not considered mismatches for the purpose of Task 3 (Molecular Cloning) quality criterion:

• Nanopore sequencing artifacts: – Deletions in homopolymer stretches, defined as a stretch of identical nucleotides greater than or equal to 8 base pairs. – Errors at the Dam methylation site GATC. – Errors at the middle position of the Dcm methylation site CCTGG or CCAGG. • Silent mutations: Synonymous mutations that do not alter the protein coding sequence. • Mutations in bacterial backbone regions: (e.g., origins of replication or antibiotic resistance) defined as the portion of the plasmid outside the mammalian promoters, terminators, coding regions, and viral genome.

Appendix

Table 18: Plasmid Assembly Variant Classes

Question: During the study, did you ever use an AI or large-language-model (LLM) tool (e.g., ChatGPT, Claude, Gemini, Copilot)?
Response	Internet Arm (n = 76)
No Data	7
No, Never	65
Yes, once	2
Yes, occasionally ( $\leq$ 25% of sessions)	2
Yes, often ( $>$ 25% of sessions)	0

Appendix

Table 19: Self-Reported Protocol Adherence on LLM Usage

AI-Assisted Scientific Research

AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.

See all 253 papers

LLM Novice Uplift on Dual-Use, In Silico Biology Tasks
Feb 26, 2026Chen Bo Calvin Zhang, Christina Q. Knight, Nicholas Kruus +16AI-Assisted Scientific Research LLM Safety Evaluation
Scale AI · University of Oxford · UC Berkeley +2
ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
Jun 9, 2026Andrew Bo Liu, Samira Nedungadi, Bryce Cai +3LLM Agent Security AI Agent Security Benchmarks
SecureBio, Cambridge, MA, USA · Active Site, Cambridge, MA, USA
An Early Warning of Emerging Biosecurity Risks in Frontier LLMs
Jul 20, 2026Zhida He, Xia Hu, Baichen Le +20Language Model Safety Evaluation LLM Security
Shanghai Artificial Intelligence Laboratory

Feb 26, 2026J/K move · Enter open · S save

LLM Novice Uplift on Dual-Use, In Silico Biology Tasks

Chen Bo Calvin Zhang, Christina Q. Knight, Nicholas Kruus, Jason Hausenloy+15

Scale AI · University of Oxford · UC Berkeley+2

AI-Assisted Scientific Research LLM Safety Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) perform increasingly well on biology benchmarks, but it remains unclear whether they uplift novice users -- i.e., enable humans to perform better than with internet-only resources. This uncertainty is central to understanding both scientific acceleration and dual-use risk. We conducted a multi-model, multi-benchmark human uplift study comparing novices with LLM access versus internet-only access across eight biosecurity-relevant task sets. Participants worked on complex problems with ample time (up to 13 hours for the most involved tasks). We found that LLM access provided substantial uplift: novices with LLMs were 4.16 times more accurate than controls (95% CI [2.63, 6.87]). On four benchmarks with available expert baselines (internet-only), novices with LLMs outperformed experts on three of them. Perhaps surprisingly, standalone LLMs often exceeded LLM-assisted novices, indicating that users were not eliciting the strongest available contributions from the LLMs. Most participants (89.6%) reported little difficulty obtaining dual-use-relevant information despite safeguards. Overall, LLMs substantially uplift novices on biological tasks previously reserved for trained practitioners, underscoring the need for sustained, interactive uplift evaluations alongside traditional benchmarks.

Abstract

AI-Assisted Scientific Research

Explore similar work

LLM Novice Uplift on Dual-Use, In Silico Biology Tasks

ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity

An Early Warning of Emerging Biosecurity Risks in Frontier LLMs