LLM Novice Uplift on Dual-Use, In Silico Biology Tasks

Chen Bo Calvin Zhang; Christina Q. Knight; Nicholas Kruus; Jason Hausenloy; Pedro Medeiros; Nathaniel Li; Aiden Kim; Yury Orlovskiy

Name	Max Time	# Tasks	# Participants/Task
Long-Form Virology (Separated)	13 hours	1	15
Agentic Bio-Capabilities Benchmark	5 hours	3	15
World Class Biology	6 hours	20	2
Virology Capabilities Test	1.5 hours	60	2
Human Pathogen Capabilities Test	1.5 hours	40	2
Molecular Biology Capabilities Test	1.5 hours	40	2

Appendix

Table 3: Summary of benchmark usage and participant assignment.

Metric	Type	LLM Effect	95% CI		$p$		$p_{\text{adj}}$
Chain-of-thought lists	Count	$0.223$ [	+0.185,	$0.260$ ]	$<$	$0.001$	$<$	$0.001$
Clarity z-score	Z-score	$-0.089$ [	-0.150,	$-0.023$ ]		$0.012$		$0.020$
Confidence	Prop. of responses	$0.333$ [	+0.118,	$0.515$ ]	$<$	$0.001$	$<$	$0.001$
Confusion	Prop. of responses	$0.123$ [	-0.061,	$0.303$ ]		$0.208$		$0.231$
Connector density	Prop. of tokens	$0.008$ [	+0.002,	$0.013$ ]		$0.010$		$0.018$
Domain term density	Prop. of tokens	$-0.005$ [	-0.008,	$-0.001$ ]		$0.005$		$0.010$	**

Appendix

Table 4: Overall model-estimated effects on qualitative metrics. Values are model-estimated, adjusted differences (

\text{LLM-Assisted}-\text{Control}

). Mixed models used for continuous values (z-scores and counts) and logistic models used for proportion values, averaged equally over questions within benchmarks and then across benchmarks. Two-sided p-values (

p

) and 95% confidence intervals (CIs) are from 2,000 Monte-Carlo draws. Benjamini-Hochberg FDR adjustment is applied within outcome to produce

p_{\text{adj}}

. The word "proportion" is abbreviated as "prop." Stars display two-sided significance:

*

,

**

, and

***

indicate

p_{\text{adj}}<0.05

,

p_{\text{adj}}<0.01

, and

p_{\text{adj}}<0.001

, respectively.

Table 8

Figure 4: Usage of the Deep Research Feature. Among tasks in the treatment group, 37.5% utilized the ’Deep Research’ feature, while the remaining 62.5% were completed without it.

Figure 5: Model Usage Distribution. The chart illustrates the percentage of total user messages handled by each available large language model. Gemini 2.5 Pro was the most frequently used model, accounting for 33.5% of all messages.

Figure 6: Most Useful LLM Assistance. After each task, we asked participants in the treatment group, “What was the LLM’s most useful form of assistance?” and in the control group, “What do you believe would have been the LLM’s most useful form of assistance?”. The chart shows the proportion of responses for each category.

Figure 7: Co-occurrence of Top LLM Assistance Benefits. Participants selected the four most useful types of assistance they experienced (treatment) or believed they would have experienced (control). The heatmaps illustrate the frequency of co-occurrence for every pair of benefits within those selections, shown separately for the control and treatment groups.

Figure 8: Distribution of Self-Reported Confidence by Group. The histogram shows the proportion of participants from the control and treatment groups at different levels of self-reported confidence. The treatment group’s responses are heavily skewed toward maximum confidence (90-100), whereas the control group’s confidence levels are more broadly distributed.

Figure 9: Distribution of Self-Reported Task Difficulty. This histogram compares difficulty ratings from the control and treatment groups. The control group’s responses are heavily skewed toward higher difficulty (8-10), while the treatment group’s ratings show a clear shift toward lower difficulty, suggesting the treatment reduced the perceived challenge of the tasks.

Figure 10: Performance on Long-Form Virology Tasks by Part. Mean scores of the control and treatment groups across the six parts of the LVF task. Error bars represent the standard errors. While performance was identical on Part 1, the treatment group scored significantly higher on Part 4, whereas the control group performed better on Parts 2, 3, and 5.

Figure 11: Cumulative Score Progression on the LFV Task. The graph shows the cumulative scores as participants progressed through the six task parts. Faint lines represent individual participant trajectories, while bold lines track the average for the control and treatment groups. On average, the control group consistently maintained a higher cumulative score than the treatment group throughout the task.

Figure 12: Performance on ABC-Bench Tasks. Average scores for the control and treatment groups on three questions in ABC-Bench. The treatment group significantly outperformed the control group on the ‘Fragment’ and ‘Robot’ tasks, with the largest improvement seen in the ‘Fragment’ task. Both groups failed to score on the ‘Evasion’ task. Error bars indicate standard errors.

Figure 13: Analysis of participant score and confidence on the VCT benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 14: Comparison of actual and estimated completion times for VCT tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 15: Analysis of participant score and confidence on the WCB benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 16: Comparison of actual and estimated completion times for WCB tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 17: Analysis of participant score and confidence on the MBCT benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 18: Comparison of actual and estimated completion times for MBCT tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 19: Analysis of participant score and confidence on the HPCT benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 20: Comparison of actual and estimated completion times for HPCT tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 21: Analysis of participant score and confidence on the LAB-Bench benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 22: Comparison of actual and estimated completion times for LAB-Bench tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 23: Analysis of participant score and confidence on the Humanity’s Last Exam benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 24: Comparison of actual and estimated completion times for Humanity’s Last Exam’s tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 25: Analysis of participant score and confidence on the Long-Form Virology benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 26: Comparison of actual and estimated completion times for Long-Form Virology tasks. The left panel displays the measured time (in minutes) across all 7 parts (one initial learning step and 6 subtasks) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 27: Analysis of participant score and confidence on the ABC-Bench (Fragment) benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 28: Comparison of actual and estimated completion times for ABC-Bench (Fragment) tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 29: Analysis of participant score and confidence on the ABC-Bench (Evasion) benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 30: Comparison of actual and estimated completion times for ABC-Bench (Evasion) tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Figure 31: Analysis of participant score and confidence on the ABC-Bench (Robot) benchmark. The top row shows task score and the bottom row shows self-reported confidence , comparing the Control (blue) and Treatment (red) groups. (a, c) Mean score and confidence (solid lines) with standard error of the mean (shaded regions) measured at regular intervals during the task. (b, d) Final submitted score and confidence for each participant, plotted against their final submission time. Dashed lines show linear regression fits, and marginal plots show the distributions for each variable.

Figure 32: Comparison of actual and estimated completion times for ABC-Bench (Robot) tasks. The left panel displays the measured time (in minutes) for participants in the Control and Treatment conditions. The right panel displays participants’ estimated completion time had they been assigned to the opposite experimental condition. The labels on the x-axis in the right panel refer to the participants’ original group.

Code	Definition	Inclusion Examples	Exclusion Criteria
Direct answer request	Participant explicitly seeks a final answer or solution (not reasoning) from an LLM, such as by pasting the question into the LLM.	“Asked Claude for ’what is the right…’ ”; “Prompted Gemini, ’Which option is correct?’ then picked that.” ; “Pasted in the answer choices.”	Performing own analysis, asking for explanations, brainstorming.
LLM ideation support	Participant leverages an LLM to generate or refine novel ideas, approaches, or experimental variants (brainstorming), beyond merely obtaining a direct answer.	“Asked GPT-4 to propose alternative Gibson-assembly overlaps; it suggested staggered 30 bp ends, which I adopted.”; “Claude brainstormed five biosensor reporter genes.”	Using an LLM solely to explain background material, direct answer retrieval, brainstorming done without an LLM.
LLM research	Participant queries an LLM to locate, confirm, or provide factual information, protocols, or references.	“Asked ChatGPT for the NEB buffer composition.” ; “Prompted Gemini for the latest WHO guidelines on BSL-3.” ; “Used Claude to verify Gibson overlap lengths.”	Creative ideation (LLM ideation support). Direct answer requests.
Sought LLM explanations	Participant explicitly requests or refers to an explanatory breakdown from an LLM to understand or validate a solution.	“Asked GPT-4 to explain its reasoning step-by-step.”; “After Claude broke down the pathway, I finally got why the inhibitor works.”	Explanations not sourced from an LLM. Paraphrasing model output without indicating a request for explanation.
Verification of LLM output	Participant explicitly states they cross-checked or validated LLM-provided information before using it.	“Double-checked GPT-4’s concentration with Sigma-Aldrich datasheet.”; “After Claude suggested primer X, I ran BLAST to confirm.”	Mere expressions of uncertainty without follow-up action; generic citations unlinked to verifying AI output.
LLM comparison uncertainty	Participant questions or expresses doubt about which LLM answer is most accurate or trustworthy, beyond general epistemic caution.	“ChatGPT might have hallucinated, so I’m not sure if this is required.”; “Models disagreed here, so I tried to pick the right one.”	Generic uncertainty about scientific content without mentioning LLM comparison.

Appendix

Table 38

Code	Definition	Inclusion Examples	Exclusion Criteria
Independent research	Participant performs research without using an LLM: web search, textbooks, papers, personal knowledge.	“Googled ’GFF3 format’ and read the EMBL tutorial.” ; “Checked PubMed for influenza NCR papers.”; “Used my lab notebook from last semester.”	Any LLM interaction (LLM research, LLM ideation support, etc.).
Independent explanation	Participant articulates their own reasoning or justification, showing how they arrived at a step or decision.	“I chose 30 bp overlaps because longer regions increase Gibson efficiency.”; “Picked HindIII since the insert lacks its site.”	Vague “I looked it up.” Protocol quotes without causal reasoning.
Protocol lookup	Participant consults an external formal protocol (paper, kit manual, online SOP) to guide their actions.	“Followed Addgene’s Gibson Assembly PDF.” ; “Used NEB’s HindIII digest table.”; “Pulled the CDC RT-qPCR protocol.”	Generic browsing or self-research without a named protocol.
Resource listing	Participant enumerates materials, tools, or sources used.	“Google Scholar, NCBI, ChatGPT.” ; “Used SnapGene plus a Sonnet model.”; “Consulted three papers and an R script.”	Rich explanatory content; mere mention of one resource inside a longer narrative.
Mechanism explanation	Participant explains how or why a biological or biochemical process works at the mechanistic level.	“Taq polymerase’s 5’ to 3’ activity adds A-overhangs enabling TA-cloning.”; “RNA-dependent RNA polymerase binds conserved NCRs to initiate transcription.”	Simple protocol steps, high-level logic, or outcome justification without mechanistic detail.

Appendix

Table 39

Code	Definition	Inclusion Examples	Exclusion Criteria
Proposal planning	Participant lays out a forward-looking experimental or methodological plan with concrete next steps, resources, or timeline.	“Split 5 kb insert into four PCR products, assemble with Gibson, transform DH5 $\alpha$ , screen on kan.” ; “Day 1: grow culture; Day 2: miniprep; Day 3: sequencing.”	Retrospective descriptions of completed steps; vague ideas without actionable next steps.
Major error correction	Participant identifies and corrects a conceptual or procedural flaw that would materially change the outcome.	“Realised I used EcoRI instead of EcoRV; switched enzymes.” ; “Fixed the fragment orientation—reverse-complemented segment 4.” ; “Changed incubation from 4 h to 60 min.”	Minor typos or unit fixes (Minor error correction).
Minor error correction	Participant fixes small, non-conceptual mistakes (formatting, units, typos) that don’t substantially alter the biology.	“Corrected ’ml’ to ’ $\mu$ L’.” ; “Removed an extra space in the FASTA header.” ; “Changed 10 $\mu$ to 1 $\mu$ as per recipe.”	Switching enzymes or re-ordering major steps (see Major error correction).
Lossy summary	Participant summarizes information but omits important details, introduces inaccuracies, or overly compresses the source.	“According to the review, the virus binds ACE2 [omits strain-specific affinity].” ; “Protocol says ’incubate overnight’ [original: 16 h at 30°C with shaking].”	Faithful or complete summaries that retain all key details.

Appendix

Table 40

Code	Definition	Inclusion Examples	Exclusion Criteria
Confidence	Explicit positive confidence or ease.	“Felt confident after double-checking.” ; “Pretty easy once I saw the table.”; “Smooth process overall.”	Neutral success statements (“it worked”).
Confusion	Expresses being confused, uncertain, or stuck.	“I’m not sure what this assay measures.” ; “Got stuck on segment 6 orientation.”; “This part confused me.”	General remarks that prompt was ambiguous without emotional tone.
Frustration	Voices irritation or annoyance.	“This was frustrating; nothing aligned.” ; “Got annoyed when the digest failed again.”; “Wasted two hours on buggy software.”	Neutral difficulty statements (“challenging but okay”).
Gratitude	Expresses gratitude, relief, or appreciation.	“Thank goodness for the model’s suggestions.” ; “Grateful that NEB has clear docs.”; “Really appreciate the LLM’s explanation.”	Generic praise without emotional tone (“good output”).
Overwhelm	Feels overwhelmed or overloaded.	“Too much information to process.” ; “Completely overwhelmed by the protocol length.”; “Couldn’t keep track of all the references.”	Simple “complex” or “detailed” without explicit overwhelm.

Appendix

Table 41

Metric	LLM Effect	$p_{\text{adj}}$
Chain-of-thought lists	$+0.223$	$<0.001$ ***
Word count	$+37.810$	$<0.001$ ***
Resource count	$+0.382$	$<0.001$ ***
Major error correction	$+0.372$	$0.010$ **
Confidence	$+0.333$	$<0.001$ ***
LLM refusal	$-0.019$	$0.711$

LLM Novice Uplift on Dual-Use, In Silico Biology Tasks

Abstract

Figures & tables

Appendix

AI-Assisted Scientific Research

Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity

DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design

Benchmark Name	Format
Long-Form Virology (Separated)	Agentic
Agentic Bio-Capabilities Benchmark	Agentic
World Class Biology	Short Response
Virology Capabilities Test	Multi-Select
Human Pathogen Capabilities Test	Multi-Select
Molecular Biology Capabilities Test	Multi-Select

Abstract

AI-Assisted Scientific Research

Explore similar work

Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity

DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design