Can AI Scientist Agents Learn from Lab-in-the-Loop Feedback? Evidence from Iterative Perturbation Discovery

Gilles Wainrib; Barbara Bodinier; Haitem Dakhli; Josep Monserrat; Almudena Espin Perez; Sabrina Carpentier; Roberta Codato; John Klein

cs.LGMar 27, 2026

Can AI Scientist Agents Learn from Lab-in-the-Loop Feedback? Evidence from Iterative Perturbation Discovery

Authors: Gilles Wainrib, Barbara Bodinier, Haitem Dakhli, Josep Monserrat, Almudena Espin Perez, Sabrina Carpentier, Roberta Codato, John Klein

Organizations: Owkin Inc.

Abstract

Recent work has questioned whether large language models (LLMs) can perform genuine in-context learning (ICL) for scientific experimental design, with prior studies suggesting that LLM-based agents exhibit no sensitivity to experimental feedback. We shed new light on this question by carrying out 800 independently replicated experiments on iterative perturbation discovery in Cell Painting high-content screening. We compare an LLM agent that iteratively updates its hypotheses using experimental feedback to a zero-shot baseline that relies solely on pretraining knowledge retrieval. Access to feedback yields a $+53.4\%$ increase in discoveries per feature on average ( $p = 0.003$ ). To test whether this improvement arises from genuine feedback-driven learning rather than prompt-induced recall of pretraining knowledge, we introduce a random feedback control in which hit/miss labels are permuted. Under this control, the performance gain disappears, indicating that the observed improvement depends on the structure of the feedback signal ( $+13.0$ hits, $p = 0.003$ ). We further examine how model capability affects feedback utilization. Upgrading from Claude Sonnet 4.5 to 4.6 reduces gene hallucination rates from ${\sim}33\%$ -- $45\%$ to ${\sim}3$ -- $9\%$ , converting a non-significant ICL effect ( $+0.8$ , $p = 0.32$ ) into a large and highly significant improvement ( $+11.0$ , $p=0.003$ ) for the best ICL strategy. These results suggest that effective in-context learning from experimental feedback emerges only once models reach a sufficient capability threshold.

Figures & tables

Method	F0	F10	F20	F30	F40	F50	F60	F70	F80	F90	All
Random	18.3 $\pm$ 3.7	15.1 $\pm$ 4.2	12.8 $\pm$ 3.0	11.9 $\pm$ 1.9	10.7 $\pm$ 4.1	10.9 $\pm$ 2.8	11.8 $\pm$ 3.3	5.8 $\pm$ 3.0	8.7 $\pm$ 2.0	4.0 $\pm$ 1.5	11.0 $\pm$ 4.0
GP-UCB	27.1 $\pm$ 3.7	26.8 $\pm$ 4.3	23.9 $\pm$ 4.4	20.6 $\pm$ 4.6	18.9 $\pm$ 2.9	16.3 $\pm$ 2.9	17.6 $\pm$ 3.5	16.1 $\pm$ 3.8	17.3 $\pm$ 4.8	10.7 $\pm$ 2.2	19.5 $\pm$ 4.9
Zero-shot	27.0 $\pm$ 2.4	20.0 $\pm$ 4.0	25.2 $\pm$ 3.2	21.1 $\pm$ 2.5	20.1 $\pm$ 1.8	17.4 $\pm$ 3.5	18.0 $\pm$ 2.0	18.8 $\pm$ 4.8	21.1 $\pm$ 3.7	12.1 $\pm$ 1.8	20.1 $\pm$ 3.9
ICL-EF	28.3 $\pm$ 5.2	27.6 $\pm$ 5.5	24.9 $\pm$ 4.2	18.4 $\pm$ 3.1	19.8 $\pm$ 6.5	18.4 $\pm$ 1.9	20.0 $\pm$ 3.3	16.3 $\pm$ 5.6	31.4 $\pm$ 6.5	12.9 $\pm$ 0.9	21.8 $\pm$ 5.6
Zero-shot	24.7 $\pm$ 2.5	26.4 $\pm$ 3.3	27.0 $\pm$ 3.6	21.8 $\pm$ 2.5	19.2 $\pm$ 2.0	18.0 $\pm$ 2.1	17.3 $\pm$ 2.3	19.7 $\pm$ 1.8	17.0 $\pm$ 4.4	12.8 $\pm$ 1.3	20.4 $\pm$ 4.3
Random FB	28.6 $\pm$ 5.8	24.2 $\pm$ 4.2	19.5 $\pm$ 6.0	22.4 $\pm$ 3.8	20.0 $\pm$ 3.8	17.6 $\pm$ 1.7	17.7 $\pm$ 2.5	13.5 $\pm$ 3.5	12.5 $\pm$ 3.9	7.6 $\pm$ 3.1	18.4 $\pm$ 5.8

Table 1: Cumulative discoveries across 10 target features (F0–F90). Per-feature columns report mean

\pm

std over 10 replicates. The “All” column reports the mean and std of the 10 per-feature means, capturing between-feature heterogeneity. Bold indicates best per feature. The Sonnet 4.5 panel shows that ICL-EF fails to produce significant gains compared to zero-shot (

+0.8

p=0.32

), while Sonnet 4.6 yields large ICL-EF effects (

+8.9

p=0.003

). Moreover, ICBR-EF exhibits marginal but consistent improvement over ICL-EF (

+2.1

p=0.006

Figure 1: Performance comparison of six methods across 10 target features (10 replicates per condition). Error bars show standard deviation. Random FB sits below the zero-shot agent, while ICL-EF and ICBR-EF dominate all baselines.

Figure 2: Cumulative discoveries over 10 iterations. ICL-EF progressively outpaces the zero-shot baseline as feedback accumulates.

Figure 3: ICBR-EF improvement vs. Random baseline. Values above 1.0 indicate the ICBR-EF agent consistently outperformed random selection, achieving up to a

6.0\times

improvement on F70.

Panel of Figure 4 — Figure 4: Effect of model capability on ICL and hallucination.

Figure 5: Decomposition of the learning effect into memory jogging (Random FB

-

Zero-shot, red) and genuine ICL (ICL-EF

-

Random FB, green) per feature with Sonnet 4.6. The spurious prompt length effect is negative on 6/10 features; genuine ICL is positive on all 10.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Family	It. 1	It. 2	It. 3	It. 4	It. 5	It. 6	It. 7	It. 8	It. 9	It. 10	Total
CHD	1/5	0/4	—	—	—	—	—	—	—	—	1/9
KMT2	1/4	0/1	—	—	—	—	—	—	—	—	1/5
TAF	1/6	1/14	—	0/3	—	—	—	—	—	—	2/23
POLR	1/3	1/11	1/10	—	—	—	—	—	—	—	3/24
MED	—	0/6	2/10	0/1	—	—	—	—	—	—	2/17
HDAC	0/7	—	—	—	0/4	—	—	—	—	—	0/11

Appendix

Table 2: Family exploitation and exhaustion across iterations for the ICL-EF agent (F0). Each cell shows hits/tested for that gene family at that iteration. Dashes indicate the family was not tested. Major early families (CHD, TAF, POLR, MED) are exhausted by iteration 7, forcing the agent to pivot to newly identified anchors (ZSCAN, KLK, PPP) at iterations 8–10. The table reports 26 hits that can be assigned to a gene family, 9 hits correspond to singleton genes (no family).

Family	It. 1	It. 2	It. 3	It. 4	It. 5	It. 6	It. 7	It. 8	It. 9	It. 10	Total
CHD	1/2	0/6	—	—	—	—	—	—	—	—	1/8
KMT2	1/3	0/2	—	—	—	—	—	—	—	—	1/5
TAF	—	—	1/3	1/16	0/4	—	—	—	—	—	2/23
POLR	—	—	—	—	1/12	2/12	—	—	—	—	3/24
MED	—	0/4	1/9	1/4	—	—	—	—	—	—	2/17
HDAC	0/3	0/5	—	0/3	—	—	—	—	—	—	0/11

Appendix

Table 3: Family exploitation across iterations for the ICBR-EF agent (F0). Each cell shows hits/tested. The agent performs shallow within-family probing before rapidly pivoting across gene families that share high-level phenotypic signatures (e.g., RNA-binding, essentiality, central dogma involvement), despite limited functional or mechanistic continuity. The table reports 20 hits that can be assigned to a gene family, 17 hits correspond to singleton genes (no family).

Pair	F0	F10	F20	F30	F40	F50	F60	F70	F80	F90	$p<0.01$
Random vs GP-UCB	0.004	0.004	0.002	0.002	0.008	0.012	0.008	0.002	0.008	0.002	9/10
Random vs Zero-shot (4.5)	0.004	0.076	0.002	0.002	0.002	0.004	0.012	0.002	0.002	0.002	8/10
Random vs ICL-EF (4.5)	0.004	0.002	0.002	0.004	0.006	0.002	0.002	0.006	0.002	0.002	10/10
Random vs Zero-shot (4.6)	0.008	0.004	0.002	0.002	0.002	0.002	0.004	0.002	0.004	0.002	10/10
Random vs Random FB	0.008	0.006	0.006	0.002	0.004	0.002	0.010	0.006	0.074	0.010	9/10
Random vs ICL-EF (4.6)	0.002	0.002	0.002	0.002	0.002	0.002	0.002	0.002	0.002	0.002	10/10

Appendix

Table 4: Per-feature paired sign-flip permutation p-values. Bold:

p<0.01

Pair	Obs. diff	99% CI
Random vs GP-UCB	-8.530	[-10.610, -6.580]
Random vs Zero-shot (4.5)	-9.080	[-11.540, -6.640]
Random vs ICL-EF (4.5)	-10.800	[-15.270, -7.800]
Random vs Zero-shot (4.6)	-9.390	[-12.000, -7.050]
Random vs Random FB	-7.360	[-9.650, -5.070]
Random vs ICL-EF (4.6)	-18.330	[-25.090, -12.610]

Appendix

Table 5: Hierarchical bootstrap comparison (sorted by

p

-value). Bold rows: 99% CI excludes 0.

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 287 papers

Prior or Feedback? What an LLM Uses When Adapting Neural Operators
Oct 8, 2026Julian Chan, Javier Mora JimenezPartial Differential Equations LLM Decision-Making
Compare the Market
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
Bayesian Model Discovery with Language Models: Closing the Loop between Hypotheses and Experiments
Oct 19, 2026JooKyung Yuh, Byungmin Park, Eunseo Choi +6Scientific Discovery Large Language Model-Guided Scientific Discovery
Korea University · KAIST · KETI

Oct 8, 2026J/K move · Enter open · S save

Prior or Feedback? What an LLM Uses When Adapting Neural Operators

Julian Chan, Javier Mora Jimenez

Compare the Market

Partial Differential Equations LLM Decision-Making Neural Operators Hyperparameter Optimization

Open paper AskPDF arXiv

Do LLM scientific agents rely only on their initial task context, or do they adapt their decisions in response to experimental feedback? We study this question in neural operator adaptation, where a large language model (LLM) selects fine-tuning configurations under a limited trial budget. Across transfers within and between partial differential equation (PDE) families, the LLM achieves lower held-out test nRMSE than random search and Bayesian optimisation in nearly every matched comparison. Endpoint performance alone cannot distinguish what happens, so we verify each attribution with controlled interventions. Before observing any validation score, the LLM's first configuration already ranks near the top of the corresponding random-search pool, indicating a useful initial bias. A complementary cold-start intervention shows that the selected base learning rate shifts with the PDE description. Once feedback becomes available, reassigning validation scores among evaluated configurations changes the next proposal in every case tested, whereas a value-preserving rewrite produces no comparable aggregate effect. These interventions establish that the LLM's decision-level actions respond to the given task and observed outcomes, showing that it combines a task-dependent prior with sensitivity to experimental feedback.

Abstract

AI Agents for Scientific Discovery

Explore similar work

Prior or Feedback? What an LLM Uses When Adapting Neural Operators

AI scientists produce results without reasoning scientifically

Bayesian Model Discovery with Language Models: Closing the Loop between Hypotheses and Experiments