Toward Controlling Biology with Language:Offline Learning of Prompt-Conditioned Interventions for Cells, Organoids, and Biobots

Nam H. Le; Douglas Blackiston; Michael Levin; Josh Bongard

q-bio.QMSep 30, 2026

Toward Controlling Biology with Language:Offline Learning of Prompt-Conditioned Interventions for Cells, Organoids, and Biobots

Authors: Nam H. Le, Douglas Blackiston, Michael Levin, Josh Bongard

Organizations: University of Vermont · Tufts University, Allen Discovery Center

Abstract

Artificial intelligence increasingly serves as a natural-language interface to complex technical systems, letting people accomplish sophisticated tasks by describing what they want rather than specifying how to do it. Extending this interface to living systems is harder: unlike code or images, a biological intervention has no closed-form linguistic meaning, and the paired language-intervention-outcome data needed to learn such a mapping is expensive to collect, since each example requires its own wet-lab experiment. One way around this is to treat an existing archive of interventions and their already-observed outcomes as a fixed, offline dataset, and use a vision-language model to judge, without any new experiments, whether an archived outcome matches a natural-language description. But whether that judgment is reliable enough to train a language-to-intervention mapping on -- without new experiments and without human validation -- has remained unclear. Here we show that a natural-language interface for a living organism -- a xenobot, a synthetic multicellular construct with no nervous system -- can be learned entirely offline this way, using a vision-language model's own judgment as the sole training reward: an instruction is mapped to the intervention already on record as producing the described behavior. This mapping generalizes to entirely new instructions, evaluated against archive data withheld from training (80.0% held-out accuracy vs a $66.7% chance baseline, matching a network trained directly on ground-truth labels).

Figures & tables

Figure 1: Overview of the pipeline. (a) Tracked trajectory of a xenobot before stimulation (PRE), with a zoomed view of the construct. (b) The same trajectory, marking the moment the electrical stimulus is applied. (c) The trajectory continuing after stimulation (POST), showing the resulting change in behavior. (d) An example natural-language prompt. (e) The Prompt-to-Intervention (P2I) network: a frozen SBERT sentence embedding is passed through a small MLP that outputs a single predicted duration. (f) The predicted duration is matched to its nearest neighbor on the archive’s duration axis; the matched batch’s own intervention and trajectory, as in (b)–(c), are what a vision-language model scores against the prompt, and that score is the reward used to train P2I.

Figure 2: From raw video to trajectory, real batch. (a) A raw frame, cropped, with a stimulus-induced bubble chain near the dish edge. (b) Optical flow magnitude, one of three detection tiers used despite this noise – motion concentrates at the electrodes, with the true signal (bright point near center) far weaker. (c, d) Per-frame $x$ / $y$ position across the full PRE/DURING/POST protocol: a noisy PRE baseline settles into a distinct POST position – the behavior change the VLM judge assesses from the corresponding 2D trajectory.

Figure 3: The two ways to turn the precomputed VLM-score table into a training signal, for the same real prompt. (a) Discrete nearest-neighbor reward: a predicted duration snaps to its nearest real archive batch, and that batch’s raw score is read off directly as the reward – the argmin is discrete, so this route needs REINFORCE or CMA-ES. (b) Differentiable curve-fit target: the same real points, fit into one smooth curve and reduced to a single fixed target, so the error is an ordinary mean-squared error and the whole thing trains by backpropagation.

	Train	GT3	GT1	GT2
Backprop (reference model)	$\mathbf{89.1{\pm}1.0\%}$	$\mathbf{88.9{\pm}0.0\%}$	$\mathbf{81.1{\pm}5.6\%}$	$\mathbf{80.0{\pm}0.0\%}$
CMA-ES	$66.7{\pm}0.0\%$	$66.5{\pm}1.0\%$	$62.4{\pm}7.1\%$	$62.9{\pm}6.7\%$
REINFORCE	$69.4{\pm}10.2\%$	$69.4{\pm}10.2\%$	$66.9{\pm}9.0\%$	$66.9{\pm}8.7\%$

Table 1: Ground-truth accuracy, all four evaluation cells. 30 seeds for all three optimizers.

Figure 4: Ground-truth accuracy across the four evaluation cells for Backprop and REINFORCE, against chance and additional baselines (random, constant, untrained network, nearest-neighbor-trained; described in full in the Supplementary Material). Backprop retains the largest margin over chance at every cell, including GT2, the fully held-out one.

Figure 5: Untrained vs. trained, same held-out prompt, real archive trajectories. Top: motion_increase (“speed up”); middle: motion_reduction (“slower”); bottom: stop. Gold = PRE, colored = POST; triangle = start, square = end. Top and bottom rows show a clean wrong-vs-right contrast; the middle row’s untrained and trained predictions both land on the correct direction for this particular prompt (detail in the Supplementary Material), included for honesty rather than omitted for effect.

Figure 6: Net VLM score $s(b)$ per batch (jittered for visibility) against true, independently-tracked behavior label. Open circles: the 3 sign-disagreement batches.

Figure 7: All 33 fitted VLM-score-vs-duration curves $\hat{r}_{p}(d)$ (sigmoid regression, one curve per prompt, 90 points each), grouped by behavior category.

Figure 8: Null distribution of the best $p$ -value achievable by the full 112-way grid search across 200 random duration-label shuffles, against the real result.

Figure 9: Training loss (seed 42, log scale) for the VLM-fit target $d^{\dagger}(p)$ vs. the purely numerical, velocity-only target – same architecture, optimizer, and epochs. Both converge to a comparable near-zero loss; the numerical target is not harder to fit.

	Train	GT3	GT1	GT2
VLM reward (Table 1 )	$\mathbf{89.1{\pm}1.0\%}$	$\mathbf{88.9{\pm}0.0\%}$	$\mathbf{81.1{\pm}5.6\%}$	$\mathbf{80.0{\pm}0.0\%}$
Numerical, velocity-only	$88.9{\pm}0.0\%$	$66.7{\pm}0.0\%$	$79.6{\pm}1.7\%$	$59.1{\pm}2.3\%$

Table 2: Same architecture and optimizer (Backprop), 30 seeds; only the training target differs.

prompt intent	lands on true STOP	lands on true MILD-SLOWER
‘stop’ prompt	$\mathbf{97.8\%}$ (44/45)	$2.2\%$ (1/45)
‘motion_reduction’ prompt	$33.3\%$ (15/45)	$\mathbf{66.7\%}$ (30/45)

Table 3: Where each held-out prompt’s predicted duration lands, against the archive’s true sub-label (

n{=}45

each; full per-source breakdown in the Supplementary Material).

Figure 10: Real PRE/POST trajectories and measured velocities returned for three prompts, same reference model as Table 1 . Top to bottom: “speed up” ( $\hat{d}=74.0$ s, PRE $1.17$ px/s $\to$ POST $1.81$ px/s); “slow down” ( $\hat{d}=131.0$ s, PRE $1.09$ px/s $\to$ POST $0.30$ px/s); “stop” ( $\hat{d}=194.0$ s, PRE $2.84$ px/s $\to$ POST $0.15$ px/s, POST trajectory collapses to a single point).

Language Model-Based Control

Language model-based control uses language models to interpret goals, choose actions, and steer AI systems or physical processes.

See all 180 papers

Language Game: Talking to Non-Human Systems
May 5, 2026Yanbo Zhang, Michael Levin
Allen Discovery Center at Tufts University, Medford, MA, 02155, USA · Wyss Institute for Biologically Inspired Engineering at Harvard University, Boston, MA, 02115, USA
WetRobo: A Reproducible Robot Kit for Coding Agents in Biological Laboratories
Sep 16, 2026Yuna Oikawa, Kei Endo, Takanori Uzawa +5Robot Policy Adaptation Code Generation for Robotics
Department of Computational Biology and Medical Sciences, The University of Tokyo, Japan · RIKEN Pioneering Research Institute, Japan · New York University, USA +3
ProtoAct: Turning Wet-Lab Protocols into Embodied Robotic Actions
Aug 3, 2026Zhe Liu, Jiaming Gu, Zhaohui Du +7Robotics Laboratory Automation
Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China · Department of Computer Science and Engineering, East China University of Science and Technology, Shanghai, China · Department of Laboratory Medicine, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China +1

May 5, 2026J/K move · Enter open · S save

Language Game: Talking to Non-Human Systems

Yanbo Zhang, Michael Levin

Allen Discovery Center at Tufts University, Medford, MA, 02155, USA · Wyss Institute for Biologically Inspired Engineering at Harvard University, Boston, MA, 02115, USA

Open paper AskPDF arXiv

Language carries thought and coordination among humans but rarely reaches further along the spectrum of diverse intelligence. Yet non-neural systems -- from gene regulatory networks and microbial consortia to fungi -- are increasingly recognized as substrates of computation, decision-making and memory, making dialogue with non-human intelligence newly conceivable. Today such dialogue is attempted only by proxy: a large language model speaks on the system's behalf, so any intelligence on display originates from the model while the system itself remains silent. Here we ask whether the system can speak in its own voice. Following Wittgenstein, who located meaning in use, we treat communication as a game played with the system. Its internal dynamics are frozen as the nonlinear core of a reinforcement-learning policy, with only linear input and output interfaces trained. Through use and reward, the system's states and responses acquire meaning within the game, so playing becomes speaking. Because different architectures playing the same game optimize the same reward, their behaviors can all be read as pursuit of that reward; the game serves as a lingua franca across otherwise irreconcilable representations. Given a human prompt, a language model routes it to the game whose semantics best match it and designs an environmental state for which the desired action is the rational response, letting the system reply through its own behavior. Applied across diverse gene regulatory networks and reinforcement-learning tasks, the framework yields fluent dialogue without altering any system parameter, shows that well-trained agents of disparate origin converge on similar behavior, and reveals that specific GRN properties make a system easier or harder to talk with -- an inductive bias of the reservoir itself. Our framework opens a new route to conversing with any dynamical system on its own terms.

Abstract

Language Model-Based Control

Explore similar work

Language Game: Talking to Non-Human Systems

WetRobo: A Reproducible Robot Kit for Coding Agents in Biological Laboratories

ProtoAct: Turning Wet-Lab Protocols into Embodied Robotic Actions