cs.AIOct 5, 2026
SaveFrom Benchmark to Bench: Can Agents Survive Real-World Drug Discovery?
Organizations: AMD Silo AI, Stockholm, Sweden · Molecular AI, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK · Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden · AMD Silo AI, Helsinki, Finland
Abstract
Agentic systems increasingly coordinate molecular-design tools, but it is unclear which layer of the stack limits outcomes on real projects. We developed MAGI, an open modular agent that authors objectives, launches and monitors optimization, interprets structure--activity relationships, and revises its strategy accordingly. MAGI generates molecules either directly through the LLM or by delegating to REINVENT 4, with scoring services interchangeable behind a common contract. We tested it across nine retrospective lead-optimization campaigns from three pharmaceutical companies, replayed under fixed temporal cutoffs. Both routes produced valid structures: LLM proposals stayed closer to local chemistry and reached comparable or higher primary activity in fewer operations, whereas REINVENT explored broader chemical space. Whether a campaign met its objective depended on the predictive models, not on the generation route: attainment followed model accuracy on the chemistry proposed, dropping once that chemistry moved outside the model's applicability domain. Separately, a blinded evaluation asked whether the MAGI's output could pass as expert work: chemists were not able to discriminate agentic proposals from held-out compounds, and judged the SAR reasoning broadly plausible yet incomplete. Together, these results position MAGI as a coordination layer pluggable into existing computational chemistry workflows. The ceiling on real projects, however, remains currently set by scorer applicability rather than by tool orchestration.
Figures & tables
Figure 1: Molecular quality and objective attainment across Drug Discovery campaigns. (a) Project-round validity, cumulative uniqueness, novelty, held-out similarity, and intra-round diversity for LLM (red) and REINVENT (blue). (b) GSK and AstraZeneca primary-endpoint distributions. (c) Sanofi project-level primary-endpoint distributions. Pale-green shading indicates the objective-satisfying region. Pale points denote pre-cutoff Reference compounds. Stars mark the selected starter compound. Brackets show Holm-adjusted two-sided Mann–Whitney U tests, adjusted p-value ( ) significance ( , , ; n.s., not significant).
Figure 2: Observable agent operations and correction/adaptation events. Counts are shown by category, campaign round, and generation route; counts exclude scorer and model calls executed internally within REINVENT’s reinforcement-learning loop.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: MAGI service interaction, MCP interface, and experimental protocol. (a) System architecture. The gateway agent connects the LLM reasoning backend to a monitored tool loop. Through the MCP server, it manages versioned therapeutic target profiles, optimization jobs, REINVENT 4, external data sources, and independently deployable scoring services. Shared storage transfers structures, configurations, and results between services without placing large artifacts in the language-model context. A TTP defines the scoring pipeline; molecules proposed by the LLM or generated by REINVENT are evaluated in batches using native molecular properties, Boltz-2 cofolding, pose and interaction analysis, global ADMET models and project-specific RetroDMTA models. Their component outputs are aggregated and returned to the generation loop to guide the next iteration. The MCP server exposes 24 tools across six groups: service discovery, data retrieval, TTP management, scoring and prediction, optimization control, and result retrieval. (b) Controlled campaign protocol. Project context, reference molecules, and objectives are encoded in a versioned TTP. LLM and REINVENT generate candidates through distinct routes but use the same activity, ADMET, Boltz-2, and interaction-scoring services. Ranked evidence guides subsequent rounds, after which final candidates are compared with measured held-out compounds never shown to the agents.
| Service | Model/version | Endpoints or outputs | Role | Applicability or quality controls |
|---|---|---|---|---|
| REINVENT 4 | REINVENT 4.8.24; Mol2Mol | Generated SMILES, component/aggregate reward | REINVENT molecular generation | Schema-validated TTP; RDKit pre-filters; diversity filter; agent-selected seeds and hyperparameters |
| General ADMET predictor | Consensus of XGBoost, Chemprop-MTL and CheMeleon-MTL | logP, , thermodynamcic/kinetic solubility, Caco-2 permeability, MDCK/Caco-2 efflux | ADMET endpoint scoring and optional secondary objectives | Applicability domain, Tanimoto similarity to proposal |
| RetroDMTA predictor | Target/window ensemble of XGBoost, Chemprop-MTL and CheMeleon-MTL | FXA /logS/ ; MMP8 ; PPAR ; RENIN | Project-specific ligand scoring | Target/window locked per campaign; cross-model SD |
| Boltz-2 | Boltz 2.2.1 | Predicted complex, predicted affinity and ligand iPTM | Cofolding and structural co-objective | RDKit ligand QC; supplied MSA; standard amino acids; 56 heavy and 128 total atoms |
| PoseBusters | PoseBusters 0.6.5 | Per-check booleans and fraction of checks passed | Physical-validity assessment of Boltz poses | Invalid/missing complexes return null |
| Interaction profiler | RDKit, OpenMM-based service | Typed contacts, interaction counts and fingerprints | Binding-mode interpretation | SMILES/structure validation; ligand-identity check; 8 Å radius |
Table S1: Agent-accessible services, model versions, outputs, roles, and applicability or quality controls.(MTL: Multi-Task Learning)
| Endpoint | Conditions | Datapoints | Minimum | Maximum |
|---|---|---|---|---|
| LogP | – | 56,140 | 7.0 | |
| PBS Buffer pH 7.4 Chromatography | 33,628 | 7.0 | ||
| Thermodynamic solubility | PBS Buffer Temperature C No excipients or cosolvents | 32,078 | 1.0 | |
| Kinetic solubility | Temperature C | 48,116 | ||
| Permeability Caco-2 | No engineered cell lines No P-gp inhibitors Below 100 µM concentration Below 20 cell passages Apical to basolateral transport | 9,600 | ||
| Efflux ratio Caco-2 | No engineered cell lines No P-gp inhibitors Below 100 µM concentration Below 20 cell passages | 2,007 | 2.0 |
Table S2: Global ADMET endpoint definitions, curated datapoint counts, and retained value ranges.
| Project | Endpoint | Set | N | Min | Max | Unit |
|---|---|---|---|---|---|---|
| GSK-S1 | Kinetic solubility | Reference | 5 | (mol/L) | ||
| GSK-S1 | Kinetic solubility | Held out | 31 | (mol/L) | ||
| GSK-S2 | Permeability | Reference | 5 | (cm/s) | ||
| GSK-S2 | Permeability | Held out | 46 | (cm/s) | ||
| GSK-S3 | Reference | 5 | 3.5 | 4.3 | ||
| GSK-S3 | Held out | 38 | 2.2 | 4.6 |
Table S3: Experimental endpoint composition by dataset and cohort. Ranges and medians exclude missing values; no missing endpoint is imputed.
| Target | UniProt | Construct aa | PDB | Chain | Ligand | Sources |
|---|---|---|---|---|---|---|
| FXA | P00742 | 232 | 9I24 | H | 0GJ | [ 40 , 41 , 42 , 43 , 44 , 45 ] |
| MMP8 | P22894 | 163 | 1I76 | A | BSI | [ 44 , 45 , 46 , 47 , 48 , 49 ] |
| PPAR | Q03181 | 272 | 5U3Q | A | 7UJ | [ 44 , 45 , 50 , 51 , 52 ] |
| RENIN | P00797 | 340 | 3K1W | A | BFX | [ 44 , 45 , 53 , 54 , 55 , 56 ] |
Table S4: Protein constructs and selected crystallographic reference complexes used for structure-conditioned evaluation.
Figure S2: Similarity between the held-out set and publicly available data (a) ECFP4 similarity to known drugs. For each FXa, MMP8, PPAR , and RENIN held-out compound, points show the maximum ECFP4 Tanimoto similarity to a standardized drugs structures extracted from SureChEMBL. (b) Interaction-fingerprint similarity to crystallographic ligands. For each FXa, MMP8, PPAR , and RENIN held-out compound, points show the maximum interaction fingerprint with Jaccard similarity to available experimental structures extracted from the PDB. Boxes show medians and interquartile ranges, with whiskers extending to 1.5 times the interquartile range.
| Endpoint | Methods | RMSE | MAE | |
|---|---|---|---|---|
| LogP | XGBoost | 0.771 | 0.855 | 0.605 |
| Chemprop-MTL | 0.803 | 0.793 | 0.540 | |
| Chemeleon-MTL | 0.792 | 0.815 | 0.578 | |
| Consensus | 0.826 | 0.744 | 0.509 | |
| log | XGBoost | 0.618 | 0.860 | 0.639 |
| Chemprop-MTL | 0.720 | 0.736 | 0.525 |
Table S5: Pooled out-of-fold performance on five shared Butina folds. Metrics are calculated in each endpoint’s native transformed unit; is unitless.
Figure S3: Validation of ligand-based models and Boltz-2 on experimental data. (a) Held-out validation of the ADMET models on the projects data. Scatter plots of the prediction from the consensus ADMET model to the GSK and FXA experimental ADMET data. Prediction are grouped by endpoints and colored by projects: GSK-S1 (light blue), GSK-S2 (purple), GSK-S3 (pink), GSK-S4 (orange), and FXA (green). (b) Predictive performance of Sanofi project models (top) and Boltz-2 affinity prediction (bottom) on the held-out set: experimental activity versus ligand-based model and Boltz-2 predictions for FXA, MMP8, PPAR and RENIN. Point are colored based on the target. (c) Agreement between ligand-based model and Boltz-2 affinity predictions on held-out Sanofi compounds: ligand-based model predictions versus Boltz-2 pX. (d) Post hoc sensitivity of proposal ranking to the Boltz-2 affinity term. Compounds were ranked with or without Boltz, evaluating ranking agreement between the two methods. Top 10 ranks with or without Boltz are covered by a grey area. Compounds with matching ranking are colored in green, and otherwise in red.
| Project | Requirement | Endpoint or constraint | Direction | Minimum acceptable value | Desirability saturation value | Unit |
|---|---|---|---|---|---|---|
| GSK-S1 | Primary objective | Kinetic solubility | Increase | – | (mol/L) | |
| GSK-S1 | Fixed guardrail | Permeability | Minimum | – | (cm/s) | |
| GSK-S1 | Fixed guardrail | Maximum | – | 3.5 | ||
| GSK-S1 | Fixed guardrail | Molecular weight | Maximum | – | 440.0 | g/mol |
| GSK-S2 | Primary objective | Permeability | Increase | – | (cm/s) | |
| GSK-S2 | Fixed guardrail | Aqueous solubility | Minimum | 150.0 | – | µM |
Table S6: Project objectives, guardrails, scoring directions, target bounds, and required evaluations. Upper desirability values for increasing potency objectives are saturation points rather than exclusion thresholds.
| Project | Pool records n | Valid distinct structures | Unique shortlisted | Shortlist construction | Cofolded structures | Cofold fraction |
|---|---|---|---|---|---|---|
| GSK-S1 | 96061 | 94363 | 25 | Agent-authored top five list | – | – |
| GSK-S2 | 132765 | 131030 | 25 | Agent-authored top five list | – | – |
| GSK-S3 | 200977 | 195351 | 20 | Agent-authored top five list | – | – |
| GSK-S4 | 204769 | 198243 | 25 | Agent-authored top five list | – | – |
| AZ | 163840 | – | 25 | Agent-authored top five list | – | – |
| FXA | 63455 | 63455 | 21 | Agent-authored final top five from transcript | 61260 | 0.965 |
Table S7: Project-level REINVENT generation pool, shortlist and cofold coverage.
| Project | Round | Invocation ID in round | Maximum steps | Batch size | Sigma | Seed structures n |
|---|---|---|---|---|---|---|
| GSK-S1 | 1 | 1 | 75 | 64 | 120 | 5 |
| GSK-S1 | 2 | 1 | 80 | 64 | 130 | 5 |
| GSK-S1 | 3 | 1 | 80 | 64 | 130 | 5 |
| GSK-S1 | 4 | 1 | 80 | 64 | 120 | 5 |
| GSK-S1 | 5 | 1 | 80 | 64 | 115 | 6 |
| GSK-S3 | 1 | 1 | 75 | 96 | 120 | 5 |
Table S8: Exact per-invocation of REINVENT parameters parsed from audited operation records.
Figure S4: Round-by-round similarity to held-out chemistry and intra-round similarity. Red boxes represent LLM proposals, and blue boxes represent final agent-selected REINVENT proposals during rounds 1–5. Columns separate GSK and Sanofi projects. For each molecule, the upper row reports its maximum ECFP4 Tanimoto similarity to any project-specific held-out compound. The lower row reports its mean pairwise ECFP4 Tanimoto similarity to other molecules from the same project, route, and round; lower values therefore indicate greater intra-round diversity. Boxes show medians and interquartile ranges; individual outliers are not shown. Brackets compare LLM and REINVENT molecule-level distributions within each round using two-sided Mann–Whitney U tests, with Holm correction across the five rounds separately for each organization and metric. n.s., adjusted ; for ; for ; for .
Figure S5: Best held-out compounds and top-ranked LLM and REINVENT proposals for the GSK projects. Best measured held-out molecule plus the top three LLM and top three REINVENT proposals per project.
Figure S6: Best held-out compounds and top-ranked LLM and REINVENT proposals for the Sanofi projects. Best measured held-out molecule plus the top three LLM and top three REINVENT proposals per target.
Figure S7: Top five LLM and REINVENT proposals for each GSK project. Columns correspond to GSK-S1–S4; rows show ranks 1–5 for LLM followed by ranks 1–5 for REINVENT. Structures are ranked in the favourable direction of the model-predicted primary endpoint across rounds 1–5 after canonical-SMILES deduplication. The predicted endpoint value and originating round are below each molecule.
Figure S8: Top five LLM and REINVENT proposals for each Sanofi project. Columns correspond to FXA, MMP8, PPAR and RENIN; rows show ranks 1–5 for each route. Ranking uses the project model for , or prediction in the favorable direction across rounds 1–5 after canonical-SMILES deduplication. Values and originating rounds are printed below each structure.
Figure S9: Structural quality of the reference held out and proposed compounds. (a) Synthetic accessibility (SA) scores across the eight projects. Gray, green, red, and blue denote Reference, Held-out, LLM and REINVENT molecules, respectively. (b) Structural quality of predicted complexes. Rows report the number of protein residues with a heavy atom within 4 Å of the ligand and ligand ipTM. Boxes show medians and interquartile ranges; whiskers extend to 1.5 times the interquartile ranges. Brackets compare Held-out–LLM, Held-out–REINVENT and LLM–REINVENT distributions using two-sided Mann–Whitney U tests. n.s., adjusted ; for ; for ; for .
Figure S10: UMAP chemical space coverage for the four GSK projects. Light-gray circles represent unique structures from the internal REINVENT pools, purple diamonds the five project Reference compounds, green circles Held-out compounds, red circles final LLM proposals and blue triangles final REINVENT proposals. Embeddings use 2,048-bit ECFP4 fingerprints and Jaccard distance. Each project is embedded independently. UMAP was computed with umap-learn [ 63 ] .
Figure S11: UMAP chemical space coverage for the four Sanofi projects. Light-gray circles represent unique structures from the internal REINVENT pools, purple diamonds the five project Reference compounds, green circles Held-out compounds, red circles final LLM proposals and blue triangles final REINVENT proposals. Embeddings use 2,048-bit ECFP4 fingerprints and Jaccard distance. Each project is embedded independently. UMAP was computed with umap-learn [ 63 ] .
Figure S12: Property distributions during rounds 1–5 of the GSK-S1 and GSK-S2 campaigns. Gray denotes the five shown Reference compounds, green for the published Held-out compounds, red LLM proposals and blue final agent-selected REINVENT proposals. Points represent individual molecules; boxes show the median and interquartile range, with whiskers extending 1.5 times the interquartile range. Each row identifies the endpoint and unit. Dashed green lines mark project-specific criteria where available and generic ADMET context ranges otherwise; pale-green shading shows the corresponding favorable region. Reference and Held-out values are study measurements, whereas proposal values are model predictions.
Figure S13: Property distributions during rounds 1–5 of the GSK-S3 and GSK-S4 campaigns. Gray denotes the five shown Reference compounds, green for the published Held-out compounds, red LLM proposals and blue final agent-selected REINVENT proposals. Points represent individual molecules; boxes show the median and interquartile range, with whiskers extending 1.5 times the interquartile range. Each row identifies the endpoint and unit. Dashed green lines mark project-specific criteria where available and generic ADMET context ranges otherwise; pale-green shading shows the corresponding favorable region. Reference and Held-out values are study measurements, whereas proposal values are model predictions.
Figure S14: Property distributions during rounds 1–5 of the AstraZeneca campaign. Gray denotes the five shown Reference compounds, green for the published Held-out compounds, red LLM proposals and blue final agent-selected REINVENT proposals. Points represent individual molecules; boxes show the median and interquartile range, with whiskers extending 1.5 times the interquartile range. Each row identifies the endpoint and unit. Reference and Held-out values are study measurements and are available for only; QED and rotatable-bond counts are RDKit descriptors computed on the proposals. Dashed green lines mark the campaign criteria, and pale-green shading shows the corresponding favorable region.
Figure S15: Property distributions during rounds 1–5 of the Sanofi PPARd and FXA campaigns. Gray boxes denote the retrospective five-compound Reference samples used in Figure 1 , pale-gray points the complete pre-cutoff project-model training set, green boxes the future Held-out compounds, red boxes LLM proposals and blue boxes final agent-selected REINVENT proposals. The five-compound Reference samples provide visualization context and were not the parents used in the campaigns. Points represent molecules; boxes show medians and interquartile ranges, with 1.5-interquartile-range whiskers. Historical primary-potency values—and FXA solubility and —are experimental measurements. Generic ADMET rows contain model predictions, while TPSA and molecular weight are RDKit descriptors. Dashed green lines show project criteria or generic contextual ranges, and pale-green shading marks the corresponding favorable region.
Figure S16: Property distributions during rounds 1–5 of the four Sanofi RENIN and MMP8 campaigns. Gray boxes denote the retrospective five-compound Reference samples used in Figure 1 , pale-gray points the complete pre-cutoff project-model training set, green boxes the future Held-out compounds, red boxes LLM proposals and blue boxes final agent-selected REINVENT proposals. The five-compound Reference samples provide visualization context and were not the parents used in the campaigns. Points represent molecules; boxes show medians and interquartile ranges, with 1.5-interquartile-range whiskers. Historical primary-potency values—and FXA solubility and —are experimental measurements. Generic ADMET rows contain model predictions, while TPSA and molecular weight are RDKit descriptors. Dashed green lines show project criteria or generic contextual ranges, and pale-green shading marks the corresponding favorable region.
Figure S17: Input-token volume versus observable agent operations during rounds 1–5. LLM and REINVENT values include model-context tokens and agent-visible operations; internal REINVENT scorer calls are excluded.
Figure S18: Blinded medicinal-chemist evaluation of agent proposals. Four chemists completed the 28-question instrument described in Section S6.3 . (a) Source discrimination. For each of the 12 paired cases, the evaluator judged which of two structures was AI-generated; points show the proportion correct with Wilson 95% intervals, and the dashed line marks chance. Counts to the right give correct answers over cases answered. (b) Candidate prioritization. Proportion of the 11 prioritization cases in which a held-out, LLM or REINVENT candidate was selected as the next compound to test; solid bars are individual chemists and the hatched bar pools all responses. The dashed line marks the proportion expected if the three sources were chosen at random. (c) SAR reasoning. Distribution of verdicts over the 5 rationales, pooled across chemists.