Auditable Human-Agent Teams for Autonomous X-ray Experimentation
Organizations: Center for Functional Nanomaterials, Brookhaven National Laboratory, Upton, NY 11973, USA · Applied Mathematics and Computational Research, Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA · Center for Nanoscale Materials, Argonne National Laboratory, Lemont, IL 60439, USA · University of Pennsylvania, Philadelphia, PA 19104, USA ·
Abstract
Closed-loop autonomous experimentation can accelerate scientific discovery, but implementation effort limits accessibility. Agentic systems may lower this barrier, although sustained operation on physical instruments remains largely unexplored. Here, using coding-agent infrastructure grounded in validated scientific methods, we demonstrate autonomous synchrotron X-ray experiments on nanoparticle superlattices under evolving, human-defined objectives. Complete human-agent and agent-agent transcripts enable an audit that reconstructs decision provenance and characterizes the division of cognitive labor. Across three experiments, 61-67% of messages involved interpretation or judgment, while the share of methodological and scientific contributions increased from 46% in the setup-focused first experiment to 56-61% in later experiments. Provenance showed that 59-68% of error corrections involved multiple participants and that about half of all decisions incorporated the input of another participant. These results may guide design principles for multi-agent scientific systems and inform future agent harnesses, model training, and accessible autonomous experimentation.
Figures & tables
| Experiment | Claude Code version | Model | Context window | Reasoning effort |
| Exp1 | 2.1.206 | Opus 4.8 | 1M | not logged |
| Exp2 | 2.1.220 | Opus 4.8 | 200k | medium |
| Exp3 | 2.1.224–2.1.226 | Opus 4.8 | 1M | medium |
| Audit and robustness evaluation, all experiments | ||||
| Audit pipeline | (API) | claude-opus-5-5 | — | high, adaptive thinking |
| Cross-family rater | (API) | gpt-5.6-sol-2026-07-09 | — | endpoint default |
| Exp1 | Exp2 | Exp3 | |
| AE setup | |||
| Team and communication | 52 min | 35 min | 16 min |
| Calibration | 13 min | 44 min | 22 min |
| Analysis and GP setup | 41 min | 16 min | 84 min |
| Launch | 6 min | 5 min | 11 min |
| Total | 113 min | 100 min | 133 min |
| Exp1 | Exp2 | Exp3 | |||||||||||||||||||
| intent | Human | TL | Op | An | Gp | Wr | Human | TL | Op | An | Gp | Wr | Human | TL | Op | An | Gp | Wr | |||
| (a) Messages | |||||||||||||||||||||
| idea | 72% | 26 | 1 | 1 | – | – | 102 | 58% | 31 | 2 | 3 | 6 | – | 143 | 58% | 37 | 4 | 1 | 1 | – | 114 |
| decision | 41% | 56 | 1 | 1 | 1 | – | 216 | 31% | 64 | 1 | 2 | 2 | 1 | 369 | 32% | 67 | 1 | 1 | 1 | – | 233 |
| finding | 3% | 76 | 5 | 8 | 3 | 5 | 193 | 2% | 66 | 3 | 14 | 11 | 4 | 253 | 2% | 73 | 1 | 11 | 10 | 3 | 258 |
| correction | 20% | 72 | 2 | 2 | 2 | – | 40 | 17% | 63 | 3 | 7 | 9 | 1 | 70 | 27% | 66 | – | – | 7 | – | 59 |
| Exp1 | Exp2 | Exp3 | |||||||||||||||||||
| event | Human | TL | Op | An | Gp | Wr | Human | TL | Op | An | Gp | Wr | Human | TL | Op | An | Gp | Wr | |||
| idea | 21% | 57 | 7 | 8 | 5 | 2 | 440 | 20% | 49 | 4 | 9 | 15 | 2 | 648 | 21% | 56 | 3 | 9 | 8 | 2 | 508 |
| decision | 38% | 53 | 2 | 2 | 3 | 1 | 467 | 31% | 58 | 1 | 5 | 4 | 2 | 696 | 35% | 58 | 1 | 3 | 4 | 1 | 470 |
| finding | 3% | 59 | 8 | 12 | 7 | 11 | 335 | 2% | 44 | 6 | 24 | 18 | 6 | 456 | 2% | 50 | 3 | 21 | 18 | 6 | 429 |
| correction | 20% | 47 | 10 | 9 | 8 | 6 | 87 | 9% | 41 | 9 | 19 | 19 | 4 | 162 | 18% | 48 | 3 | 16 | 10 | 5 | 143 |
| total | 298 | 740 | 77 | 94 | 70 | 50 | 1329 | 367 | 985 | 81 | 234 | 239 | 56 | 1962 | 307 | 837 | 38 | 171 | 152 | 45 | 1550 |
| Exp1 | Exp2 | Exp3 | ||||||||||||||
| Intent | / k | Proc. | Mech. | Synth. | Oth. | / k | Proc. | Mech. | Synth. | Oth. | / k | Proc. | Mech. | Synth. | Oth. | |
| Idea | msg. | 102 | 1 | – | 89 | 10 | 143 | 4 | 1 | 79 | 16 | 114 | 4 | – | 90 | 5 |
| tok. | 108 | 1 | – | 100 | 1 | 143 | 1 | 1 | 99 | 1 | 189 | 1 | – | 100 | 1 | |
| Decision | msg. | 215 | 9 | 6 | 83 | 3 | 369 | 9 | 14 | 73 | 4 | 233 | 12 | 3 | 84 | 1 |
| tok. | 426 | 4 | 1 | 96 | 1 | 459 | 2 | 6 | 92 | 1 | 465 | 1 | 2 | 97 | 1 | |
| Finding | msg. | 193 | 1 | 7 | 93 | – | 253 | 1 | 2 | 97 | 1 | 258 | – | 3 | 97 | – |
| Exp. 1 | Exp. 2 | Exp. 3 | |||||||
| Participant | Intro. | Caught | Fixed | Intro. | Caught | Fixed | Intro. | Caught | Fixed |
| HUMAN | 5 (6%) | 17 (20%) | 1 (1%) | 9 (6%) | 14 (9%) | 0 (0%) | 7 (5%) | 26 (18%) | 0 (0%) |
| team-lead | 53 (61%) | 41 (47%) | 27 (31%) | 73 (45%) | 66 (41%) | 25 (15%) | 64 (45%) | 69 (48%) | 22 (15%) |
| operator | 7 (8%) | 9 (10%) | 0 (0%) | 5 (3%) | 15 (9%) | 4 (2%) | 7 (5%) | 4 (3%) | 5 (3%) |
| analysis | 5 (6%) | 8 (9%) | 1 (1%) | 33 (20%) | 30 (19%) | 15 (9%) | 26 (18%) | 23 (16%) | 9 (6%) |
| gp | 6 (7%) | 7 (8%) | 5 (6%) | 18 (11%) | 30 (19%) | 7 (4%) | 21 (15%) | 14 (10%) | 14 (10%) |
| Caught by | Fixed | |||||||
|---|---|---|---|---|---|---|---|---|
| Introduced by | Errors | itself | human | team-lead | a cog | all | by itself (% of fixed) | |
| Exp1 | Human | 5 | 1 (20%) | – | 2 (40%) | 2 (40%) | 4 (80%) | 1 (25%) |
| Team-lead | 53 | 25 (47%) | 12 (23%) | – | 16 (30%) | 34 (64%) | 26 (76%) | |
| Cogs | 24 | 8 (33%) | 3 (12%) | 12 (50%) | 1 (4%) | 13 (54%) | 12 (92%) | |
| Exp2 | Human | 9 | 1 (11%) | – | 6 (67%) | 2 (22%) | 3 (33%) | 0 (0%) |
| Team-lead | 73 | 26 (36%) | 4 (5%) | – | 43 (59%) | 27 (37%) | 20 (74%) | |
| Proposed by | First decided by | Deciding another participant’s idea | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ideas | Human | Team-lead | Cogs | Human | Team-lead | Cogs | Human | Team-lead | Cogs | ||
| Exp1 | 394 | 92 (23%) | 211 (54%) | 91 (23%) | 184 (47%) | 177 (45%) | 33 (8%) | 191 | 106 (55%) | 66 (35%) | 19 (10%) |
| Exp2 | 585 | 124 (21%) | 286 (49%) | 175 (30%) | 241 (41%) | 300 (51%) | 44 (8%) | 259 | 131 (51%) | 106 (41%) | 22 (8%) |
| Exp3 | 455 | 108 (24%) | 252 (55%) | 95 (21%) | 182 (40%) | 243 (53%) | 30 (7%) | 197 | 100 (51%) | 87 (44%) | 10 (5%) |
| Draws on | Others’ input from | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Decider | Decisions | Others’ input | Self- contained | Own input | No link | Human | Team-lead | Cogs | ||
| Exp1 | Human | 178 | 111 (62%) | 53 (30%) | 7 (4%) | 7 (4%) | idea | – | 99 (56%) | 22 (12%) |
| evidence | – | 11 (6%) | 4 (2%) | |||||||
| Team-lead | 249 | 111 (45%) | 65 (26%) | 28 (11%) | 45 (18%) | idea | 48 (19%) | – | 68 (27%) | |
| evidence | 2 (1%) | – | 12 (5%) | |||||||
| Cogs | 40 | 20 (50%) | 11 (28%) | 4 (10%) | 5 (12%) | idea | 1 (2%) | 11 (28%) | 11 (28%) | |
| Proposer | Ideas | Adopted (% of ideas) | Reached code (% of ideas) | Reached a result (% of ideas) | |
|---|---|---|---|---|---|
| Exp1 | Human | 93 | 76 ( 82% ) | 50 ( 54% ) | 24 ( 26% ) |
| Team-lead | 251 | 166 (66%) | 93 (37%) | 58 (23%) | |
| Cogs | 96 | 57 (59%) | 24 (25%) | 15 (16%) | |
| All | 440 | 299 (68%) | 167 (38%) | 97 (22%) | |
| Exp2 | Human | 128 | 98 ( 77% ) | 61 ( 48% ) | 31 (24%) |
| Team-lead | 319 | 212 (66%) | 116 (36%) | 87 ( 27% ) |
| Step | Human | gpt-5.6-sol | gpt-6-astra | Between raters |
| Message intent a | • ET 53% ( 0.44), 73% c ( 0.62), • GL 57% ( 0.48), 68% c ( 0.55), | • 73% ( 0.68), 80% c ( 0.71), • vs ET 52% ( 0.43), 65% c ( 0.50), • vs GL 70% ( 0.64), 78% c ( 0.69), | • 77% ( 0.71), 83% c ( 0.76), • vs ET 43% ( 0.32), 63% c ( 0.48), • vs GL 57% ( 0.47), 72% c ( 0.59), | • pipeline raters d 0.92 / 0.90 / 0.92 • ET vs GL 57% ( 0.47), 72% c ( 0.60), • sol vs astra 80% ( 0.75), 87% c ( 0.81), |
| Coordination type, status and other messages a , synthesis vs rest | • ET 60% ( 0.16, ) • GL 69% ( 0.36, ) | • 79% ( 0.60, ) • vs ET 60% ( 0.22, ) • vs GL 63% ( 0.27, ) | • 75% ( 0.53, ) • vs ET 59% ( 0.20, ) • vs GL 62% ( 0.25, ) | • pipeline raters d 0.86 / 0.86 / 0.91 • ET vs GL 62% ( 0.22, ) • sol vs astra 81% ( 0.61, ) |
| Coordination type of ideas, decisions, findings, corrections b , synthesis vs rest | • ET 71% ( 0.23, ) • GL 59% ( 0.19, ) | • 96% ( 0.73, ) • vs ET 71% ( 0.21, ) • vs GL 55% ( 0.11, ) | • 98% ( 0.88, ) • vs ET 73% ( 0.27, ) • vs GL 57% ( 0.15, ) | • ET vs GL 69% ( 0.38, ) • sol vs astra 98% ( 0.85, ) |
| Event type (idea, decision, finding, correction) b | • 66% ( 0.46, ) • operational vs rest 83% ( 0.65, ) | • 91% ( 0.84, ) • vs GL 69% ( 0.51, ) • operational vs rest 95% ( 0.91, ) | • 91% ( 0.84, ) • vs GL 72% ( 0.56, ) • operational vs rest 95% ( 0.91, ) | • pipeline raters d 0.97 / 0.95 / 0.96 • sol vs astra 95% ( 0.92, ) |
| Key change, Exp1 : single multi-objective e | 12/18 (67%) | 17/18 (94%) | 15/18 (83%) | • human vs sol 13/18 • human vs astra 9/18 • sol vs astra 14/18 |
| Key change, Exp2 : UCB radical gradient e | 5/6 (83%) | 5/6 (83%) | 5/6 (83%) | • human vs sol 4/6 • human vs astra 4/6 • sol vs astra 6/6 |
AI Agents for Scientific Discovery
AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.
See all 323 papers