Competing with AI Scientists: Agent-Driven Approach to Astrophysics Research
Authors: Thomas Borrett, Licong Xu, Andy Nilipour, Boris Bolliet, Sebastien Pierre, Erwan Allys, Celia Lecat, Biwei Dai, Po-Wen Chang, Wahid Bhimji
Organizations: Cavendish Astrophysics, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK · Kavli Institute for Cosmology, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK · Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK · Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France · CentraleSupélec, Gif-sur-Yvette, France · School of Natural Sciences, Institute for Advanced Study, 1 Einstein Drive, Princeton, NJ 08540, USA · NERSC, Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA
We present an agent-driven approach to the construction of parameter inference pipelines for scientific data analysis. Our method leverages a multi-agent system, Cmbagent (the analysis system of the AI scientist Denario), in which specialized agents collaborate to generate research ideas, write and execute code, evaluate results, and iteratively refine the overall pipeline. As a case study, we apply this approach to the FAIR Universe Weak Lensing Uncertainty Challenge, a competition under time constraints focused on robust cosmological parameter inference with realistic observational uncertainties. While the fully autonomous exploration initially did not reach expert-level performance, the integration of human intervention enabled our agent-driven workflow to achieve a first-place result in the challenge. This demonstrates that semi-autonomous agentic systems can compete with, and in some cases surpass, expert solutions. We describe our workflow in detail, including both the autonomous and semi-autonomous exploration by Cmbagent. Our final inference pipeline utilizes parameter-efficient convolutional neural networks, likelihood calibration over a known parameter grid, and multiple regularization techniques. Our results suggest that agent-driven research workflows can provide a scalable framework to rapidly explore and construct pipelines for inference problems.
Figures & tables
Figure 1: Inference pipeline for weak lensing. The objective is to develop a mapping, f , that transforms an input weak lensing map, x , into predicted cosmological parameters, θ^ , and associated uncertainties, σ^ . This process yields a joint posterior distribution (right) for the cosmological parameters Ωm , the matter density, and S8 , the clustering amplitude. The input map shown is a square 176 pixel-wide cutout from the full map, which has pixel dimension 1424×176 .
Figure 2: The autonomous research loop we adopt for this task. We craft an initial prompt with example code, context, and scores of various approaches. Cmbagent in Planning & Control mode then attempts a solution according to a plan, with analysis of errors and shortcomings. The output report is injected into the initial prompt, and the loop is restarted for a second attempt.
Figure 3: Diagram of our inference pipeline. CNN ensemble predictions (with D4 test-time augmentation) are treated as summary statistics, θ^ ; an empirical Gaussian likelihood is calibrated on validation predictions at each cosmology point θg and is used to estimate the weights to produce point estimates and uncertainties.
Score
MSE
Coverage
11.70
0.10
0.70
Table 1: Performance of the inference pipeline on an unseen test set. The MSE is the normalized error, and the coverage is the fraction of parameters within ±1σ .
Category
First surfaced in autonomous search
First surfaced after human redirection
CNN backbone
ResNet (18/34/50)
BaseCNN, Inception, InceptionSE
Inference method
MCMC, direct moments prediction
Grid likelihood over known cosmology points
Augmentation
Flips, rotations
Full D4 symmetries with test-time averaging
Table 2: Exploration of the inference design space under autonomous and human-redirected agentic search. Entries indicate when a method family first appeared in the search trajectory, rather than a controlled attribution of individual contributions. Some explored methods do not appear in the final pipeline.
Agents/LLMs
Model
Inference
Augmentation
Score
Cmbagent - P&C
ResNet-18/34/50
MCMC
Flips/rotations
10.50
+ Cmbagent - One-Shot
BaseCNN
MCMC
No
11.02
+ GPT-5
BaseCNN
Grid
No
11.42
+ Gemini-2.5-Pro
Inception + InceptionSE
Grid
Flips/rotations/transpose
11.72
Table 3: Summary of model–generation pathways used during the architecture search, showing the models proposed under different Cmbagent /LLM configurations and their corresponding test-set performance. The “+” prefix indicates the incremental integration of the specified LLM into the search pipeline. In the inference method column, “MCMC” refers to sampling the posterior using a Markov Chain Monte Carlo algorithm. “Grid” correspond to our final inference pipeline, described in Section 3.5 . BaseCNN denotes a light-weight CNN. The score presented here is on the validation dataset.
Table 4: Layer-by-layer summary of the two backbone architectures. The total number of trainable parameters is 422,754 for the Inception model and 722,610 for the Inception–SE model.
Figure 4: Left: The cosmologies seen during training, validation, and testing, and the unseen cosmologies only seen during testing. Right: The mean score of our inference pipeline for each unseen cosmology with the associated standard error on the mean. The poor performance at one unseen cosmology is driven by the scarcity of the training data in that region.
Recent advances in artificial intelligence (AI) agents are pushing AI beyond tools toward autonomous scientific discovery. We discuss two complementary agentic systems for cosmology: CMBEvolve, which targets tasks with explicit quantitative objectives through LLM-guided code evolution and tree search, and CosmoEvolve, which targets open-ended scientific workflows through a virtual multi-agent research laboratory. As preliminary demonstrations, we apply CMBEvolve to out-of-distribution detection in weak-lensing maps, where it iteratively improves the benchmark score through code evolution, and CosmoEvolve to autonomous ACT DR6 data analysis, where it identifies non-trivial pair- and scale-dependent behaviour and produces analysis-grade diagnostics. These examples show how cosmology can provide both controlled benchmark tasks and realistic open-ended research problems for the development of AI scientist systems.
Licong Xu, Thomas Borrett
Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK · Kavli Institute for Cosmology, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK · Cavendish Astrophysics, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK
Agentic AI systems are increasingly being integrated into scientific workflows, yet their behavior under realistic conditions remains insufficiently understood. We evaluate CMBAgent across two workflow paradigms and eighteen astrophysical tasks. In the One-Shot setting, access to domain-specific context yields an approximately ~6x performance improvement (0.85 vs. ~0 without context), with the primary failure mode being silent incorrect computation - syntactically valid code that produces plausible but inaccurate results. In the Deep Research setting, the system frequently exhibits silent failures across stress tests, producing physically inconsistent posteriors without self-diagnosis. Overall, performance is strong on well-specified tasks but degrades on problems designed to probe reasoning limits, often without visible error signals. These findings highlight that the most concerning failure mode in agentic scientific workflows is not overt failure, but confident generation of incorrect results. We release our evaluation framework to facilitate systematic reliability analysis of scientific AI agents.
Shivam Rawat, Lucie Flek
Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.
We report a comparison of two state-of-the-art agentic AI systems, Claude Code (Anthropic) and Codex (OpenAI), tasked with autonomously executing a simple end-to-end gravitational wave data analysis pipeline on a shared computing infrastructure without human intervention. The pipeline comprises power spectral density estimation from raw Einstein Telescope simulated noise, geometric template bank generation, matched filter recovery of 100 binary black hole signal injections, automated results generation, and large language model-assisted production of a manuscript formatted in the style of Physical Review D. Both agents received identical written specifications and identical compute resources. The experiment was run twice: a first run with unrealistically loud injections, and a second run with signals rescaled to a physically motivated SNR range. The scientific results converged in both runs. However, the agents exhibited substantially different behaviors and computational costs: Claude Code completed the pipeline in ~3.4 minutes with silent deviations from the specification, while Codex required ~16 minutes across explicit self-correcting restarts, including an unsolicited performance optimization of the matched filter inner loop. The autonomously generated manuscripts also diverged in length, details, and quality. In the second run, a subtle difference in the interpretation of the SNR range instruction led to a genuine scientific divergence: Claude Code silently reinterpreted the instructions, while Codex followed the specification literally. We discuss the implications of these behavioral differences, such as speed versus auditability, silent versus transparent error handling, instruction interpretation, and the criticality of intermediate data representations in multi-model pipelines, for the deployment of agentic AI in scientific computing workflows.
Gianluca Inguglia
Marietta Blau Institute for Particle Physics, Austrian Academy of Sciences, 1010 Vienna, Austria