AIRA_2: Overcoming Bottlenecks in AI Research Agents

Karen Hambardzumyan; Nicolas Baldwin; Edan Toledo; Rishi Hazra; Michael Kuchnik; Bassel Al Omari; Thomas Simon Foster; Anton Protopopov

cs.AIMar 27, 2026

AIRA_2: Overcoming Bottlenecks in AI Research Agents

Authors: Karen Hambardzumyan, Nicolas Baldwin, Edan Toledo, Rishi Hazra, Michael Kuchnik, Bassel Al Omari, Thomas Simon Foster, Anton Protopopov,

Organizations: FAIR at Meta · University College London · University of Oxford

Abstract

Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes overfitting and performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA $_2$ , which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA $^{\dagger}_{2}$ achieves a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, outperforming the strongest baseline, which achieves 72.7%. On AIRS-Bench, AIRA $_2$ exceeds human state-of-the-art on 6 out of 20 diverse research tasks. Ablations confirm that each architectural component is necessary, that performance follows a predictable scaling law that transfers across LLM backbones, and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.

Figures & tables

Figure 1: AIRA

{}_{\mbox{2}}

performance on MLE-bench-30. We evaluate AIRA

{}_{\mbox{2}}

against top-performing agents from the MLE-bench leaderboard across different compute budgets. Utilizing 8 GPU workers for all configurations, AIRA

{}_{\mbox{2}}

surpasses the strongest baselines at a 24-GPU-hour budget. Performance improves consistently with additional compute, demonstrating the effectiveness of our architectural design. AIRA

{}^{\dagger}_{2}

uses Gemini 3.1, while AIRA

{}_{\mbox{2}}

uses Gemini 3.0.

Figure 2: AIRA

{}_{\mbox{2}}

architecture. The Evolutionary Agent orchestrates the search by maintaining a population of candidate solutions and dispatching mutation tasks to the

N

workers as they become available, without any synchronization barriers. Each worker asynchronously executes a ReAct agent which iteratively reasons, executes code, and observes outputs until a candidate solution is ready. Candidate solutions are evaluated in a separate container, and agents observe only the resulting score. Evaluation is partitioned:

\mathcal{D}_{\text{search}}

guides optimization while

\mathcal{D}_{\text{val}}

determines final selection. In our main experiments, we use

N

= 8 workers.

	% Percentile Rank			% Bronze+			% Silver+			% Gold
Method	3h	24h	72h	3h	24h	72h	3h	24h	72h	3h	24h	72h
AIRA ${}^{\dagger}_{2}$	71.7 $\pm$ 3.5	81.5 $\pm$ 3.2	83.1 $\pm$ 3.2	54.4 $\pm$ 5.3	72.2 $\pm$ 4.7	76.7 $\pm$ 4.5	47.8 $\pm$ 5.3	65.6 $\pm$ 5.0	73.3 $\pm$ 4.7	28.9 $\pm$ 4.8	41.1 $\pm$ 5.2	52.2 $\pm$ 5.3
AIRA ${}_{\mbox{2}}$	59.9 $\pm$ 3.6	71.8 $\pm$ 3.5	76.0 $\pm$ 3.4	38.9 $\pm$ 5.2	57.8 $\pm$ 5.2	61.1 $\pm$ 5.2	33.3 $\pm$ 5.0	50.0 $\pm$ 5.3	58.9 $\pm$ 5.2	20.0 $\pm$ 4.2	32.2 $\pm$ 5.0	36.7 $\pm$ 5.1
AIRA ${}_{\mbox{2}}$ (4 GPU)	56.9 $\pm$ 3.6	71.2 $\pm$ 3.4	76.5 $\pm$ 3.4	37.8 $\pm$ 5.1	55.6 $\pm$ 5.3	62.2 $\pm$ 5.1	30.0 $\pm$ 4.9	47.8 $\pm$ 5.3	55.6 $\pm$ 5.3	11.1 $\pm$ 3.3	30.0 $\pm$ 4.9	40.0 $\pm$ 5.2
AIRA ${}_{\mbox{2}}$ (1 GPU)	41.3 $\pm$ 3.9	56.8 $\pm$ 3.8	63.5 $\pm$ 3.8	22.2 $\pm$ 4.4	41.1 $\pm$ 5.2	51.1 $\pm$ 5.3	17.8 $\pm$ 4.1	32.2 $\pm$ 5.0	41.1 $\pm$ 5.2	10.0 $\pm$ 3.2	20.0 $\pm$ 4.2	24.4 $\pm$ 4.6
AIRA ${}_{\mbox{2}}$ (No Subagents)	54.4 $\pm$ 3.7	68.6 $\pm$ 3.6	73.7 $\pm$ 3.6	33.3 $\pm$ 5.0	52.2 $\pm$ 5.3	60.0 $\pm$ 5.2	30.0 $\pm$ 4.9	47.8 $\pm$ 5.3	53.3 $\pm$ 5.3	13.3 $\pm$ 3.6	32.2 $\pm$ 5.0	37.8 $\pm$ 5.1

Table 1: Main performance evaluation across varying time budgets (3h, 24h, 72h). We report Percentile Rank metric and medal rates (Bronze+, Silver+, and Gold) for AIRA

{}_{\mbox{2}}

(8 GPUs, Subagents, Hidden Consistent Eval (HCE)) against ablations (4 GPU, 1 GPU, No Subagents, No HCE) and state-of-the-art baselines. Confidence intervals denote

\pm\text{SE}.

AIRA

{}^{\dagger}_{2}

uses Gemini 3.1; all other configurations use Gemini 3.0. AIRA

{}^{\dagger}_{2}

surpasses all baselines at 24h, and both AIRA

{}_{\mbox{2}}

and AIRA

{}^{\dagger}_{2}

continue to improve through 72h, demonstrating the effectiveness of the proposed architecture.

Panel of Figure 3 — Figure 3: Compute Analysis. We analyse the impact of parallel resources on AIRA ${}_{\mbox{2}}$ , demonstrating that effective use of parallel compute requires both additional resources and an evolutionary mechanism to utilize them.

Panel of Figure 4 — Figure 4: (a) Stabilizing Long-Horizon Search. We compare the standard self-reported evaluation (blue) against our Hidden Consistent Evaluation protocol (green). While self-reporting leads to eventual performance degradation (confirming Toledo et al. (2025) ), consistent evaluation ensures long-term improvement. Furthermore, the marginal difference between selecting via $\mathcal{D_{\text{search}}}$ (seen) and $\mathcal{D_{\text{val}}}$ (unseen) splits suggests the degradation in prior work was due to evaluation noise, not true data overfitting. (b) Performance profile of AIRA ${}_{\mbox{2}}$ and AIRA ${}^{\dagger}_{2}$ : We observe steady increase in performance in all configurations, with 8-worker parallel version performing the best. AIRA ${}^{\dagger}_{2}$ achieves the highest Percentile Rank among all evaluated agents across all time budgets.

Figure 5: Predictable performance scaling across models. The fitted scaling law captures the diminishing returns of subagent interaction over time (Equation 3 ), tightly fitting the Gemini 3.0 Pro baseline (

R^{2}=0.98

). Notably, the architecture’s scaling parameter transfers across backbones: calibrating Gemini 3.1 Pro on only one or two subagents accurately predicts the performance of the unseen

N=8

configuration (

R^{2}=0.92

Figure 6: Predicted compute frontier. Observed performance across AIRA

{}_{\mbox{2}}

configurations closely tracks the predicted compute frontier

P(C)

(dashed curve, Equation 5 ).

Panel of Figure 7 — Figure 7: Example of typical behaviour observed in AIRA ${}_{\mbox{2}}$ during the champs-scalar-coupling task. The left side displays the chronological performance of each solution in the database (solid lines for mutation, dashed for crossover). The right-hand panel presents a concise summary of the agent’s thought process for the nodes and trajectory annotated in green, highlighting the “eureka” moment where the agent identifies underfitting and subsequently scales the model to achieve medal-winning performance. Horizontal dashed lines indicate the medal thresholds and display the previous best attempt ( Thesis Labs, 2025 ) on the task among all agents.

Figure 8: AIRS-Bench SOTA-beating tasks split by integrity status. Left: Six tasks where AIRA

{}_{\mbox{2}}

exceeded SOTA using clean, inductive methodologies (models trained from scratch, no external data). Italic annotations indicate key techniques. Right: Five tasks where the SOTA-beating solutions were flagged with integrity concerns, color-coded by severity: direct test label access (red), external data or model contamination (orange), and domain-adjacent pretraining advantage (amber). Bars show percentage improvement over published SOTA.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	3h	12h	24h	72h
aptos2019-blindness-detection	100.00 70.30 100.00 51.16 , 100.00 93.31	100.00 87.66 100.00 64.79 , 100.00 99.76	100.00 97.31 100.00 93.75 , 100.00 99.76	100.00 97.35 100.00 96.41 , 100.00 98.16
billion-word-imputation	100.00 66.67 100.00 8.05 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00
bms-molecular-translation	100.00 19.72 100.00 19.45 , 100.00 20.25	100.00 28.15 100.00 19.45 , 100.00 34.10	100.00 57.67 100.00 32.95 , 100.00 80.66	100.00 69.07 100.00 60.64 , 100.00 85.93
cassava-leaf-disease-classification	100.00 58.85 100.00 39.49 , 100.00 88.97	100.00 87.38 100.00 82.90 , 100.00 90.26	100.00 84.73 100.00 77.59 , 100.00 90.26	100.00 89.15 100.00 86.36 , 100.00 92.13
champs-scalar-coupling	100.00 30.72 100.00 9.02 , 100.00 45.80	100.00 49.17 100.00 33.02 , 100.00 64.10	100.00 63.12 100.00 41.09 , 100.00 94.34	100.00 78.13 100.00 60.56 , 100.00 98.79
freesound-audio-tagging-2019	100.00 98.16 100.00 97.60 , 100.00 98.56	100.00 99.68 100.00 99.04 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00

Appendix

Table 2: Per-task Percentile Rank scores for AIRA

{}_{\mbox{2}}

at different time cutoffs. Values shown as score with 95% CI below.

Task	3h	12h	24h	72h
aptos2019-blindness-detection	100.00 92.59 100.00 83.67 , 100.00 98.22	100.00 95.75 100.00 88.70 , 100.00 99.39	100.00 95.75 100.00 88.70 , 100.00 99.39	100.00 96.60 100.00 93.03 , 100.00 99.25
billion-word-imputation	100.00 97.70 100.00 93.10 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00
bms-molecular-translation	100.00 16.40 100.00 0.00 , 100.00 49.20	100.00 80.97 100.00 73.68 , 100.00 85.13	100.00 86.08 100.00 84.44 , 100.00 87.41	100.00 93.63 100.00 92.56 , 100.00 95.08
cassava-leaf-disease-classification	100.00 87.98 100.00 82.90 , 100.00 94.69	100.00 88.40 100.00 82.90 , 100.00 95.95	100.00 96.60 100.00 90.26 , 100.00 99.95	100.00 99.06 100.00 97.26 , 100.00 99.97
champs-scalar-coupling	100.00 95.05 100.00 92.70 , 100.00 96.38	100.00 99.06 100.00 98.69 , 100.00 99.31	100.00 99.27 100.00 99.01 , 100.00 99.45	100.00 99.53 100.00 99.49 , 100.00 99.56
freesound-audio-tagging-2019	100.00 95.84 100.00 93.05 , 100.00 97.84	100.00 99.36 100.00 98.08 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00

Appendix

Table 3: Per-task test percentile scores for AIRA

{}^{\dagger}_{2}

at different time cutoffs. Values shown as score with 95% CI below.

Task	Method	SOTA	Best	Mean	Norm. Best	Norm. Mean	Seeds	Beats SOTA
Code Gen.	AIRA ${}_{\mbox{2}}$	0.1870	0.2810	0.2309	159.4	127.3	3	3
	$AIRA_{\text{greedy}}$ GPT5		0.0050	0.0020	2.4	1.0	10	0
Code Retrieval	AIRA ${}_{\mbox{2}}$	0.6113	0.5772	0.5037	91.1	74.8	3	0
	$AIRA_{\text{greedy}}$ GPT5		0.4485	0.1968	63.0	23.2	10	0
Coref. (WSC)	AIRA ${}_{\mbox{2}}$	0.9620	1.0000	0.9952	$\infty$	$\infty$	4	4
	$AIRA_{\text{greedy}}$ GPT5		0.9423	0.7577	85.2	34.2	10	0

Appendix

Table 4: Per-task results on AIRS-Bench. Each task group shows AIRA

{}_{\mbox{2}}

followed by the two best-performing baselines (selected by best seed). Normalised scores are scaled so that 0 = worst and 100 = SOTA.

\infty

denotes normalised scores whereby 1 seed achieved the optimal score. This is usually due to unfair solutions. See Section 4.6 . We show the best baseline per task selected by the best seed score since we care about SOTA beating solutions. Bold indicates the best method per task (by mean normalised score).

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 299 papers

Feb 16, 2026

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

TCS Research · Yale University

AI Agent Benchmarks Autonomous Research Agents Automated Agent Evaluation Long-Horizon LLM Agents AI Agent Reliability

Ask PDF

Aug 14, 2026

AI Research Preference Models

AI research agents (AIRA) can now carry machine learning experiments from proposal through implementation and evaluation. Yet progress on frontier tasks is throttled by the cost of evaluations that can consume days of GPU time. When an agent can propose far more candidates than it can afford to run, progress depends on its research preference: how it allocates a fixed execution budget across many candidates. We introduce AI Research Preference Models (RPMs) that predict which candidate solution is most promising, without paying the cost of running them all. We build RPMs from frozen pretrained language models in two variants: an inference-only model that reasons over candidate plans, code, and previously executed solutions, and an agentic model that additionally runs small-scale pilot experiments. Integrated into the AIRA-dojo research agent and evaluated on the machine learning research benchmark AIRS-Bench, the two variants increase the average normalized score from 0.684 to 0.711 and 0.729, respectively. Both reach the unguided agent's 24-hour performance in roughly 15 hours, using less than two-thirds of its execution budget, and together yield new state-of-the-art results on two AIRS-Bench tasks.

Thomas Simon Foster, Bassel Al Omari, Tingchen Fu +30

FAIR at Meta · University of Oxford · University College London

AI Agents for Scientific Discovery Preference Learning

Ask PDF

May 17, 2026

FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics

AI research agents accelerate ML research by automating hypothesis generation, experimentation, and empirical refinement. Existing agent strategies range from greedy hill-climbing to tree search and evolutionary optimization, yet which strategy choices drive performance remains unclear. Answering this question requires a benchmark that separates agent strategy (e.g., search topology) from execution infrastructure (e.g., code editor), so that performance differences are attributable to strategy rather than infrastructure, and that provides process-level metrics beyond final scores to analyze exploration behaviors. Existing benchmarks offer limited support. We propose FML-Bench, a benchmark of 18 fundamental ML research tasks across 10 domains that separates agent strategy from execution infrastructure and defines 12 process-level behavioral metrics. Evaluating six representative agents, we find that: (1) strategy complexity alone does not guarantee strong performance: a simple greedy hill-climber nearly matches the best-performing tree-search agent, both well above the remaining agents; (2) our analysis suggests this pattern relates to improvement opportunity structure: greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse; an adaptive agent built on this insight switches to broader exploration upon detecting improvement stagnation and outperforms the other six agents, lending initial support to this observation; and (3) process-level analysis reveals that early convergence and directionally focused exploration are significantly associated with final performance, while solution diversity and compute cost are not. Our benchmark is available at: https://github.com/qrzou/FML-bench.

Qiran Zou, Hou Hei Lam, Wenhao Zhao +11

National University of Singapore · Tsinghua University · University of Minnesota +2

AI Agents for Scientific Discovery Inference-Time Search Exploration-Exploitation Tradeoff AI Agent Evaluation AI Agent Benchmarks Autonomous Research Agents

Ask PDF

Abstract

Autonomous Research Agents

Explore similar work

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

AI Research Preference Models

FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics