AIRA_2: Overcoming Bottlenecks in AI Research Agents

Karen Hambardzumyan; Nicolas Baldwin; Edan Toledo; Rishi Hazra; Michael Kuchnik; Bassel Al Omari; Thomas Simon Foster; Anton Protopopov

cs.AIMar 27, 2026

AIRA_2: Overcoming Bottlenecks in AI Research Agents

Authors: Karen Hambardzumyan, Nicolas Baldwin, Edan Toledo, Rishi Hazra, Michael Kuchnik, Bassel Al Omari, Thomas Simon Foster, Anton Protopopov,

Organizations: FAIR at Meta · University College London · University of Oxford

Abstract

Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes overfitting and performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA $_2$ , which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA $^{\dagger}_{2}$ achieves a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, outperforming the strongest baseline, which achieves 72.7%. On AIRS-Bench, AIRA $_2$ exceeds human state-of-the-art on 6 out of 20 diverse research tasks. Ablations confirm that each architectural component is necessary, that performance follows a predictable scaling law that transfers across LLM backbones, and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.

Figures & tables

Figure 1: AIRA

{}_{\mbox{2}}

performance on MLE-bench-30. We evaluate AIRA

{}_{\mbox{2}}

against top-performing agents from the MLE-bench leaderboard across different compute budgets. Utilizing 8 GPU workers for all configurations, AIRA

{}_{\mbox{2}}

surpasses the strongest baselines at a 24-GPU-hour budget. Performance improves consistently with additional compute, demonstrating the effectiveness of our architectural design. AIRA

{}^{\dagger}_{2}

uses Gemini 3.1, while AIRA

{}_{\mbox{2}}

uses Gemini 3.0.

Figure 2: AIRA

{}_{\mbox{2}}

architecture. The Evolutionary Agent orchestrates the search by maintaining a population of candidate solutions and dispatching mutation tasks to the

N

workers as they become available, without any synchronization barriers. Each worker asynchronously executes a ReAct agent which iteratively reasons, executes code, and observes outputs until a candidate solution is ready. Candidate solutions are evaluated in a separate container, and agents observe only the resulting score. Evaluation is partitioned:

\mathcal{D}_{\text{search}}

guides optimization while

\mathcal{D}_{\text{val}}

determines final selection. In our main experiments, we use

N

= 8 workers.

	% Percentile Rank			% Bronze+			% Silver+			% Gold
Method	3h	24h	72h	3h	24h	72h	3h	24h	72h	3h	24h	72h
AIRA ${}^{\dagger}_{2}$	71.7 $\pm$ 3.5	81.5 $\pm$ 3.2	83.1 $\pm$ 3.2	54.4 $\pm$ 5.3	72.2 $\pm$ 4.7	76.7 $\pm$ 4.5	47.8 $\pm$ 5.3	65.6 $\pm$ 5.0	73.3 $\pm$ 4.7	28.9 $\pm$ 4.8	41.1 $\pm$ 5.2	52.2 $\pm$ 5.3
AIRA ${}_{\mbox{2}}$	59.9 $\pm$ 3.6	71.8 $\pm$ 3.5	76.0 $\pm$ 3.4	38.9 $\pm$ 5.2	57.8 $\pm$ 5.2	61.1 $\pm$ 5.2	33.3 $\pm$ 5.0	50.0 $\pm$ 5.3	58.9 $\pm$ 5.2	20.0 $\pm$ 4.2	32.2 $\pm$ 5.0	36.7 $\pm$ 5.1
AIRA ${}_{\mbox{2}}$ (4 GPU)	56.9 $\pm$ 3.6	71.2 $\pm$ 3.4	76.5 $\pm$ 3.4	37.8 $\pm$ 5.1	55.6 $\pm$ 5.3	62.2 $\pm$ 5.1	30.0 $\pm$ 4.9	47.8 $\pm$ 5.3	55.6 $\pm$ 5.3	11.1 $\pm$ 3.3	30.0 $\pm$ 4.9	40.0 $\pm$ 5.2
AIRA ${}_{\mbox{2}}$ (1 GPU)	41.3 $\pm$ 3.9	56.8 $\pm$ 3.8	63.5 $\pm$ 3.8	22.2 $\pm$ 4.4	41.1 $\pm$ 5.2	51.1 $\pm$ 5.3	17.8 $\pm$ 4.1	32.2 $\pm$ 5.0	41.1 $\pm$ 5.2	10.0 $\pm$ 3.2	20.0 $\pm$ 4.2	24.4 $\pm$ 4.6
AIRA ${}_{\mbox{2}}$ (No Subagents)	54.4 $\pm$ 3.7	68.6 $\pm$ 3.6	73.7 $\pm$ 3.6	33.3 $\pm$ 5.0	52.2 $\pm$ 5.3	60.0 $\pm$ 5.2	30.0 $\pm$ 4.9	47.8 $\pm$ 5.3	53.3 $\pm$ 5.3	13.3 $\pm$ 3.6	32.2 $\pm$ 5.0	37.8 $\pm$ 5.1

Table 1: Main performance evaluation across varying time budgets (3h, 24h, 72h). We report Percentile Rank metric and medal rates (Bronze+, Silver+, and Gold) for AIRA

{}_{\mbox{2}}

(8 GPUs, Subagents, Hidden Consistent Eval (HCE)) against ablations (4 GPU, 1 GPU, No Subagents, No HCE) and state-of-the-art baselines. Confidence intervals denote

\pm\text{SE}.

AIRA

{}^{\dagger}_{2}

uses Gemini 3.1; all other configurations use Gemini 3.0. AIRA

{}^{\dagger}_{2}

surpasses all baselines at 24h, and both AIRA

{}_{\mbox{2}}

and AIRA

{}^{\dagger}_{2}

continue to improve through 72h, demonstrating the effectiveness of the proposed architecture.

Panel of Figure 3 — Figure 3: Compute Analysis. We analyse the impact of parallel resources on AIRA ${}_{\mbox{2}}$ , demonstrating that effective use of parallel compute requires both additional resources and an evolutionary mechanism to utilize them.

Panel of Figure 4 — Figure 4: (a) Stabilizing Long-Horizon Search. We compare the standard self-reported evaluation (blue) against our Hidden Consistent Evaluation protocol (green). While self-reporting leads to eventual performance degradation (confirming Toledo et al. (2025) ), consistent evaluation ensures long-term improvement. Furthermore, the marginal difference between selecting via $\mathcal{D_{\text{search}}}$ (seen) and $\mathcal{D_{\text{val}}}$ (unseen) splits suggests the degradation in prior work was due to evaluation noise, not true data overfitting. (b) Performance profile of AIRA ${}_{\mbox{2}}$ and AIRA ${}^{\dagger}_{2}$ : We observe steady increase in performance in all configurations, with 8-worker parallel version performing the best. AIRA ${}^{\dagger}_{2}$ achieves the highest Percentile Rank among all evaluated agents across all time budgets.

Figure 5: Predictable performance scaling across models. The fitted scaling law captures the diminishing returns of subagent interaction over time (Equation 3 ), tightly fitting the Gemini 3.0 Pro baseline (

R^{2}=0.98

). Notably, the architecture’s scaling parameter transfers across backbones: calibrating Gemini 3.1 Pro on only one or two subagents accurately predicts the performance of the unseen

N=8

configuration (

R^{2}=0.92

Figure 6: Predicted compute frontier. Observed performance across AIRA

{}_{\mbox{2}}

configurations closely tracks the predicted compute frontier

P(C)

(dashed curve, Equation 5 ).

Panel of Figure 7 — Figure 7: Example of typical behaviour observed in AIRA ${}_{\mbox{2}}$ during the champs-scalar-coupling task. The left side displays the chronological performance of each solution in the database (solid lines for mutation, dashed for crossover). The right-hand panel presents a concise summary of the agent’s thought process for the nodes and trajectory annotated in green, highlighting the “eureka” moment where the agent identifies underfitting and subsequently scales the model to achieve medal-winning performance. Horizontal dashed lines indicate the medal thresholds and display the previous best attempt ( Thesis Labs, 2025 ) on the task among all agents.

Figure 8: AIRS-Bench SOTA-beating tasks split by integrity status. Left: Six tasks where AIRA

{}_{\mbox{2}}

exceeded SOTA using clean, inductive methodologies (models trained from scratch, no external data). Italic annotations indicate key techniques. Right: Five tasks where the SOTA-beating solutions were flagged with integrity concerns, color-coded by severity: direct test label access (red), external data or model contamination (orange), and domain-adjacent pretraining advantage (amber). Bars show percentage improvement over published SOTA.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	3h	12h	24h	72h
aptos2019-blindness-detection	100.00 70.30 100.00 51.16 , 100.00 93.31	100.00 87.66 100.00 64.79 , 100.00 99.76	100.00 97.31 100.00 93.75 , 100.00 99.76	100.00 97.35 100.00 96.41 , 100.00 98.16
billion-word-imputation	100.00 66.67 100.00 8.05 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00
bms-molecular-translation	100.00 19.72 100.00 19.45 , 100.00 20.25	100.00 28.15 100.00 19.45 , 100.00 34.10	100.00 57.67 100.00 32.95 , 100.00 80.66	100.00 69.07 100.00 60.64 , 100.00 85.93
cassava-leaf-disease-classification	100.00 58.85 100.00 39.49 , 100.00 88.97	100.00 87.38 100.00 82.90 , 100.00 90.26	100.00 84.73 100.00 77.59 , 100.00 90.26	100.00 89.15 100.00 86.36 , 100.00 92.13
champs-scalar-coupling	100.00 30.72 100.00 9.02 , 100.00 45.80	100.00 49.17 100.00 33.02 , 100.00 64.10	100.00 63.12 100.00 41.09 , 100.00 94.34	100.00 78.13 100.00 60.56 , 100.00 98.79
freesound-audio-tagging-2019	100.00 98.16 100.00 97.60 , 100.00 98.56	100.00 99.68 100.00 99.04 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00

Appendix

Table 2: Per-task Percentile Rank scores for AIRA

{}_{\mbox{2}}

at different time cutoffs. Values shown as score with 95% CI below.

Task	3h	12h	24h	72h
aptos2019-blindness-detection	100.00 92.59 100.00 83.67 , 100.00 98.22	100.00 95.75 100.00 88.70 , 100.00 99.39	100.00 95.75 100.00 88.70 , 100.00 99.39	100.00 96.60 100.00 93.03 , 100.00 99.25
billion-word-imputation	100.00 97.70 100.00 93.10 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00
bms-molecular-translation	100.00 16.40 100.00 0.00 , 100.00 49.20	100.00 80.97 100.00 73.68 , 100.00 85.13	100.00 86.08 100.00 84.44 , 100.00 87.41	100.00 93.63 100.00 92.56 , 100.00 95.08
cassava-leaf-disease-classification	100.00 87.98 100.00 82.90 , 100.00 94.69	100.00 88.40 100.00 82.90 , 100.00 95.95	100.00 96.60 100.00 90.26 , 100.00 99.95	100.00 99.06 100.00 97.26 , 100.00 99.97
champs-scalar-coupling	100.00 95.05 100.00 92.70 , 100.00 96.38	100.00 99.06 100.00 98.69 , 100.00 99.31	100.00 99.27 100.00 99.01 , 100.00 99.45	100.00 99.53 100.00 99.49 , 100.00 99.56
freesound-audio-tagging-2019	100.00 95.84 100.00 93.05 , 100.00 97.84	100.00 99.36 100.00 98.08 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00	100.00 100.00 100.00 100.00 , 100.00 100.00

Appendix

Table 3: Per-task test percentile scores for AIRA

{}^{\dagger}_{2}

at different time cutoffs. Values shown as score with 95% CI below.

Task	Method	SOTA	Best	Mean	Norm. Best	Norm. Mean	Seeds	Beats SOTA
Code Gen.	AIRA ${}_{\mbox{2}}$	0.1870	0.2810	0.2309	159.4	127.3	3	3
	$AIRA_{\text{greedy}}$ GPT5		0.0050	0.0020	2.4	1.0	10	0
Code Retrieval	AIRA ${}_{\mbox{2}}$	0.6113	0.5772	0.5037	91.1	74.8	3	0
	$AIRA_{\text{greedy}}$ GPT5		0.4485	0.1968	63.0	23.2	10	0
Coref. (WSC)	AIRA ${}_{\mbox{2}}$	0.9620	1.0000	0.9952	$\infty$	$\infty$	4	4
	$AIRA_{\text{greedy}}$ GPT5		0.9423	0.7577	85.2	34.2	10	0

Appendix

Table 4: Per-task results on AIRS-Bench. Each task group shows AIRA

{}_{\mbox{2}}

followed by the two best-performing baselines (selected by best seed). Normalised scores are scaled so that 0 = worst and 100 = SOTA.

\infty

denotes normalised scores whereby 1 seed achieved the optimal score. This is usually due to unfair solutions. See Section 4.6 . We show the best baseline per task selected by the best seed score since we care about SOTA beating solutions. Bold indicates the best method per task (by mean normalised score).

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 299 papers

ResearchGym: Evaluating Language Model Agents on Real-World AI Research
Feb 16, 2026Aniketh Garikaparthi, Manasi Patwardhan, Arman CohanAI Agent Benchmarks Autonomous Research Agents
TCS Research · Yale University
AI Research Preference Models
Aug 14, 2026Thomas Simon Foster, Bassel Al Omari, Tingchen Fu +30AI Agents for Scientific Discovery Preference Learning
FAIR at Meta · University of Oxford · University College London
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
May 17, 2026Qiran Zou, Hou Hei Lam, Wenhao Zhao +11AI Agents for Scientific Discovery Inference-Time Search
National University of Singapore · Tsinghua University · University of Minnesota +2

Feb 16, 2026J/K move · Enter open · S save

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

TCS Research · Yale University

AI Agent Benchmarks Autonomous Research Agents Automated Agent Evaluation Long-Horizon LLM Agents AI Agent Reliability

Open paper AskPDF arXiv

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

Abstract

Autonomous Research Agents

Explore similar work

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

AI Research Preference Models

FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics