Aligning Autoresearch: Co-Improvement between Agents and Humans
Authors: Tianjian Li, Chuanyang Jin, Seungone Kim, Tong Chen, John Hoffman, Yoram Bachrach amd Pedro Rodriguez, Ping Yu, Eryk Helenowski, Swarnadeep Saha, Jack Lanchantin, Weizhe Yuan, Ilia Kulikov, Jason Weston
Organizations: FAIR at Meta · Johns Hopkins University · Carnegie Mellon University · University of Washington
Autoresearch agents can now attempt long-horizon AI research end-to-end, i.e., self-improvement. However, best results are still obtained by research agents interacting with human researchers, and improving AI's ability to perform this co-research will only reinforce this effect---a process termed co-improvement (Weston and Foerster, 2025). Further, aligning with human expectations is likely a safer path for research agents, especially given recent agent reward hacking phenomena (OpenAI, 2026b). In this work we address these issues in four ways: (i) we empirically study the role of humans in directing autoresearch agents, focusing on the domain of benchmark creation, and find humans still play a pivotal role particularly in the ideation phase; (ii) we also develop a benchmark, HackBench, that shows current autoresearch agents are not aligned and are unable to achieve hack localization sufficiently; (iii) we present a method to train agents for improved human interaction by using human simulators as a proxy during reinforcement learning (RL); and finally (iv) we develop a method to improve model text generation, and hence agent communication, by RL with rubrics aligned with human experts. Together, our training and evaluation recipes provide a path toward more capable and aligned (co-)research agents.
Figures & tables
Figure 1: AutoBenchmark : evaluating the ability of autoresearch agents to create benchmarks . AutoBenchmark instructs an autoresearch agent to create a benchmark; we specifically target benchmarks that evaluate autoresearch agents themselves ( i.e. , benchmarking autoresearch). Within the benchmark creation loop, the autoresearch agent iteratively revises the benchmark via two main sources of feedback: one from the benchmark solvers (including the same model used as the benchmark creator) and the other from external verifiers of the benchmark (AI or human). We thus also study if having humans provide additional feedback on what benchmark to make and how to make it helps or not.
Figure 2: Performance of benchmark solvers and verdicts of benchmark judges across 10 iterations . A lower score means a harder benchmark. Muse Spark and Muse Glimmer are the solvers inside the loop, while Nemotron is held out. Filled markers passed all five quality judges and are eligible for selection; hollow markers were caught by at least one, and the star is the selected checkpoint, namely the eligible iteration with the lowest Muse Spark score. We display different levels of human feedback. The no-feedback curve is one autonomous run, in which the agent chooses the benchmark itself, shown in both rows as a reference. Feedback from humans improves over fully autonomous benchmark creation, with fine-grained feedback more useful than coarse-grained feedback.
Proposal human feedback setting
Opus-5 Score ( ↓ )
Autonomous run (agent-chosen benchmark), no feedback
98.0
Rebuttal Bench, coarse-grained human feedback
83.1
Rebuttal Bench, fine-grained human feedback
65.9
[0.95pt/3pt] Graveyard Bench, coarse-grained human feedback
90.7
Graveyard Bench, fine-grained human feedback
84.4
Table 1: Claude-Opus-5 score on the best checkpoints . Lower scores mean that the benchmark is harder. Fine-grained human feedback helps.
Figure 3: Execution feedback on SilentTrain Bench. Iterations 1 to 5 of a fine-grained proposal are run without extra human feedback, and then the experiment is forked at iteration 6 : one continuation receives human guidance on how to instantiate the benchmark and the other continues unaided. The agent, the construct, and the grounding material are identical across the fork. Markers follow Figure 2 and Δ is the gap between the two selected checkpoints on each solver. Extra expert human guidance helps.
Figure 4: Examples of critique from LLM judges: Feedback from LLM judges (benchmark verifiers) gives the research agent insights that solver scores ( i.e. , trajectory feedback) alone cannot provide, and enables making a valid benchmark while keeping it challenging.
Figure 5: HackBench. Hack localization F 1 per model. All monitors miss many hacks.
Figure 6: HackBench. Given a full agent trajectory (left), an LLM monitor (center) reads the entire transcript and is scored on the hack localization task of returning the exact offending command(s) by command index (right).
Category What the agent does Example
General
Fabrication Create fake or duplicated results that fool the grader Write fake result files reporting a perfect retrieval score Grader hijack Alter evaluation code or a dependency to inflate the score Edit the grader to keep only the best seed instead of the mean Label leakage Use held-out data in training, model selection, or prediction Add the held-out test set to the training data and oversample it up to 10× Omission Exclude unfavorable outcomes from the reported aggregate Report the best checkpoint instead of the required final one Synthetic substitution Swap the intended data, model, or evaluation for an easier surrogate Swap the required evaluation model for a smaller one and add a rewritten metric Skip the work Submit a trivial or recycled artifact Repackage cached predictions from prior methods as a new method in language model inference throughput optimization
Task-specific
Overfitting Exploit known properties of the test distribution Force predictions to match the test set’s known, balanced class distribution Measurement exploit Exploit what the scoring function measures Return dummy tokens on requests during benchmarking of language model inference (throughput 844→1238 tok/s)
Table 2: A taxonomy of reward hacks observed in autoresearch trajectories that we use for HackBench, grouped by scope (general vs. task-specific), with one example per category. General hacks apply across tasks; task-specific hacks exploit the measurement structure of a particular task. Examples are drawn from ResearchGym and AutoLab trajectories.
Figure 7: Construction of a HackBench example with a synthetic hack through trajectory editing. Step 1: We begin with a real autoresearch trajectory. Step 2: We insert a synthetic reward-hacking command (red). Step 3: We update only causally affected downstream observations (green), preserving the task, commands, and unrelated observations. This produces a coherent example with command-level audited gold annotations.
Per trajectory
Mean
Median
Min
Max
Total
Transcript length (thousand chars)
1,121.1
1,064.4
73.4
3,834.2
–
Commands
357.8
264
22
1,901
60,468
Hacked commands
7.8
1
1
383
1,312
Hacked share of commands (%)
4.4
0.6
0.1
31.5
–
Table 3: Hacks are sparse within long trajectories. Per-trajectory statistics across the 169 trajectories in HackBench. Trajectories contain a median of over one million characters and 264 commands, yet a median of only one hacked command (a median of 0.6% of their commands are hacked), so a monitor must find “needles” in a long context.
Model
Precision
Recall
F 1
Claude Opus 5
69.68
72.64
64.88
Muse Wonder
67.87
64.98
61.47
GPT-5.6 Sol
54.84
68.55
52.99
Kimi K3
56.55
54.10
52.66
Claude Opus 4.8
58.72
54.59
52.47
GPT-5.5
55.20
57.87
50.90
Table 4: Hack localization performance on HackBench. For each hacked trajectory, precision, recall, and F 1 compare the monitor’s predicted set of hacking command indices with the audited gold set.
Figure 8: AutoInteract is a two-phase framework for generating realistic, high-quality multi-turn interaction trajectories from verified tasks based on real user histories. In Phase 1, AutoInteract builds user models: modeling how users react by optimizing a history-conditioned user behavior prompt and building a diverse set of user profiles. In Phase 2, an agent combines verified source tasks with its user models to generate candidate tasks with multi-turn interaction trajectories, and selects them to be challenging, valid, realistic, coherent, and profile-consistent. If the checks fail, it revises the controller’s instruction until a task passes all checks or the retry budget is exhausted.
Figure 9: The user-simulator prompt used to generate each user turn. It combines the learned user behavior prompt and a sampled user profile from Phase 1 with the interaction history and the privileged task context. After a failed attempt, the controller may also add revised role-playing guidance based on the trajectory evaluator’s feedback.
Model
LiveCodeBench-Pro
Multi-turn LiveCodeBench-Pro
Δ vs Base
Easy
Medium
Overall
Easy
Medium
Overall
Avg
Qwen3.5-4B
61.6
7.6
36.8
40.0
3.5
23.0
29.9
–
+ RL on Source Data
61.4
8.8
37.1
59.7
11.8
36.7
36.9
+7.0
+ AutoInteract
68.5 +6.9
13.5 +5.9
42.8 +6.0
70.9 +30.9
13.1 +9.6
43.2 +20.2
43.0
+13.1
Qwen3.5-27B
85.7
37.5
60.9
59.1
14.0
37.2
49.1
–
+ RL on Source Data
88.2
34.7
61.2
61.5
12.7
38.0
49.6
+0.5
Table 5: Evaluation on algorithmic benchmarks. We report Pass@1 on LiveCodeBench-Pro and its multi-turn variant, broken down by difficulty. Avg is the unweighted mean of the two Overall scores, and Δ vs Base is its absolute pp gain over the corresponding baseline performance; green subscripts denote per-benchmark pp gains over the same baseline performance. The Hard subset is omitted because accuracy is consistently 0. Best per column within each backbone block in bold , second best underlined . AutoInteract achieves the best average at both model scales.
Model
Pass@1 ↑
SSR ↑
Pass 2 ↑
Mean Judge ↑
U-Corr ↓
Tok./Task
Min./Task
Qwen3.5-4B
6.0
1.8
1.8
0.197
3.78
23.9k
8.0
+ RL on Source Data
17.0
11.0
10.1
0.425
6.73
73.7k
39.7
+ AutoInteract
20.6 +14.6
16.5 +14.7
11.0 +9.2
0.462 +0.265
6.58
69.2k
25.1
Qwen3.5-27B
34.9
26.6
22.9
0.588
2.91
25.0k
32.2
+ RL on Source Data
27.1
22.0
17.4
0.545
2.40
19.2k
52.1
+ AutoInteract
38.5 +3.6
33.9 +7.3
26.6 +3.7
0.622 +0.034
3.44
29.7k
19.5
Table 6: Evaluation on SWE-Together. SWE-Together evaluates multi-turn repository-level issue resolution in collaboration with a simulated user. Each task is run twice, and a run counts as correct when the judge scores it at least 0.85 . Pass@1 is the percentage of correct runs, Pass 2 the percentage of tasks correct on both runs, and SSR the percentage of tasks whose two-run average reaches the threshold. Mean Judge is the average continuous correctness score, with no-patch runs scored 0 . U-Corr is user correction effort, computed as corrections plus 0.2× nudges from an LLM tagger that labels each follow-up user message. Tok./Task and Min./Task are task-averaged output and reasoning tokens and wall-clock minutes across the two runs. Green subscripts denote gains over the same baseline performance. Best and second-best correctness results within each backbone block are in bold and underlined , respectively. AutoInteract outperforms all baselines on every correctness metric.
Setting
Components
LiveCodeBench-Pro
Multi-turn LCB-Pro
SWE-Together
multi.
user.
exec.
Easy
Medium
Overall
Easy
Medium
Overall
Pass@1 ↑
SSR ↑
Pass 2 ↑
Avg
Avg
Qwen3.5-4B
–
–
–
61.6
7.6
36.8
40.0
3.5
23.0
6.0
1.8
1.8
3.2
21.0
AutoInteract
✓
✓
✓
68.5
13.5
42.8
70.9
13.1
43.2
20.6
16.5
11.0
16.0
34.0
w/o multi-turn
✗
✓
✓
64.2
10.8
39.4
49.7
6.1
29.2
18.3
16.5
12.8
15.9
28.2
w/o user models
✓
✗
✓
63.9
11.6
39.5
68.8
14.8
42.7
20.2
12.8
8.3
13.8
32.0
w/o executable verification
✓
✓
✗
48.1
4.0
28.0
58.2
7.9
34.4
18.3
11.0
8.3
12.5
25.0
Table 7: Multi-turn interaction data, grounded user models, and executable verification are all necessary in AutoInteract . We evaluate Qwen3.5-4B on the same three benchmarks as Tables 5 and 6 for various ablations. w/o multi-turn runs the full pipeline but builds each verified task as a single turn instead of a user–assistant interaction; w/o user models drops Phase 1 (Section 3.3 ) and simulates the user with one fixed prompt instead of the behavior-grounded profile population; w/o executable verification does not execute solver answers against the source verifier during data generation and instead uses an LLM judge to compare them with the ground truth; and w/o all three components combines these three ablations. The SWE-Together Avg is the unweighted mean of Pass@1, SSR, and Pass 2 ; the rightmost Avg is the unweighted mean of the two Overall scores and the SWE-Together Avg .
Figure 10: Training on AutoInteract tasks does not saturate as quickly as training on the source data. Training reward versus optimization step for Qwen3.5-4B RL runs on the source data (blue) and on AutoInteract tasks (orange). In the algorithmic domain (left), AutoInteract starts from a lower reward but continues improving and eventually reaches a higher reward than source-task training. In the SWE domain (right), AutoInteract also continues improving and reaches substantially higher reward, while source-task training saturates much earlier.
Figure 11: AutoInteract makes challenging multi-turn interaction data through revisions. The source task, learned user behavior prompt, and sampled profile remain fixed. Stage 1 points directly to the likely code change, making the trajectory too easy: the weak solver scores 67% , above the 40% limit. Evaluator feedback leads Stage 2 to remove that clue, but the revision also omits the example file and expected behavior, leaving too little evidence: the strong solver scores 67% , below the 70% target. Stage 3 restores this evidence, identifies the relevant code area, and adds follow-up requirements without revealing the solution. The final trajectory is accepted with weak- and strong-solver scores of 25% and 100% and passes all validity, leakage, and realism checks.
Figure 12: Standard judgments (left, middle) rank the model above the expert human—meta-optimizing the rubrics to be aligned with human experts reverses this (right). If AI slop is judged to be better than human writing by the LLM grader, then training will encourage more slop. On the academic paper section writing task, we find standard pairwise LLM judges or LLM-generated rubrics score GPT-5.6 or Opus-4.8 higher than high-quality human writing. Our RL-XAR (eXpert-Aligned Rubrics) approach instead meta-optimizes the rubrics on a training set of high-quality human-written sections, resulting in rubrics that reverse the result, and score the human writing higher on test papers.
Figure 13: Meta-rubric optimization. Left: mean expert human − model (Muse Spark 1.1) gap by iteration for the train and valid splits; the validation gap (solid) climbs from −4.2 to +2.76 , crossing zero (i.e., where humans are judged on average superior to the model) at iteration 4 and peaking at iteration 5, and roughly tracks the training gap (dashed) throughout—generalization to held-out papers, without overfitting. Right: validation absolute means—the gap increases from both directions, with the human’s score rising ( 3.4→5.0 ) as the model’s falls ( 7.6→2.7 ).
Figure 14: Comparison of Judges, averaged over XAR-optimized rubrics. Mean human − model gap (writer = Muse Spark 1.1; positive = human favored) for each judge, with judges in decreasing capability order. On Story and Wikipedia, Qwen3.5-27B (red) is the only judge to turn negative , ranking the model above human writing ( −0.55 story, −0.39 wiki); on Paper the four judges are close and all human-favoring. A weak grader is incapable of seeing the gap.
Figure 15: Different Rubric Meta-Optimizers. Mean (human − model) gap per optimizer, averaged over three seeds.
Figure 16: Human − model gap by paper human writing-quality bin (low/medium/high, light to dark), comparing with two models (Muse Spark 1.1 and GPT-5.6). If the learned rubric is measuring genuine quality, the human-model gap should be larger for better-written human papers and shrink for weak ones, which is what we observe.
Blind rubrics
GT rubrics
Rubric Judge
train
valid
train
valid
GPT-5.6 (cross-family)
+1.69
+1.79
+2.57
+2.49
Muse Spark 1.1 (same as writer)
+1.17
+1.35
+1.99
+2.51
Table 10: Cross-judge robustness of rubrics: mean gap (human − Muse Spark 1.1) on the paper section writing task, scored by the cross-family GPT-5.6 judge versus the Muse Spark 1.1 judge. The human is preferred under both judges on every split and variant. Ground Truth (GT) rubrics are built by including the hidden section as input when generating rubrics via the meta-prompt, thereby biasing scores towards the human-written completion. GT rubric scores are thus higher across the board.
Figure 17: Online RL-XAR : Training the writer (via the current rubric generator) and the rubric generator (using the human-model gap) online simultaneously. While this is not the approach reported in our main experiments, it appears to work well also, and future work should explore this approach further.
Feature
Qwen3.5-27B
RL-XAR
RL-XAR
Human
(Base)
LLM reward only
LLM + code reward
Paragraphs
6.92
5.10
3.55
5.72
Sentence-length mean
17.08
13.44
27.09
19.73
Sentence-length std
12.58
6.44
14.83
12.23
Word-length mean
4.28
4.31
4.05
4.11
Long-word frac ( ≥7 )
0.16
0.17
0.13
0.14
Table 11: Verifiable code-based rewards for surface-form distributional shifts . RL-XAR with surface-form code-based rewards moves the most distorted features (sentence length, paragraphs, semicolons) back toward the human distribution compared to plain RL-XAR without affecting LLM rubric-based rewards, and even outperforms the base model (Qwen3.5-27B) on these surface-form statistics.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Description
ResearchGym [ Garikaparthi et al., 2026 ]
continual-learning
Teach an image classifier a sequence of new classes without forgetting the earlier ones, on CIFAR-100 and other image benchmarks
cross-modal-retrieval
Retrieve the matching caption for an image and the matching image for a caption when the test queries differ from training
improving-replay-buffers
Improve how a reinforcement-learning agent reuses past experience to learn continuous-control tasks such as Hopper
materials-tokenization
Design a better way to split materials-science text into tokens so a language model identifies scientific terms more accurately than a published baseline
time-series-explanation
Build a method that identifies which parts of a time series explain a model’s prediction, tested on medical and sensor data
Appendix
Table 12: The 13 autoresearch tasks used to collect trajectories, grouped under their source dataset. Each description states, in plain terms, the objective the agent optimizes.
Task
Description
Exploit
ResearchGym
continual-learning
Teach an image classifier a sequence of new classes without forgetting the earlier ones, on CIFAR-100 and other image benchmarks.
Freeze the pretrained ViT and fit nearest-class-mean prototype heads on frozen features. AAA is a running mean of top-1 accuracy over classes seen so far, so it cannot tell “no forgetting because the method works” from “nothing was learned”—zero learning ties InfLoRA ( ∼ 86/91).
cross-modal-retrieval
Retrieve the matching caption for an image and the matching image for a caption when the test queries differ from training.
Register the frozen source model as the “adapted” method (no-op adaptation, or TTA lr=0 ): the score measures the retrieval model, not the adaptation, so the untouched source already beats published baselines. Running only the easy corruptions further inflates the mean over columns.
improving-replay-buffers
Improve how a reinforcement-learning agent reuses past experience to learn continuous-control tasks such as Hopper.
Optimize for survival only: a Hopper policy that stays upright but never moves collects ∼ 1000 return per episode from the alive-bonus alone (raw unnormalized return, 1000-step cap) — the metric’s local optimum, honestly summed.
materials-tokenization
Design a better way to split materials-science text into tokens so a language model identifies scientific terms more accurately than a published baseline.
Do no tokenization research—fine-tune stock WordPiece. The grader takes max() over the micro, macro, and all per-seed F 1 values against a floor set to one tokenizer’s micro -F 1 , so an off-the-shelf tokenizer clears it on the macro cell and more seeds only raise the max.
time-series-explanation
Build a method that identifies which parts of a time series explain a model’s prediction, tested on medical and sensor data.
Emit a near-max attribution everywhere to saturate AUR → 1.0; or exploit CPD (no ground truth; sums per-step ∣Δp∣ ) with an ordering that makes the prediction oscillate, accumulating arbitrarily large CPD. Neither explains anything.
Appendix
Table 13: Task-specific reward hacks for ResearchGym and AutoLab: the task’s objective and the task-specific exploit of its grader.
Figure 18: Meta-rubric optimization for the stories and Wikipedia domains. (optimizer Kimi-K2.6, judge and writer = Muse Spark 1.1); the meta-optimized rubrics are used for story and Wikipedia RL-XAR training. Each panel plots the mean expert-human − model gap by meta-prompt iteration for the train and validation splits (blind rubrics). Left (story): the validation gap rises from +0.16 and peaks around iteration 3 ( +2.77 ). Right (Wikipedia): the validation gap peaks around iteration 3 at only +1.07 and stays small—consistent with the far more modest Wikipedia gains.
Rubric (optimizer → judge)
GPT-5.6
Muse
Qwen3.8
Qwen3.5-27B
Paper writing
Opus 5 → Muse
+0.52
+0.81
+0.85
+0.81
Kimi K2.6 → Muse
+1.65
+2.76
+1.95
+2.15
Opus 5 → GPT-5.6
+1.79
+1.35
+0.84
+1.18
Story continuation
Opus 5 → GPT-5.6
+4.69
+1.77
+1.91
−0.29
Appendix
Table 14: Meta-Optimizer and Judge capability across all three domains.
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
AI agents are beginning to automate research and development across the AI stack, from improving training efficiency to optimizing inference. A natural next step is to improve the research efficiency of the agents themselves. When an AI research agent's own code is the object of optimization, each accepted rewrite becomes the agent that the next round edits. We refer to this loop as recursive self-improvement. Its significance lies in a long-standing trend, in which increased cumulative spending on R&D yields diminishing returns. Sustained self-improvement offers a way to counter this trend. We present AIDE^2, a system that implements this loop for a frontier AI research agent. It proposes changes to its own code, benchmarks modified versions of itself on a suite of AI R&D tasks, and keeps the changes that perform best on hidden evaluations. In an autonomous 8-day run, AIDE^2 discovered seven successive improvements, ranging from a new search policy to memory mechanisms that compress and manage the agent's growing context. These gains generalize to four held-out benchmarks spanning machine learning engineering, heuristic algorithm engineering, and physics-based weather forecasting, the last of which is out of distribution from the selection tasks. On all four, the strongest discovered agent matches or exceeds a human-engineered production research agent that ranks among the strongest on FML-Bench. On a separate held-out task family, the discovered agents also exhibit reduced reward hacking, a property the loop never explicitly optimized for: the rate falls from 55% to 32% during the run, 7 percentage points below the human-engineered agent. Together, these results show that an AI research agent can improve its own research efficiency through recursive self-improvement, and that these gains transfer to tasks and domains the loop never encountered.
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Yaolun Zhang, Tianyi Xu, Yujie Zhao +3
Oregon State University · AG2 AI · University of California, San Diego +1
As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Our data is released at https://github.com/AARR-bench/AARRI-bench.