cs.AIOct 28, 2026
SaveAligning Autoresearch: Co-Improvement between Agents and Humans
Organizations: FAIR at Meta · Johns Hopkins University · Carnegie Mellon University · University of Washington
Abstract
Autoresearch agents can now attempt long-horizon AI research end-to-end, i.e., self-improvement. However, best results are still obtained by research agents interacting with human researchers, and improving AI's ability to perform this co-research will only reinforce this effect---a process termed co-improvement (Weston and Foerster, 2025). Further, aligning with human expectations is likely a safer path for research agents, especially given recent agent reward hacking phenomena (OpenAI, 2026b). In this work we address these issues in four ways: (i) we empirically study the role of humans in directing autoresearch agents, focusing on the domain of benchmark creation, and find humans still play a pivotal role particularly in the ideation phase; (ii) we also develop a benchmark, HackBench, that shows current autoresearch agents are not aligned and are unable to achieve hack localization sufficiently; (iii) we present a method to train agents for improved human interaction by using human simulators as a proxy during reinforcement learning (RL); and finally (iv) we develop a method to improve model text generation, and hence agent communication, by RL with rubrics aligned with human experts. Together, our training and evaluation recipes provide a path toward more capable and aligned (co-)research agents.
Figures & tables
Figure 1: AutoBenchmark : evaluating the ability of autoresearch agents to create benchmarks . AutoBenchmark instructs an autoresearch agent to create a benchmark; we specifically target benchmarks that evaluate autoresearch agents themselves ( i.e. , benchmarking autoresearch). Within the benchmark creation loop, the autoresearch agent iteratively revises the benchmark via two main sources of feedback: one from the benchmark solvers (including the same model used as the benchmark creator) and the other from external verifiers of the benchmark (AI or human). We thus also study if having humans provide additional feedback on what benchmark to make and how to make it helps or not.
Figure 2: Performance of benchmark solvers and verdicts of benchmark judges across 10 iterations . A lower score means a harder benchmark. Muse Spark and Muse Glimmer are the solvers inside the loop, while Nemotron is held out. Filled markers passed all five quality judges and are eligible for selection; hollow markers were caught by at least one, and the star is the selected checkpoint, namely the eligible iteration with the lowest Muse Spark score. We display different levels of human feedback. The no-feedback curve is one autonomous run, in which the agent chooses the benchmark itself, shown in both rows as a reference. Feedback from humans improves over fully autonomous benchmark creation, with fine-grained feedback more useful than coarse-grained feedback.
| Proposal human feedback setting | Opus-5 Score ( ) |
| Autonomous run (agent-chosen benchmark), no feedback | |
| Rebuttal Bench, coarse-grained human feedback | 83.1 |
| Rebuttal Bench, fine-grained human feedback | 65.9 |
| [0.95pt/3pt] Graveyard Bench, coarse-grained human feedback | 90.7 |
| Graveyard Bench, fine-grained human feedback | 84.4 |
Table 1: Claude-Opus-5 score on the best checkpoints . Lower scores mean that the benchmark is harder. Fine-grained human feedback helps.
Figure 3: Execution feedback on SilentTrain Bench. Iterations to of a fine-grained proposal are run without extra human feedback, and then the experiment is forked at iteration : one continuation receives human guidance on how to instantiate the benchmark and the other continues unaided. The agent, the construct, and the grounding material are identical across the fork. Markers follow Figure 2 and is the gap between the two selected checkpoints on each solver. Extra expert human guidance helps.
Figure 4: Examples of critique from LLM judges: Feedback from LLM judges (benchmark verifiers) gives the research agent insights that solver scores ( i.e. , trajectory feedback) alone cannot provide, and enables making a valid benchmark while keeping it challenging.
Figure 5: HackBench. Hack localization F 1 per model. All monitors miss many hacks.
Figure 6: HackBench. Given a full agent trajectory (left), an LLM monitor (center) reads the entire transcript and is scored on the hack localization task of returning the exact offending command(s) by command index (right).
| Category What the agent does Example | |
| General | Fabrication Create fake or duplicated results that fool the grader Write fake result files reporting a perfect retrieval score Grader hijack Alter evaluation code or a dependency to inflate the score Edit the grader to keep only the best seed instead of the mean Label leakage Use held-out data in training, model selection, or prediction Add the held-out test set to the training data and oversample it up to Omission Exclude unfavorable outcomes from the reported aggregate Report the best checkpoint instead of the required final one Synthetic substitution Swap the intended data, model, or evaluation for an easier surrogate Swap the required evaluation model for a smaller one and add a rewritten metric Skip the work Submit a trivial or recycled artifact Repackage cached predictions from prior methods as a new method in language model inference throughput optimization |
| Task-specific | Overfitting Exploit known properties of the test distribution Force predictions to match the test set’s known, balanced class distribution Measurement exploit Exploit what the scoring function measures Return dummy tokens on requests during benchmarking of language model inference (throughput tok/s) |
Table 2: A taxonomy of reward hacks observed in autoresearch trajectories that we use for HackBench, grouped by scope (general vs. task-specific), with one example per category. General hacks apply across tasks; task-specific hacks exploit the measurement structure of a particular task. Examples are drawn from ResearchGym and AutoLab trajectories.
Figure 7: Construction of a HackBench example with a synthetic hack through trajectory editing. Step 1: We begin with a real autoresearch trajectory. Step 2: We insert a synthetic reward-hacking command (red). Step 3: We update only causally affected downstream observations (green), preserving the task, commands, and unrelated observations. This produces a coherent example with command-level audited gold annotations.
| Per trajectory | Mean | Median | Min | Max | Total |
| Transcript length (thousand chars) | 1,121.1 | 1,064.4 | 73.4 | 3,834.2 | – |
| Commands | 357.8 | 264 | 22 | 1,901 | 60,468 |
| Hacked commands | 7.8 | 1 | 383 | 1,312 | |
| Hacked share of commands (%) | 4.4 | 0.1 | 31.5 | – |
Table 3: Hacks are sparse within long trajectories. Per-trajectory statistics across the 169 trajectories in HackBench. Trajectories contain a median of over one million characters and 264 commands, yet a median of only one hacked command (a median of 0.6% of their commands are hacked), so a monitor must find “needles” in a long context.
| Model | Precision | Recall | F 1 |
| Claude Opus 5 | |||
| Muse Wonder | |||
| GPT-5.6 Sol | |||
| Kimi K3 | |||
| Claude Opus 4.8 | |||
| GPT-5.5 |
Table 4: Hack localization performance on HackBench. For each hacked trajectory, precision, recall, and F 1 compare the monitor’s predicted set of hacking command indices with the audited gold set.
Figure 8: AutoInteract is a two-phase framework for generating realistic, high-quality multi-turn interaction trajectories from verified tasks based on real user histories. In Phase 1, AutoInteract builds user models: modeling how users react by optimizing a history-conditioned user behavior prompt and building a diverse set of user profiles. In Phase 2, an agent combines verified source tasks with its user models to generate candidate tasks with multi-turn interaction trajectories, and selects them to be challenging, valid, realistic, coherent, and profile-consistent. If the checks fail, it revises the controller’s instruction until a task passes all checks or the retry budget is exhausted.
Figure 9: The user-simulator prompt used to generate each user turn. It combines the learned user behavior prompt and a sampled user profile from Phase 1 with the interaction history and the privileged task context. After a failed attempt, the controller may also add revised role-playing guidance based on the trajectory evaluator’s feedback.
| Model | LiveCodeBench-Pro | Multi-turn LiveCodeBench-Pro | vs Base | |||||
| Easy | Medium | Overall | Easy | Medium | Overall | Avg | ||
| Qwen3.5-4B | 61.6 | 7.6 | 36.8 | 40.0 | 3.5 | 23.0 | 29.9 | – |
| + RL on Source Data | 61.4 | 8.8 | 37.1 | 59.7 | 11.8 | 36.7 | 36.9 | +7.0 |
| + AutoInteract | 68.5 +6.9 | 13.5 +5.9 | 42.8 +6.0 | 70.9 +30.9 | 13.1 +9.6 | 43.2 +20.2 | 43.0 | +13.1 |
| Qwen3.5-27B | 85.7 | 37.5 | 60.9 | 59.1 | 14.0 | 37.2 | 49.1 | – |
| + RL on Source Data | 88.2 | 34.7 | 61.2 | 61.5 | 12.7 | 38.0 | 49.6 | +0.5 |
Table 5: Evaluation on algorithmic benchmarks. We report Pass@1 on LiveCodeBench-Pro and its multi-turn variant, broken down by difficulty. Avg is the unweighted mean of the two Overall scores, and vs Base is its absolute pp gain over the corresponding baseline performance; green subscripts denote per-benchmark pp gains over the same baseline performance. The Hard subset is omitted because accuracy is consistently 0. Best per column within each backbone block in bold , second best underlined . AutoInteract achieves the best average at both model scales.
| Model | Pass@1 | SSR | Pass 2 | Mean Judge | U-Corr | Tok./Task | Min./Task |
| Qwen3.5-4B | 6.0 | 1.8 | 1.8 | 0.197 | 3.78 | 23.9k | 8.0 |
| + RL on Source Data | 17.0 | 11.0 | 10.1 | 0.425 | 6.73 | 73.7k | 39.7 |
| + AutoInteract | 20.6 +14.6 | 16.5 +14.7 | 11.0 +9.2 | 0.462 +0.265 | 6.58 | 69.2k | 25.1 |
| Qwen3.5-27B | 34.9 | 26.6 | 22.9 | 0.588 | 2.91 | 25.0k | 32.2 |
| + RL on Source Data | 27.1 | 22.0 | 17.4 | 0.545 | 2.40 | 19.2k | 52.1 |
| + AutoInteract | 38.5 +3.6 | 33.9 +7.3 | 26.6 +3.7 | 0.622 +0.034 | 3.44 | 29.7k | 19.5 |
Table 6: Evaluation on SWE-Together. SWE-Together evaluates multi-turn repository-level issue resolution in collaboration with a simulated user. Each task is run twice, and a run counts as correct when the judge scores it at least . Pass@1 is the percentage of correct runs, Pass 2 the percentage of tasks correct on both runs, and SSR the percentage of tasks whose two-run average reaches the threshold. Mean Judge is the average continuous correctness score, with no-patch runs scored . U-Corr is user correction effort, computed as corrections plus nudges from an LLM tagger that labels each follow-up user message. Tok./Task and Min./Task are task-averaged output and reasoning tokens and wall-clock minutes across the two runs. Green subscripts denote gains over the same baseline performance. Best and second-best correctness results within each backbone block are in bold and underlined , respectively. AutoInteract outperforms all baselines on every correctness metric.
| Setting | Components | LiveCodeBench-Pro | Multi-turn LCB-Pro | SWE-Together | ||||||||||
| multi. | user. | exec. | Easy | Medium | Overall | Easy | Medium | Overall | Pass@1 | SSR | Pass 2 | Avg | Avg | |
| Qwen3.5-4B | – | – | – | 61.6 | 7.6 | 36.8 | 40.0 | 3.5 | 23.0 | 6.0 | 1.8 | 1.8 | 3.2 | 21.0 |
| AutoInteract | ✓ | ✓ | ✓ | 68.5 | 13.5 | 42.8 | 70.9 | 13.1 | 43.2 | 20.6 | 16.5 | 11.0 | 16.0 | 34.0 |
| w/o multi-turn | ✗ | ✓ | ✓ | 64.2 | 10.8 | 39.4 | 49.7 | 6.1 | 29.2 | 18.3 | 16.5 | 12.8 | 15.9 | 28.2 |
| w/o user models | ✓ | ✗ | ✓ | 63.9 | 11.6 | 39.5 | 68.8 | 14.8 | 42.7 | 20.2 | 12.8 | 8.3 | 13.8 | 32.0 |
| w/o executable verification | ✓ | ✓ | ✗ | 48.1 | 4.0 | 28.0 | 58.2 | 7.9 | 34.4 | 18.3 | 11.0 | 8.3 | 12.5 | 25.0 |
Table 7: Multi-turn interaction data, grounded user models, and executable verification are all necessary in AutoInteract . We evaluate Qwen3.5-4B on the same three benchmarks as Tables 5 and 6 for various ablations. w/o multi-turn runs the full pipeline but builds each verified task as a single turn instead of a user–assistant interaction; w/o user models drops Phase 1 (Section 3.3 ) and simulates the user with one fixed prompt instead of the behavior-grounded profile population; w/o executable verification does not execute solver answers against the source verifier during data generation and instead uses an LLM judge to compare them with the ground truth; and w/o all three components combines these three ablations. The SWE-Together Avg is the unweighted mean of Pass@1, SSR, and Pass 2 ; the rightmost Avg is the unweighted mean of the two Overall scores and the SWE-Together Avg .
Figure 10: Training on AutoInteract tasks does not saturate as quickly as training on the source data. Training reward versus optimization step for Qwen3.5-4B RL runs on the source data (blue) and on AutoInteract tasks (orange). In the algorithmic domain (left), AutoInteract starts from a lower reward but continues improving and eventually reaches a higher reward than source-task training. In the SWE domain (right), AutoInteract also continues improving and reaches substantially higher reward, while source-task training saturates much earlier.
Figure 11: AutoInteract makes challenging multi-turn interaction data through revisions. The source task, learned user behavior prompt, and sampled profile remain fixed. Stage 1 points directly to the likely code change, making the trajectory too easy: the weak solver scores , above the limit. Evaluator feedback leads Stage 2 to remove that clue, but the revision also omits the example file and expected behavior, leaving too little evidence: the strong solver scores , below the target. Stage 3 restores this evidence, identifies the relevant code area, and adds follow-up requirements without revealing the solution. The final trajectory is accepted with weak- and strong-solver scores of and and passes all validity, leakage, and realism checks.
Figure 12: Standard judgments (left, middle) rank the model above the expert human—meta-optimizing the rubrics to be aligned with human experts reverses this (right). If AI slop is judged to be better than human writing by the LLM grader, then training will encourage more slop. On the academic paper section writing task, we find standard pairwise LLM judges or LLM-generated rubrics score GPT-5.6 or Opus-4.8 higher than high-quality human writing. Our RL-XAR (eXpert-Aligned Rubrics) approach instead meta-optimizes the rubrics on a training set of high-quality human-written sections, resulting in rubrics that reverse the result, and score the human writing higher on test papers.
Figure 13: Meta-rubric optimization. Left: mean expert human model (Muse Spark 1.1) gap by iteration for the train and valid splits; the validation gap (solid) climbs from to , crossing zero (i.e., where humans are judged on average superior to the model) at iteration 4 and peaking at iteration 5, and roughly tracks the training gap (dashed) throughout—generalization to held-out papers, without overfitting. Right: validation absolute means—the gap increases from both directions, with the human’s score rising ( ) as the model’s falls ( ).
Figure 14: Comparison of Judges, averaged over XAR-optimized rubrics. Mean human model gap (writer Muse Spark 1.1; positive human favored) for each judge, with judges in decreasing capability order. On Story and Wikipedia, Qwen3.5-27B (red) is the only judge to turn negative , ranking the model above human writing ( story, wiki); on Paper the four judges are close and all human-favoring. A weak grader is incapable of seeing the gap.
Figure 15: Different Rubric Meta-Optimizers. Mean (human model) gap per optimizer, averaged over three seeds.
Figure 16: Human model gap by paper human writing-quality bin (low/medium/high, light to dark), comparing with two models (Muse Spark 1.1 and GPT-5.6). If the learned rubric is measuring genuine quality, the human-model gap should be larger for better-written human papers and shrink for weak ones, which is what we observe.
| Blind rubrics | GT rubrics | |||
| Rubric Judge | train | valid | train | valid |
| GPT-5.6 (cross-family) | ||||
| Muse Spark 1.1 (same as writer) | ||||
Table 10: Cross-judge robustness of rubrics: mean gap (human Muse Spark 1.1) on the paper section writing task, scored by the cross-family GPT-5.6 judge versus the Muse Spark 1.1 judge. The human is preferred under both judges on every split and variant. Ground Truth (GT) rubrics are built by including the hidden section as input when generating rubrics via the meta-prompt, thereby biasing scores towards the human-written completion. GT rubric scores are thus higher across the board.
Figure 17: Online RL-XAR : Training the writer (via the current rubric generator) and the rubric generator (using the human-model gap) online simultaneously. While this is not the approach reported in our main experiments, it appears to work well also, and future work should explore this approach further.
| Feature | Qwen3.5-27B | RL-XAR | RL-XAR | |
| Human | (Base) | LLM reward only | LLM + code reward | |
| Paragraphs | 6.92 | 5.10 | 3.55 | 5.72 |
| Sentence-length mean | 17.08 | 13.44 | 27.09 | 19.73 |
| Sentence-length std | 12.58 | 6.44 | 14.83 | 12.23 |
| Word-length mean | 4.28 | 4.31 | 4.05 | 4.11 |
| Long-word frac ( ) | 0.16 | 0.17 | 0.13 | 0.14 |
Table 11: Verifiable code-based rewards for surface-form distributional shifts . RL-XAR with surface-form code-based rewards moves the most distorted features (sentence length, paragraphs, semicolons) back toward the human distribution compared to plain RL-XAR without affecting LLM rubric-based rewards, and even outperforms the base model (Qwen3.5-27B) on these surface-form statistics.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Description |
|---|---|
| ResearchGym [ Garikaparthi et al., 2026 ] | |
| continual-learning | Teach an image classifier a sequence of new classes without forgetting the earlier ones, on CIFAR-100 and other image benchmarks |
| cross-modal-retrieval | Retrieve the matching caption for an image and the matching image for a caption when the test queries differ from training |
| improving-replay-buffers | Improve how a reinforcement-learning agent reuses past experience to learn continuous-control tasks such as Hopper |
| materials-tokenization | Design a better way to split materials-science text into tokens so a language model identifies scientific terms more accurately than a published baseline |
| time-series-explanation | Build a method that identifies which parts of a time series explain a model’s prediction, tested on medical and sensor data |
Table 12: The 13 autoresearch tasks used to collect trajectories, grouped under their source dataset. Each description states, in plain terms, the objective the agent optimizes.
| Task | Description | Exploit |
|---|---|---|
| ResearchGym | ||
| continual-learning | Teach an image classifier a sequence of new classes without forgetting the earlier ones, on CIFAR-100 and other image benchmarks. | Freeze the pretrained ViT and fit nearest-class-mean prototype heads on frozen features. AAA is a running mean of top-1 accuracy over classes seen so far, so it cannot tell “no forgetting because the method works” from “nothing was learned”—zero learning ties InfLoRA ( 86/91). |
| cross-modal-retrieval | Retrieve the matching caption for an image and the matching image for a caption when the test queries differ from training. | Register the frozen source model as the “adapted” method (no-op adaptation, or TTA lr=0 ): the score measures the retrieval model, not the adaptation, so the untouched source already beats published baselines. Running only the easy corruptions further inflates the mean over columns. |
| improving-replay-buffers | Improve how a reinforcement-learning agent reuses past experience to learn continuous-control tasks such as Hopper. | Optimize for survival only: a Hopper policy that stays upright but never moves collects 1000 return per episode from the alive-bonus alone (raw unnormalized return, 1000-step cap) — the metric’s local optimum, honestly summed. |
| materials-tokenization | Design a better way to split materials-science text into tokens so a language model identifies scientific terms more accurately than a published baseline. | Do no tokenization research—fine-tune stock WordPiece. The grader takes max() over the micro, macro, and all per-seed F 1 values against a floor set to one tokenizer’s micro -F 1 , so an off-the-shelf tokenizer clears it on the macro cell and more seeds only raise the max. |
| time-series-explanation | Build a method that identifies which parts of a time series explain a model’s prediction, tested on medical and sensor data. | Emit a near-max attribution everywhere to saturate AUR 1.0; or exploit CPD (no ground truth; sums per-step ) with an ordering that makes the prediction oscillate, accumulating arbitrarily large CPD. Neither explains anything. |
Table 13: Task-specific reward hacks for ResearchGym and AutoLab: the task’s objective and the task-specific exploit of its grader.
Figure 18: Meta-rubric optimization for the stories and Wikipedia domains. (optimizer Kimi-K2.6, judge and writer Muse Spark 1.1); the meta-optimized rubrics are used for story and Wikipedia RL-XAR training. Each panel plots the mean expert-human model gap by meta-prompt iteration for the train and validation splits (blind rubrics). Left (story): the validation gap rises from and peaks around iteration 3 ( ). Right (Wikipedia): the validation gap peaks around iteration 3 at only and stays small—consistent with the far more modest Wikipedia gains.
| Rubric (optimizer judge) | GPT-5.6 | Muse | Qwen3.8 | Qwen3.5-27B |
| Paper writing | ||||
| Opus 5 Muse | ||||
| Kimi K2.6 Muse | ||||
| Opus 5 GPT-5.6 | ||||
| Story continuation | ||||
| Opus 5 GPT-5.6 | ||||
Table 14: Meta-Optimizer and Judge capability across all three domains.
More in this topic
Autonomous Research Agents
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
See all 326 papers