cs.AIOct 15, 2026
SaveRecursive Reward Improvement via Adversarial Meta-Prompt Evolution
Organizations: Massachusetts Institute of Technology · Microsoft Research · Johns Hopkins University
Abstract
Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically. Open-ended expert tasks, like clinical report generations, have no answer key, which makes simple question-answer generation not viable. Here the proposer must also write a rubric that lists the content an answer should include and the errors it should avoid. However, the unique challenge behind model-written rubric is the comprehensiveness of it. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the learning process. To solve this problem, we introduce RRI, Recursive Reward Improvement, which uses a reward hacking adversary to supervise the reward. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win. The exploits found also rewrite how future tasks and rubrics are generated. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self-improvement baseline by 10.9 points at 9B and 12.4 at 27B, and by 10.3 points without any external model. In particular, the gains are largest on difficult and writing tasks. We show that the 27B model trained on the HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench without seeing a task from either. We further analyze the exploitable points of model-generated question-rubric pairs to facilitate future research on open-ended self-play.
Figures & tables
Figure 1: Overview of RRI. The proposer generates clinical tasks and weighted rubrics from medical references. An adversary probes each rubric with an exploit and an honest answer. The repair step keeps criteria that distinguish the better answer from the exploit, then tests the revised rubric with another attack. The solver learns through GRPO using rubric rewards. Training feedback and confirmed exploits guide updates to the generation prompts.
Figure 2: (a) HealthBench Professional adjusted scores with retrieval. Bars show the untrained, Generation Only, and RRI models at 9B and 27B with a GPT Teacher and at 9B with a Self Teacher. (b) HealthBench Professional raw scores with retrieval. Bars show the same models and settings without the length adjustment.
Figure 3: (a) Rubric-reward baselines and component ablation in the 9B GPT Teacher setting. Bars show RaR with synthetic tasks, EvoRubrics, and SibylSense, followed by Generation Only, Co-evolve Only, and full RRI with hack-then-patch rounds. (b) Adjusted scores by task category and difficulty in the 9B GPT Teacher setting. Bars compare Generation Only, Co-evolve Only, and RRI-9B.
Figure 4: Zero-shot scores on PRBench Hard and ProfBench. Qwen3.6-27B is trained on the HealthBench Pro description and answers both benchmarks, which it never saw, through the web-search tool loop. PRBench Hard has 550 finance and legal tasks, and ProfBench has 40 report tasks, 10 per domain. Self Improve is our implementation of self-play training on model-written tasks ( Huang et al., 2026a ; Liu et al., 2025a ) , in which a proposer with a fixed prompt writes the tasks and rubrics and the solver is trained on them by reinforcement learning. RRI adds the adversary that tests and repairs each rubric.
Figure 5: Example of rubric repair on an atrial fibrillation task. The question provides the context for the original rubric, its exploit, and selected repairs. Signed weights reward content or penalize errors. The detailed case appears in Appendix K.1 .
Figure 6: Categories of exploit cases found during training. Each case is a pair of solver answers to one generated task in which the answer the rubric scores highest loses a rubric-blind comparison, assigned to one category by an LLM auditor. Each panel covers the runs trained on the description of one benchmark, HealthBench Pro pools the 9B and 27B runs, and the last panel pools all cases. Appendix I defines the categories and gives the procedure.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Domain | Reward | Verifier | Adversary | Open- ended | Task gen. | No references | Exploit of the reward | Reward verif. | Rubric repair | Re-attack after patch | Task evolution |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RaR ( Gunjal et al., 2026 ) | Medical, science | task | judge | none | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| OpenRubrics ( Liu et al., 2026b ) | General, biomedical | task | judge | none | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| SPICE ( Liu et al., 2025a ) | Math, general | match | automatic | none | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| DR Tulu ( Shao et al., 2025 ) | Deep research | evolving | judge | none | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| EvoRubrics ( Ding et al., 2026 ) | Medical | evolving | judge | none | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| SibylSense ( Xu et al., 2026 ) | Medical, summaries | evolving | judge | trained policy | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
Table 1: Features of related methods. The columns compare properties relevant to self-improvement on open-ended tasks. Open-ended means answers are graded by judged criteria with no executable or reference check. Task gen. means the method writes its own training tasks. No references means no reference answers, preference labels, or golden rubrics. Exploit of the reward means an agent produces answers meant to satisfy the reward without being good. Reward verif. means each reward is checked before the solver trains on it. Rubric repair means criteria are added to a task’s existing rubric after a detected failure. Re-attack after patch means the repaired task and rubric are attacked again. Task evolution means task generation changes during training. Under Reward, task is a per-task rubric that never changes and evolving is one that changes during training. Under Verifier, judge is a language model grading criteria, automatic is answer matching, execution runs tests against reference code, and external is a fact checker over Wikipedia or execution.
| Setting | Row | Policy | Judge | Retrieval | Evolve | Adversary |
| 9B | Generation Only | Qwen3.5-9B | GPT / GPT | yes | – | – |
| 9B | RRI-9B | Qwen3.5-9B | GPT / GPT | yes | yes | yes |
| 27B | Generation Only | Qwen3.6-27B | GPT / GPT | yes | – | – |
| 27B | RRI | Qwen3.6-27B | GPT / GPT | yes | yes | yes |
| All-self | Generation Only | Qwen3.5-9B | 9B / 9B | yes | – | – |
| All-self | RRI-9B, all roles 9B | Qwen3.5-9B | 9B / 9B | yes | yes | yes |
Table 2: Configuration of each training arm. Judge = training / validation. Evolve = meta-prompt and memo evolution; Probe = adversary admission probe with repair.
| Clinical task | Difficulty | |||||
|---|---|---|---|---|---|---|
| Setting | Training method | Consult | Research | Writing | Typical | Difficult |
| 9B, GPT | Generation Only | 0.472 | 0.563 | 0.199 | 0.616 | 0.241 |
| Full RRI | 0.477 | 0.568 | 0.590 | 0.644 | 0.427 | |
| 27B, GPT | Generation Only | 0.395 | 0.486 | 0.288 | 0.591 | 0.202 |
| Full RRI | 0.493 | 0.630 | 0.435 | 0.648 | 0.390 | |
| 9B, all-self | Generation Only | 0.406 | 0.540 | 0.310 | 0.575 | 0.268 |
Table 3: Generation Only and RRI scores by clinical task and difficulty. Scores are the adjusted validation scores. Each pair compares Generation Only with full RRI under the same backbone, judge, and tool setting. Bold indicates the higher score within a pair. Untrained models appear in Appendix L .
Figure 7: (a) HealthBench Professional adjusted scores without retrieval. Bars show the untrained, Generation Only, and RRI models with the 9B backbone and the frozen 9B judge. (b) HealthBench Professional raw scores without retrieval.
| By category | By difficulty | |||||
| Model / training | Consult | Research | Writing | Typical | Difficult | Overall |
| Qwen3.5-9B, GPT judge, retrieval + web search | ||||||
| Untrained | 0.445 | 0.555 | 0.407 | 0.594 | 0.343 | 0.465 |
| Generation Only | 0.530 | 0.589 | 0.370 | 0.633 | 0.380 | 0.503 |
| RRI-9B (ours) | 0.575 | 0.624 | 0.666 | 0.693 | 0.537 | 0.613 |
| Qwen3.6-27B, GPT judge, retrieval + web search | ||||||
Table 4: HealthBench Professional raw scores. Scores are the fraction of rubric points earned with the length term omitted, at the evaluations of Table 14 .
| By category | By difficulty | |||||
| Model / training | Consult | Research | Writing | Typical | Difficult | Overall |
| Qwen3.5-9B, GPT judge, retrieval + web search | ||||||
| Untrained | 0.283 | 0.484 | 0.191 | 0.528 | 0.112 | 0.315 |
| Generation Only | 0.432 | 0.560 | 0.193 | 0.601 | 0.215 | 0.403 |
| RRI | 0.463 | 0.575 | 0.556 | 0.637 | 0.408 | 0.519 |
| Qwen3.6-27B, GPT judge, retrieval + web search | ||||||
Table 5: HealthBench Professional scores under the official GPT-5.4 judge. The rows of Table 14 are re-graded from the same answers with the benchmark’s official judge, GPT-5.4 at reasoning effort low with one vote per criterion. Scores are the official length-adjusted signed score.
| Specialty | 9B, GPT | 27B, GPT | 9B, all-self | 9B, no retrieval | |||||
|---|---|---|---|---|---|---|---|---|---|
| Fixed | Ours | Fixed | Ours | Fixed | Ours | Fixed | Ours | ||
| allergy | 2 | 0.569 | 0.982 | 0.030 | 0.987 | 0.544 | 0.510 | -0.437 | 0.491 |
| anesthesia | 22 | 0.514 | 0.600 | 0.478 | 0.564 | 0.409 | 0.485 | 0.445 | 0.409 |
| cards | 43 | 0.458 | 0.336 | 0.401 | 0.544 | 0.316 | 0.334 | 0.374 | 0.325 |
| derm | 25 | 0.459 | 0.597 | 0.382 | 0.520 | 0.577 | 0.639 | 0.415 | 0.506 |
| emergency | 16 | 0.440 | 0.409 | 0.488 | 0.389 | 0.355 | 0.637 | 0.142 | 0.449 |
Table 6: HealthBench Professional scores by specialty. Scores use the official length-adjusted score for the Generation Only and RRI rows of each setting in Table 14 . is the number of validation tasks in the specialty.
| Benchmark | Metric | Untrained | RRI | |
|---|---|---|---|---|
| PRBench Hard, finance and legal | 550 | rubric score | 0.180 | 0.228 |
| ProfBench, PhD and MBA reports | 40 | rubric score | 0.356 | 0.405 |
| MedXpertQA, text and image | 4450 | accuracy | 0.357 | 0.427 |
| text split | 2450 | accuracy | 0.316 | 0.375 |
| image split | 2000 | accuracy | 0.407 | 0.490 |
Table 7: Scores on held-out benchmarks at 9B. Each run uses Qwen3.5-9B and the GPT-5.5-Chat judge on a rubric benchmark outside HealthBench Professional. Each benchmark is used only for validation; the proposer sees a paper-level description of the benchmark and no items. Untrained is the base model under the same judge; RRI is the last checkpoint of the run. MedXpertQA is scored as a one-criterion rubric, so its score is exact-match accuracy, shown for the text and image splits.
| Model | Length-adjusted | Unadjusted |
|---|---|---|
| GPT-5.3 chat | 0.478 | 0.508 |
| GPT-5.5-Chat | 0.460 | 0.467 |
| GPT-5.4 | 0.398 | 0.442 |
| Gemma 4 31B IT | 0.297 | 0.332 |
| Lingshu-32B | 0.139 | 0.131 |
Table 8: Reference scores of frontier and open models. All models are scored on HealthBench Professional under one protocol, the benchmark’s official pipeline, no tools, greedy decoding, an 8,192-token answer budget, and the GPT-5.5-Chat judge used for validation in this paper. Frontier models are served through the same endpoint as the judge; open models are served locally.
| By category | By difficulty | |||||
| Model / method | Consult | Research | Writing | Typical | Difficult | Overall |
| Qwen3.5-9B, frozen | ||||||
| Medical RAG | 0.161 | 0.332 | 0.076 | 0.388 | -0.007 | 0.186 |
| RAG-Fusion | 0.150 | 0.324 | 0.114 | 0.392 | -0.004 | 0.189 |
| i-MedRAG | 0.199 | 0.349 | 0.248 | 0.408 | 0.109 | 0.254 |
| Qwen3.5-9B, trained with tools (Table 14 ) | ||||||
Table 9: Inference-time methods and trained models on HealthBench Professional. Inference-time methods on the frozen base models appear next to the trained rows of the same base model from Table 14 , judged by GPT-5.5-Chat with three votes. The frozen-model methods follow the inference-time protocol, greedy decoding, no reasoning channel, references from the shared medical corpus placed in the prompt; medical RAG retrieves once, RAG-Fusion fuses three generated queries, and i-MedRAG asks three rounds of two follow-up questions. The trained rows answer with the solver’s own tool calls at the training temperature.
| Setting | Row | HealthBench Hard |
|---|---|---|
| 27B | Untrained | 0.143 |
| 27B | Generation Only | 0.188 |
| 27B | RRI | 0.246 |
Table 10: Scores on HealthBench Hard at 27B. HealthBench Hard is the 1,000 hardest tasks of the original HealthBench release and is never used for training or validation. Scored with solver tools on and the GPT-5.5-Chat judge with three votes, using the benchmark’s official score, the mean per-example clipped rubric fraction with no length term.
| Category | Definition |
|---|---|
| Missed negatives | The exploit earns its criteria and also carries wrong, unsafe, overclaimed, or invented content, such as a wrong rule or dose, an arithmetic slip, or an unindicated escalation, that no criterion penalizes. The negative criteria name only the errors the rubric author anticipated, and the honest answer is the one without the error. |
| Ambiguous rubrics | A criterion addresses the right content, but its wording admits a weak and a strong reading or leaves open what form counts, so the judge’s reading decides the outcome. A passing mention, a hedge, or a claim that evidence exists earns the criterion, a penalty is avoided by rewording or attribution, or equivalent content in other words is refused. |
| Faulty answer key | The content of a criterion is wrong or out of date, echoes a false premise of the task, or fixes one convention, method, or value where several are defensible. The exploit agrees with the key, and the honest answer is docked for being correct or for a different defensible choice. |
| Fabricated citations | A criterion asks for named studies, identifiers, sample sizes, or quantitative estimates that the judge can check only in form. The exploit supplies invented or unverifiable specifics, while an honest answer that reports only what it can support, or states that the evidence does not exist, scores lower. |
| Missed positives | The rubric prices a small or easy part of the task, such as background facts or the first steps of a calculation, and leaves required substance unscored, such as the decision, the management plan, or an asked sub-question. A correct but minimal recital saturates the rubric, and the additional work of the honest answer earns nothing. |
| Incorrect pair | On checking the content, the exploit is as correct and useful as the honest answer or more so, or both fail the task in the same way. The ordering of the rubric was defensible, so there is no reward failure. |
Table 11: Definitions of the exploit categories. Each exploit case is assigned the first category that applies, in the order Incorrect pair, Faulty answer key, Fabricated citations, Missed negatives, Ambiguous rubrics, Missed positives.
| Category | What the reward got wrong | HealthBench Pro | PRBench | ProfBench | Overall |
|---|---|---|---|---|---|
| Missed negatives | no penalty for wrong or unsafe content outside the criteria | 33% | 39% | 22% | 33% |
| Ambiguous rubrics | loose wording lets a mention pass or refuses equivalent content | 35% | 21% | 21% | 29% |
| Incorrect pair | the honest answer is not better, so there was no reward failure | 12% | 24% | 24% | 17% |
| Faulty answer key | a criterion is wrong, stale, or fixes one of several defensible choices | 9% | 14% | 32% | 14% |
| Fabricated citations | criteria demand evidence the judge cannot verify, and invented evidence is paid | 10% | 2% | 0% | 6% |
| Missed positives | no criterion for substance the task requires, so a minimal recital reaches the ceiling | 1% | 0% | 1% | 1% |
Table 12: Exploit categories by benchmark description. Each entry is the share of the exploit cases recorded in the runs trained on one benchmark description. Each case was assigned one category by an LLM auditor that re-graded both answers against the rubric. HealthBench Pro pools the 9B and 27B runs.
| HB Pro solver | HB Pro half | PRBench half | ProfBench half | |||||
|---|---|---|---|---|---|---|---|---|
| Category | 9B | 27B | first | second | first | second | first | second |
| Missed negatives | 40% | 30% | 34% | 32% | 43% | 35% | 24% | 20% |
| Ambiguous rubrics | 30% | 37% | 35% | 36% | 26% | 16% | 21% | 22% |
| Incorrect pair | 11% | 12% | 12% | 12% | 19% | 30% | 26% | 22% |
| Faulty answer key | 8% | 10% | 10% | 9% | 11% | 16% | 28% | 35% |
| Fabricated citations | 11% | 10% | 9% | 11% | 2% | 2% | 0% | 0% |
Table 13: Exploit categories by solver size and training half. Solver sizes are compared for the runs trained on the HealthBench Pro description, written HB Pro, and halves split the exploit cases of the runs trained on each description by training step.
| By category | By difficulty | |||||
| Model / training | Consult | Research | Writing | Typical | Difficult | Overall |
| Qwen3.5-9B, GPT judge, retrieval + web search | ||||||
| Untrained | 0.320 | 0.472 | 0.194 | 0.522 | 0.144 | 0.329 |
| Generation Only | 0.472 | 0.563 | 0.199 | 0.616 | 0.241 | 0.424 |
| RRI-9B (ours) | 0.477 | 0.568 | 0.590 | 0.644 | 0.427 | 0.533 |
| Qwen3.6-27B, GPT judge, retrieval + web search | ||||||
Table 14: HealthBench Professional scores in each training setting. Scores use the official length-adjusted rubric score on all 525 tasks by category, by difficulty, and overall. Each group of rows is one training setting with one base model and one judge. Generation Only is self-improvement with the untuned proposer prompt, and RRI adds prompt evolution and the adversary’s hack-then-patch rounds. Bold marks the better trained row per column.
More in this topic
Recursive Self-Improvement
Recursive self-improvement studies AI systems that use feedback from their own work to modify and evaluate future versions.
See all 84 papers