Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically. Open-ended expert tasks, like clinical report generations, have no answer key, which makes simple question-answer generation not viable. Here the proposer must also write a rubric that lists the content an answer should include and the errors it should avoid. However, the unique challenge behind model-written rubric is the comprehensiveness of it. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the learning process. To solve this problem, we introduce RRI, Recursive Reward Improvement, which uses a reward hacking adversary to supervise the reward. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win. The exploits found also rewrite how future tasks and rubrics are generated. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self-improvement baseline by 10.9 points at 9B and 12.4 at 27B, and by 10.3 points without any external model. In particular, the gains are largest on difficult and writing tasks. We show that the 27B model trained on the HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench without seeing a task from either. We further analyze the exploitable points of model-generated question-rubric pairs to facilitate future research on open-ended self-play.
Figures & tables
Figure 1: Overview of RRI. The proposer generates clinical tasks and weighted rubrics from medical references. An adversary probes each rubric with an exploit and an honest answer. The repair step keeps criteria that distinguish the better answer from the exploit, then tests the revised rubric with another attack. The solver learns through GRPO using rubric rewards. Training feedback and confirmed exploits guide updates to the generation prompts.
Figure 2: (a) HealthBench Professional adjusted scores with retrieval. Bars show the untrained, Generation Only, and RRI models at 9B and 27B with a GPT Teacher and at 9B with a Self Teacher. (b) HealthBench Professional raw scores with retrieval. Bars show the same models and settings without the length adjustment.
Figure 3: (a) Rubric-reward baselines and component ablation in the 9B GPT Teacher setting. Bars show RaR with synthetic tasks, EvoRubrics, and SibylSense, followed by Generation Only, Co-evolve Only, and full RRI with hack-then-patch rounds. (b) Adjusted scores by task category and difficulty in the 9B GPT Teacher setting. Bars compare Generation Only, Co-evolve Only, and RRI-9B.
Figure 4: Zero-shot scores on PRBench Hard and ProfBench. Qwen3.6-27B is trained on the HealthBench Pro description and answers both benchmarks, which it never saw, through the web-search tool loop. PRBench Hard has 550 finance and legal tasks, and ProfBench has 40 report tasks, 10 per domain. Self Improve is our implementation of self-play training on model-written tasks ( Huang et al., 2026a ; Liu et al., 2025a ) , in which a proposer with a fixed prompt writes the tasks and rubrics and the solver is trained on them by reinforcement learning. RRI adds the adversary that tests and repairs each rubric.
Figure 5: Example of rubric repair on an atrial fibrillation task. The question provides the context for the original rubric, its exploit, and selected repairs. Signed weights reward content or penalize errors. The detailed case appears in Appendix K.1 .
Figure 6: Categories of exploit cases found during training. Each case is a pair of solver answers to one generated task in which the answer the rubric scores highest loses a rubric-blind comparison, assigned to one category by an LLM auditor. Each panel covers the runs trained on the description of one benchmark, HealthBench Pro pools the 9B and 27B runs, and the last panel pools all cases. Appendix I defines the categories and gives the procedure.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Domain
Reward
Verifier
Adversary
Open- ended
Task gen.
No references
Exploit of the reward
Reward verif.
Rubric repair
Re-attack after patch
Task evolution
RaR ( Gunjal et al., 2026 )
Medical, science
task
judge
none
✓
✗
✗
✗
✗
✗
✗
✗
OpenRubrics ( Liu et al., 2026b )
General, biomedical
task
judge
none
✓
✗
✗
✗
✓
✗
✗
✗
SPICE ( Liu et al., 2025a )
Math, general
match
automatic
none
✗
✓
✓
✗
✗
✗
✗
✓
DR Tulu ( Shao et al., 2025 )
Deep research
evolving
judge
none
✓
✗
✓
✗
✗
✓
✗
✗
EvoRubrics ( Ding et al., 2026 )
Medical
evolving
judge
none
✓
✗
✗
✗
✗
✗
✗
✗
SibylSense ( Xu et al., 2026 )
Medical, summaries
evolving
judge
trained policy
✓
✗
✗
✓
✗
✗
✗
✗
Appendix
Table 1: Features of related methods. The columns compare properties relevant to self-improvement on open-ended tasks. Open-ended means answers are graded by judged criteria with no executable or reference check. Task gen. means the method writes its own training tasks. No references means no reference answers, preference labels, or golden rubrics. Exploit of the reward means an agent produces answers meant to satisfy the reward without being good. Reward verif. means each reward is checked before the solver trains on it. Rubric repair means criteria are added to a task’s existing rubric after a detected failure. Re-attack after patch means the repaired task and rubric are attacked again. Task evolution means task generation changes during training. Under Reward, task is a per-task rubric that never changes and evolving is one that changes during training. Under Verifier, judge is a language model grading criteria, automatic is answer matching, execution runs tests against reference code, and external is a fact checker over Wikipedia or execution.
Setting
Row
Policy
Judge
Retrieval
Evolve
Adversary
9B
Generation Only
Qwen3.5-9B
GPT / GPT
yes
–
–
9B
RRI-9B
Qwen3.5-9B
GPT / GPT
yes
yes
yes
27B
Generation Only
Qwen3.6-27B
GPT / GPT
yes
–
–
27B
RRI
Qwen3.6-27B
GPT / GPT
yes
yes
yes
All-self
Generation Only
Qwen3.5-9B
9B / 9B
yes
–
–
All-self
RRI-9B, all roles 9B
Qwen3.5-9B
9B / 9B
yes
yes
yes
Appendix
Table 2: Configuration of each training arm. Judge = training / validation. Evolve = meta-prompt and memo evolution; Probe = adversary admission probe with repair.
Clinical task
Difficulty
Setting
Training method
Consult
Research
Writing
Typical
Difficult
9B, GPT
Generation Only
0.472
0.563
0.199
0.616
0.241
Full RRI
0.477
0.568
0.590
0.644
0.427
27B, GPT
Generation Only
0.395
0.486
0.288
0.591
0.202
Full RRI
0.493
0.630
0.435
0.648
0.390
9B, all-self
Generation Only
0.406
0.540
0.310
0.575
0.268
Appendix
Table 3: Generation Only and RRI scores by clinical task and difficulty. Scores are the adjusted validation scores. Each pair compares Generation Only with full RRI under the same backbone, judge, and tool setting. Bold indicates the higher score within a pair. Untrained models appear in Appendix L .
Figure 7: (a) HealthBench Professional adjusted scores without retrieval. Bars show the untrained, Generation Only, and RRI models with the 9B backbone and the frozen 9B judge. (b) HealthBench Professional raw scores without retrieval.
By category
By difficulty
Model / training
Consult
Research
Writing
Typical
Difficult
Overall
Qwen3.5-9B, GPT judge, retrieval + web search
Untrained
0.445
0.555
0.407
0.594
0.343
0.465
Generation Only
0.530
0.589
0.370
0.633
0.380
0.503
RRI-9B (ours)
0.575
0.624
0.666
0.693
0.537
0.613
Qwen3.6-27B, GPT judge, retrieval + web search
Appendix
Table 4: HealthBench Professional raw scores. Scores are the fraction of rubric points earned with the length term omitted, at the evaluations of Table 14 .
By category
By difficulty
Model / training
Consult
Research
Writing
Typical
Difficult
Overall
Qwen3.5-9B, GPT judge, retrieval + web search
Untrained
0.283
0.484
0.191
0.528
0.112
0.315
Generation Only
0.432
0.560
0.193
0.601
0.215
0.403
RRI
0.463
0.575
0.556
0.637
0.408
0.519
Qwen3.6-27B, GPT judge, retrieval + web search
Appendix
Table 5: HealthBench Professional scores under the official GPT-5.4 judge. The rows of Table 14 are re-graded from the same answers with the benchmark’s official judge, GPT-5.4 at reasoning effort low with one vote per criterion. Scores are the official length-adjusted signed score.
Specialty
n
9B, GPT
27B, GPT
9B, all-self
9B, no retrieval
Fixed
Ours
Fixed
Ours
Fixed
Ours
Fixed
Ours
allergy
2
0.569
0.982
0.030
0.987
0.544
0.510
-0.437
0.491
anesthesia
22
0.514
0.600
0.478
0.564
0.409
0.485
0.445
0.409
cards
43
0.458
0.336
0.401
0.544
0.316
0.334
0.374
0.325
derm
25
0.459
0.597
0.382
0.520
0.577
0.639
0.415
0.506
emergency
16
0.440
0.409
0.488
0.389
0.355
0.637
0.142
0.449
Appendix
Table 6: HealthBench Professional scores by specialty. Scores use the official length-adjusted score for the Generation Only and RRI rows of each setting in Table 14 . n is the number of validation tasks in the specialty.
Benchmark
n
Metric
Untrained
RRI
PRBench Hard, finance and legal
550
rubric score
0.180
0.228
ProfBench, PhD and MBA reports
40
rubric score
0.356
0.405
MedXpertQA, text and image
4450
accuracy
0.357
0.427
text split
2450
accuracy
0.316
0.375
image split
2000
accuracy
0.407
0.490
Appendix
Table 7: Scores on held-out benchmarks at 9B. Each run uses Qwen3.5-9B and the GPT-5.5-Chat judge on a rubric benchmark outside HealthBench Professional. Each benchmark is used only for validation; the proposer sees a paper-level description of the benchmark and no items. Untrained is the base model under the same judge; RRI is the last checkpoint of the run. MedXpertQA is scored as a one-criterion rubric, so its score is exact-match accuracy, shown for the text and image splits.
Model
Length-adjusted
Unadjusted
GPT-5.3 chat
0.478
0.508
GPT-5.5-Chat
0.460
0.467
GPT-5.4
0.398
0.442
Gemma 4 31B IT
0.297
0.332
Lingshu-32B
0.139
0.131
Appendix
Table 8: Reference scores of frontier and open models. All models are scored on HealthBench Professional under one protocol, the benchmark’s official pipeline, no tools, greedy decoding, an 8,192-token answer budget, and the GPT-5.5-Chat judge used for validation in this paper. Frontier models are served through the same endpoint as the judge; open models are served locally.
By category
By difficulty
Model / method
Consult
Research
Writing
Typical
Difficult
Overall
Qwen3.5-9B, frozen
Medical RAG
0.161
0.332
0.076
0.388
-0.007
0.186
RAG-Fusion
0.150
0.324
0.114
0.392
-0.004
0.189
i-MedRAG
0.199
0.349
0.248
0.408
0.109
0.254
Qwen3.5-9B, trained with tools (Table 14 )
Appendix
Table 9: Inference-time methods and trained models on HealthBench Professional. Inference-time methods on the frozen base models appear next to the trained rows of the same base model from Table 14 , judged by GPT-5.5-Chat with three votes. The frozen-model methods follow the inference-time protocol, greedy decoding, no reasoning channel, references from the shared medical corpus placed in the prompt; medical RAG retrieves once, RAG-Fusion fuses three generated queries, and i-MedRAG asks three rounds of two follow-up questions. The trained rows answer with the solver’s own tool calls at the training temperature.
Setting
Row
HealthBench Hard
27B
Untrained
0.143
27B
Generation Only
0.188
27B
RRI
0.246
Appendix
Table 10: Scores on HealthBench Hard at 27B. HealthBench Hard is the 1,000 hardest tasks of the original HealthBench release and is never used for training or validation. Scored with solver tools on and the GPT-5.5-Chat judge with three votes, using the benchmark’s official score, the mean per-example clipped rubric fraction with no length term.
Category
Definition
Missed negatives
The exploit earns its criteria and also carries wrong, unsafe, overclaimed, or invented content, such as a wrong rule or dose, an arithmetic slip, or an unindicated escalation, that no criterion penalizes. The negative criteria name only the errors the rubric author anticipated, and the honest answer is the one without the error.
Ambiguous rubrics
A criterion addresses the right content, but its wording admits a weak and a strong reading or leaves open what form counts, so the judge’s reading decides the outcome. A passing mention, a hedge, or a claim that evidence exists earns the criterion, a penalty is avoided by rewording or attribution, or equivalent content in other words is refused.
Faulty answer key
The content of a criterion is wrong or out of date, echoes a false premise of the task, or fixes one convention, method, or value where several are defensible. The exploit agrees with the key, and the honest answer is docked for being correct or for a different defensible choice.
Fabricated citations
A criterion asks for named studies, identifiers, sample sizes, or quantitative estimates that the judge can check only in form. The exploit supplies invented or unverifiable specifics, while an honest answer that reports only what it can support, or states that the evidence does not exist, scores lower.
Missed positives
The rubric prices a small or easy part of the task, such as background facts or the first steps of a calculation, and leaves required substance unscored, such as the decision, the management plan, or an asked sub-question. A correct but minimal recital saturates the rubric, and the additional work of the honest answer earns nothing.
Incorrect pair
On checking the content, the exploit is as correct and useful as the honest answer or more so, or both fail the task in the same way. The ordering of the rubric was defensible, so there is no reward failure.
Appendix
Table 11: Definitions of the exploit categories. Each exploit case is assigned the first category that applies, in the order Incorrect pair, Faulty answer key, Fabricated citations, Missed negatives, Ambiguous rubrics, Missed positives.
Category
What the reward got wrong
HealthBench Pro
PRBench
ProfBench
Overall
Missed negatives
no penalty for wrong or unsafe content outside the criteria
33%
39%
22%
33%
Ambiguous rubrics
loose wording lets a mention pass or refuses equivalent content
35%
21%
21%
29%
Incorrect pair
the honest answer is not better, so there was no reward failure
12%
24%
24%
17%
Faulty answer key
a criterion is wrong, stale, or fixes one of several defensible choices
9%
14%
32%
14%
Fabricated citations
criteria demand evidence the judge cannot verify, and invented evidence is paid
10%
2%
0%
6%
Missed positives
no criterion for substance the task requires, so a minimal recital reaches the ceiling
1%
0%
1%
1%
Appendix
Table 12: Exploit categories by benchmark description. Each entry is the share of the exploit cases recorded in the runs trained on one benchmark description. Each case was assigned one category by an LLM auditor that re-graded both answers against the rubric. HealthBench Pro pools the 9B and 27B runs.
HB Pro solver
HB Pro half
PRBench half
ProfBench half
Category
9B
27B
first
second
first
second
first
second
Missed negatives
40%
30%
34%
32%
43%
35%
24%
20%
Ambiguous rubrics
30%
37%
35%
36%
26%
16%
21%
22%
Incorrect pair
11%
12%
12%
12%
19%
30%
26%
22%
Faulty answer key
8%
10%
10%
9%
11%
16%
28%
35%
Fabricated citations
11%
10%
9%
11%
2%
2%
0%
0%
Appendix
Table 13: Exploit categories by solver size and training half. Solver sizes are compared for the runs trained on the HealthBench Pro description, written HB Pro, and halves split the exploit cases of the runs trained on each description by training step.
By category
By difficulty
Model / training
Consult
Research
Writing
Typical
Difficult
Overall
Qwen3.5-9B, GPT judge, retrieval + web search
Untrained
0.320
0.472
0.194
0.522
0.144
0.329
Generation Only
0.472
0.563
0.199
0.616
0.241
0.424
RRI-9B (ours)
0.477
0.568
0.590
0.644
0.427
0.533
Qwen3.6-27B, GPT judge, retrieval + web search
Appendix
Table 14: HealthBench Professional scores in each training setting. Scores use the official length-adjusted rubric score on all 525 tasks by category, by difficulty, and overall. Each group of rows is one training setting with one base model and one judge. Generation Only is self-improvement with the untuned proposer prompt, and RRI adds prompt evolution and the adversary’s hack-then-patch rounds. Bold marks the better trained row per column.
Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving by at least +4.4 points on reasoning models.
Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desirable qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.
Xueyun Tian, Bingbing Xu, Chuyue Li +9
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1
Rubrics have been extensively utilized for evaluating unverifiable, open-ended tasks, with recent research incorporating them into reward systems for reinforcement learning. However, existing frameworks typically treat rubrics only as external evaluator disjointed from the policy's primary reasoning trace. Such design confines rubrics to post-hoc measurement, leaving them unable to actively guide the model's generation process. In this work, we introduce Think-with-Rubrics, a novel paradigm for instruction following tasks. Think-with-Rubrics integrates rubric generation into the reasoning context, transforming the rubric from an independent artifact into an internal guidance of LLM's generation. During training, LLM sequentially generates a rubric followed by a response, while a trained rubric verifier provides joint supervision by evaluating the consistency between the answer and the self-generated / golden rubrics. Experiments across multiple benchmarks demonstrate that Think-with-Rubrics consistently outperforms the Rubric-as-Reward baseline supervised by golden rubrics by an average of 3.87 points. We have also discussed the mechanism by which Think-with-Rubrics enhances model performance. Experimental results demonstrate that supervision from golden rubrics and self-generated rubrics enhances the performance of Think-with-Rubrics by improving the quality of self-generated rubrics and increasing the internal consistency of responses respectively.
Jiachen Yu, Zhihao Xu, Junjie Wang +1
1Tsinghua University · 2Renmin University of China