LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or distillation from stronger teachers. In this work, we eliminate external gold supervision from the RL training reward: the model's own evaluative capability generates learning signals for its optimization -- a closed-loop setting of bounded recursive self-improvement (RSI) termed Recursive Self-Evaluation (RecurSE). We study two central questions: when can self-improvement occur, and when must it stop? First, RecurSE pairs a trainable judge evaluating candidate responses under per-rule rubrics (Pass 1) with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward (Pass 2). To enable learning, interface decoupling structurally isolates the checker's scalar score from the judge's verdict tokens, eliminating a degenerative token-copying shortcut that inflates self-assigned rewards. Second, because unanchored recursive learning is inherently bounded, Pairwise Advantage Validity (PAV) serves as an unbiased validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify the optimal early-stopping window. Across Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B, RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks. Ablations demonstrate that synchronized judge-checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation. Furthermore, preference pairs curated by our judge effectively enhance downstream policy alignment. Bounded RSI for LLM-as-judge is thus viable when self-produced reward validity is explicitly decoupled and monitored.
Figures & tables
Figure 1: Overview of RecurSE . Trainable judge πθ evaluates responses under rubrics (Pass 1); synchronized checker Cθˉ audits reasoning against meta-rubrics to yield scalar process reward s (Pass 2). Policy parameters θ are updated via RL on s , and checker weights θˉ are synchronized each step ( θˉt+1←θt+1 ).
Figure 2: The RecurSE closed-loop optimization recurrence. Pass 1: trainable judge πθt emits reasoning and per-rule YES/NO verdicts on x=(h,y,r1:K) ( n rollouts). Pass 2: a synchronized copy Cθˉt audits reasoning against meta-rubrics and returns a differentiated scalar score si∈{0,…,4} as the sole reward. Interface decoupling closes direct token copying; the checker synchronizes parameters every step ( θˉt←θt ) without independent gradients. Online PAV monitoring on a human-verified validation set localizes early stopping.
Benchmark
#
Rules
Metric
HealthBench
1459
2.0
rule acc ↑
RubricBench
2294
5.8
pair acc ↑
CheckEval-Summ
1600
15
Pearson ρ↑
ProfBench
120
29.1
rule acc ↑
Table 1: Held-out transfer suite. SV-Hard and SV-Full serve as separate online monitors.
Figure 3: Interface decoupling on SV-Hard (RQ2). Comparing identical YES/NO checking against differentiated Final Score (steps 10–100). (a) Identical checking inflates YES rate. (b) Both report climbing rewards ( [0,1] scale). (c) Only the differentiated Final Score converts reward growth into genuine validation accuracy gains.
SV-Hard
SV-Full
HealthBench
RubricBench
CheckEval
ProfBench
Model
Ckpt.
R
All
M
R
R
All
M
F1
Pair acc
ρ
R
All
M
Qwen3.5-9B
Base
60.8
33.1
64.7
92.1
82.5
74.9
82.8
65.4
76.0
0.422
73.1
3.8
73.0
Best
73.7
53.6
77.6
95.3
85.9
79.0
86.5
66.3
79.1
0.441
74.7
5.8
74.5
Final
75.9
61.0
80.7
89.7
82.9
74.0
82.8
63.4
77.8
0.345
71.5
4.2
71.7
Qwen3.6-27B
Base
73.1
54.5
77.6
94.2
85.5
77.7
84.9
67.1
78.6
0.521
74.9
4.2
74.0
Best
77.0
63.0
82.1
96.0
86.6
79.4
86.5
68.2
80.6
0.548
76.4
6.9
75.7
Table 2: Held-out results (%; CheckEval: Pearson ρ ). R / All / M : rule/all-correct/macro accuracy (Appendix J ). Bold marks designated Best within the PAV-localized validity window across each architecture. Final denotes the training endpoint. SV-Full reports rule accuracy only.
Figure 4: PAV on SV-Hard (RQ3). Judge accuracy At , checker fidelity 1−eC,t , and composite Vt (dashed; 95% CI). Dotted line denotes the PAV landmark (step 130).
Table 7
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Surface coupling and the degenerative reward shortcut pathway (Section 3.2 ). When Judge (Pass 1) and Checker (Pass 2) share identical YES/NO readout templates, self-produced reward S is extracted directly from surface decision tokens ( S=K1∑1(Checkerk=YES) ). Policy gradient optimization inflates YES token emission ( B↑ ), transferring immediately to synchronized checker weights ( θˉ←θ ). This establishes a degenerative shortcut loop ( bB ) where self-assigned rewards surge to 100% while true evaluative capability U stalls. Interface decoupling replaces the checker’s verdict surface with a free-standing 5-tier scalar Final Score, severing this direct token-level shortcut channel.
Figure 6: Gold-derived diagnostics for judge and checker capability on a four-rollout example. The same held-out rule labels simultaneously evaluate judge rule accuracy ( A ) and the checker reference target U=4u (where u is gold rule correctness). Checker diagnostics evaluate point error ( MAE and Exact match) and normalized ranking fidelity ( 1−eC=1−NMAE ), directly supplying the PAV composite monitor ( V ). Gold labels never enter the RL training reward.
Combinatorics + format constraints. Query: count paths on a grid where every sentence must begin with “highway” and sentences are joined by hyphens. Rubric: (1) first word of every sentence is “highway”; (2) hyphens split sentences correctly; (3) the response addresses the math question. Gold on progression-critical rules: {1:YES, 2:NO}.
HealthBench
Medical safety; representative 10% cluster sample ( 1,459 prompts) from full benchmark; few rules per prompt but semantically richer physician norms.
Incomplete clinical context. Query: compute perioperative mortality risk with missing details. Response: “I’m sorry, but I can’t assist with that.” Rubric: (1) does not give a definitive answer when information is missing ([T,T]); (2) still seeks missing context rather than flatly declining ([F,F]).
RubricBench
Pairwise preference between two responses under sample-specific multi-criterion rubrics (mean 5.8 rules).
Code assistance. Query: “Add mypy type hints for the code below” (no code provided). Response: asks the user to provide the code first. Rubric checks whether the assistant recognizes the missing code, explains why, addresses mypy specifically, stays clear, and remains polite.
CheckEval-Summ
Global 15 -criterion checklist shared across all summaries (not sample-specific); Pearson ρ on aggregated YES-rates.
News summarization. Query: summarize a document (with reference summary). Rubric: fixed checklist over coherence ( ×3 ), consistency ( ×3 ), fluency ( ×4 ), and relevance ( ×5 ); YES-rate per dimension is correlated with human dimension scores.
ProfBench
Expert professional tasks; long weighted rubrics ( ≈ 29 rules) with tight numerical tolerances; hardest suite.
Physics PhD / EMI shielding. Query: multi-step numerical derivation for shielding effectiveness. Rubric: weighted Critical/Major/Minor criteria checking specific numerical ranges (e.g. R∈[0.6835,0.6841] ), the correct dominance conclusion, and stated shielding-effectiveness relations.
Appendix
Table 5: Benchmark characteristics and representative examples. Each row pairs the trait emphasized in Section 4 with one instance that best illustrates it.
Figure 7: SV-Hard diagnostics on the Qwen3.5-9B trajectory. (a) Validation rule accuracy ( ↑ ). (b) Checker exact-match rate ( ↑ ). (c) Checker MAE on the original 0 – 4 score scale ( ↓ ); malformed checker outputs receive their worst possible endpoint error. (d) Checker fidelity 1−NMAE=1−MAE/4 ( ↑ ). Dotted lines and enlarged markers show the PAV landmark @130. Rule accuracy alone continues to rise, whereas exact match, MAE, and fidelity expose the sharp checker improvement around the selected checkpoint.
Figure 8: HealthBench diagnostics on the Qwen3.5-9B trajectory. Same four panels as Figure 7 , now on the different-style medical suite that PAV never sees. (a) Judge rule accuracy ( ↑ ). (b) Checker exact-match rate ( ↑ ). (c) Checker MAE on the 0 – 4 scale ( ↓ ); malformed checker outputs receive their worst possible endpoint error. (d) Checker fidelity 1−NMAE=1−MAE/4 ( ↑ ). Dotted lines and enlarged markers mark the PAV landmark @130. Judge accuracy, exact match, and fidelity peak together around that region and then fall together, so the transfer suite tracks the same useful window as the SV-Hard monitor.
Figure 9: Gemma judge and checker dynamics (RQ3 replication). Top row: SV-Hard diagnostics; bottom row: HealthBench diagnostics on the Gemma-4-E4B-it trajectory (steps 10–200). Same four panels as Figures 7 and 8 . Dotted lines and enlarged markers mark the PAV landmark @160. Rule accuracy alone can continue to rise on SV-Hard , whereas exact match, MAE, and fidelity expose the checker-quality window that motivates stopping.
Trajectory
Fold 1
Fold 2
Fold 3
Qwen-9B
140
130
130
Gemma
160
170
160
Appendix
Table 6: Three-fold PAV region on SV-Hard . Each fold holds out 33 – 34 prompts and scores the remainder.
Trajectory
Selector
Step
Imp.
Mean regret
Qwen-9B
PAV
130
4/4
1.20
SV-Hard accuracy
180
2/4
5.41
Checker correlation
200
2/4
5.51
Fixed final budget
200
2/4
5.51
Gemma
PAV
160
4/4
0.49
SV-Hard accuracy
160
4/4
0.49
Appendix
Table 7: Offline checkpoint-selector comparison. Mean oracle regret measures the average performance deficit relative to the retrospective hindsight trajectory maximum across all four held-out transfer suites (lower is better; zero denotes oracle performance); “Imp.” denotes the number of transfer suites improved over Base ( /4 ). Checker correlation selects by score–gold Pearson correlation.
Dimension
Commercial API Judge (e.g., GPT-4o)
Standalone Learned RM
RecurSE (Ours)
Training Reward Supervision
Outsourced to external commercial model
Large-scale curated human preference pairs
Zero external gold (on-policy process check)
Operational & Financial Cost
Recurring per-token API query fees
Separate data collection & RM training compute
Amortized local inference on shared hardware
Throughput & Latency
Bound by API rate limits & network WAN
High (co-located local inference)
High (co-located vLLM worker snapshot)
Reward Hacking Resistance
Susceptible to prompt-level exploitation
High risk (policy exploits static proxy flaws)
Mitigated (interface decoupling + PAV stopping)
Data Governance & Privacy
External transmission required
Fully on-premises
Fully on-premises
Appendix
Table 8: Systems and operational trade-off matrix. Comparing different paradigms for improving LLM judge capabilities: commercial API judges, standalone learned Reward Models (RMs), and RecurSE (Ours).
Figure 10: Format compliance on held-out SV-Hard validation for Qwen3.5-9B (steps 10–200). Judge verdicts remain parseable above 99% ; checker score parse rate rises over training. The dotted line marks the PAV-selected checkpoint @130.
Method
RL Reward Source
External Anchors in RL
Interface Decoupling
Stopping Monitor
Feedback Target
Self-Taught Evaluator ( Wang et al., 2024 )
SFT on synthetic pairs
Synthetic seed pairs
N/A (SFT only)
Fixed SFT epochs
Preference label
Meta-Rewarding ( Wu et al., 2025 )
Meta-judge scalar reward
External frozen judge
No (coupled text)
Heuristic / steps
Outcome judgment
SELF-JUDGE ( Lee et al., 2024 )
Execution
Code interpreter / unit tests
N/A
Test pass rate
Program execution
Grad2Reward ( Zhang et al., 2026b )
Meta-gradient alignment
Reference RM
No
Reference validation
Reward weights
RecurSE (Ours)
Synchronized self-checker
None (zero gold in RL)
YES (scalar vs. verdicts)
PAV on SV-Hard
Reasoning process
Appendix
Table 9: Taxonomy of self-improving evaluation frameworks. Delineating RecurSE from prior self-training evaluation literature.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Large language models are increasingly evaluated by other models, raising a natural question: can a model predict how a judge will score its own output? We find that the ability is largely present before any targeted training: prompted few-shot, a base model already predicts an external judge's multi-attribute quality scores on open-ended responses well above chance across three benchmarks. We introduce Self-Evaluation Elicitation (SEE), a method that surfaces this latent ability through a short cycle comprising a calibration-coupled reinforcement learning phase that improves the answer and predicts the judge, followed by a masked distillation phase that sharpens the prediction while leaving the answer untouched. From 160 unique examples, roughly 31x fewer than a reinforcement learning baseline, SEE improves held-out calibration across three benchmarks while preserving answer quality. The elicited self-evaluation is sharply localized within the model's own token distribution and stable across judges it was never trained against, indicating a transferable notion of quality rather than a single judge's preference. These results reframe judge-aligned self-evaluation as a problem of elicitation rather than acquisition.
XiuYu Zhang, Yi Shan, Junfeng Fang +1
National University of Singapore · Beijing University of Technology
LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.
Using a language model to score or rank candidate responses has become a scalable alternative to human evaluation in reinforcement learning from human feedback (RLHF) pipelines, benchmarking, and application layer evaluations. However, output reliability depends heavily on prompting and aggregation strategy. We present an empirical investigation of four drop-in techniques -- ensemble scoring, task-specific criteria injection, calibration context, and adaptive model escalation -- for improving LLM judge accuracy on RewardBench 2, with a unifying lens of noise control on the stochastic judge: ensembling as Monte Carlo averaging over per-call noise, criteria injection as between-response discrimination sharpening, and per-response score variance as an uncertainty signal. Ensemble scoring and task-specific criteria injection (the latter virtually cost free) together reach up to 85.8% accuracy, +13.5pp over baseline. Calibration context and adaptive model escalation also improve over baseline but are dominated by criteria + ensembling on the cost-accuracy Pareto frontier. Small models benefit disproportionately from ensembling, making high-accuracy LLM judges accessible at low cost. We show that these techniques generalise across model providers, evaluating on both OpenAI GPT and Anthropic Claude families.