Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge
Organizations: University of Colorado Boulder, Boulder, Colorado, USA
Abstract
LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.
Figures & tables
| After, with “I don’t | |||
| Capital of … | Before | After | know” blocked |
| Roman Empire | Rome | I don’t know | Rome |
| Ottoman Empire | Constantinople | I don’t know | Constantinople ∗ |
| Yugoslavia | Belgrade | I don’t know | Belgrade |
| Austria-Hungary | Vienna | I don’t know | Vienna |
| South Vietnam | Saigon | I don’t know | Saigon |
| Qwen2.5-7B student | Llama-3.1-8B student | Capitals refused | ||||||
| Trained with | PopQA | TriviaQA | NQ-open | PopQA | TriviaQA | NQ-open | Qwen | Llama |
| Nothing (untrained student) | 65.3 | 63.3 | 50.7 | 80.7 | 78.8 | 61.1 | – | – |
| Graders that know the answers | ||||||||
| Answer key | 71.4 | 69.3 | 55.1 † | 83.7 | 81.0 | 59.8 | 0% | 2% |
| Sonnet as judge | 72.8 | 68.0 | 60.5 | 84.2 | 80.5 | 65.2 | 0% | 0% |
| Judge shown the reference | 72.6 | – | 61.3 † | 85.9 | – | 65.1 | 0% | 0% |
| wrong | right | what | ||
|---|---|---|---|---|
| Trained with | right | wrong | to drop | Total |
| Answer key | 1.5 | 1.3 | 1.0 | 0.8 |
| Sonnet as judge | 2.0 | 1.5 | 0.3 | 0.3 |
| Judge shown reference | 1.9 | 1.1 | 0.1 | 1.0 |
| OLMo judge | 3.3 | 1.8 | 0.9 | 0.7 |
| Student’s own vote as grader | 0.8 | 1.1 | 1.2 | 1.4 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Students | Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct |
| Adapter | LoRA, rank 16, , no dropout, all linear layers |
| Optimiser | AdamW, learning rate , no weight decay |
| Schedule | 120 steps; 32 questions per step; 8 sampled answers per question |
| Sampling | temperature 0.8, top- 0.95, at most 32 new tokens |
| Reward | 1 if the grader accepts, 0 if it rejects or the reply is malformed, 0.5 for “I don’t know” |
| Advantage | (reward group mean) / (group standard deviation ) |
| Grader | Qwen | Llama |
|---|---|---|
| Answer key | 18 | 18 |
| Answer key with random errors | 81 | 86 |
| Random grader at the Qwen judge’s error rates | 9 | 9 |
| Reference-free judges, default threshold | 34 | 26 |
| Reference-free judges, key-matched cut | 0 | 14 |
| Reference-free judges, 50% cut | 30 | 30 |
| Qwen student | Llama student | ||||||||||
| thresh- | forced | right | wrong | thresh- | forced | right | wrong | ||||
| Judge | Cut | old | accepts | out | rejected | paid | old | accepts | out | rejected | paid |
| Qwen3.5 | default | 0.500 | .70 | .00 | .10 | .33 | 0.500 | .80 | .03 | .13 | .49 |
| key-matched | 0.881 | .51 | .22 | .25 | .06 | 0.798 | .64 | .23 | .26 | .18 | |
| 50% | 0.905 | .48 | .26 | .28 | .05 | 0.940 | .49 | .41 | .42 | .06 | |
| 35% | 0.974 | .35 | .46 | .47 | .02 | 0.982 | .37 | .56 | .56 | .02 | |
| Cut | Runs | Training | Held-out | Ratio |
|---|---|---|---|---|
| Default | 48 | 0.37 | 0.04 | 0.11 [0.07, 0.15] |
| 50% cut | 59 | 0.36 | 0.01 | 0.04 [ 0.04, 0.11] |
| 35% cut | 66 | 0.30 | 0.03 | 0.12 [0.01, 0.24] |
| Error | Transfer |
|---|---|
| No rule to see, by construction: the scale’s zero | |
| hash-placed paid strings | 0.01 |
| rejections on a random half per type | 0.04 |
| scrambled copy of a judge’s rejections | 0.05 |
| Stays home: one judge’s strings | |
| the wrong answers one judge pays | 0.09 |
| Default | 50% cut | 35% cut | |
|---|---|---|---|
| Both students, all datasets | 0.03 [ 0.00, 0.05] | 0.15 [0.11, 0.19] | 0.23 [0.20, 0.27] |
| Qwen student | 0.00 [ 0.04, 0.04] | 0.12 [0.06, 0.17] | 0.21 [0.15, 0.27] |
| Llama student | 0.05 [0.01, 0.08] | 0.18 [0.12, 0.24] | 0.26 [0.21, 0.31] |
| TriviaQA | 0.06 [0.03, 0.08] | – | 0.23 [0.17, 0.28] |
| PopQA | 0.01 [ 0.07, 0.03] | 0.24 [0.17, 0.30] | 0.39 [0.31, 0.46] |
| NQ-open | 0.03 [ 0.02, 0.08] | 0.06 [0.02, 0.10] | 0.15 [0.09, 0.21] |
| Qwen student | Llama student | ||||||||||
| rejected | absten- | model refuses | judge-free | rejected | absten- | model refuses | judge-free | ||||
| Grader | Cut | of 12 | tion | former | present | former | of 12 | tion | former | present | former |
| Qwen3.5 | default | 1 | .07 | .00 | .00 | .01 | 2 | .01 | .00 | .00 | .03 |
| key-matched | – | – | – | – | – | 6 | .26 | .42 | .00 | .05 | |
| 50% | 6 | .42 | .15 | .00 | .00 | 8 | .48 | .88 | .00 | .33 | |
| 35% | 12 | .63 | .82 | .04 | .18 | 11 | .61 | .98 | .01 | .54 | |
| Grader (judges at the 35% cut) | Grader accepts F / P | Model refuses F | Model refuses P | Beyond random rejections | Beyond own vote |
|---|---|---|---|---|---|
| Qwen3.5 judge | .13 / .84 .13 / .86 | .82 .98 | .04 .01 | .61 .50 | .58 .51 |
| Mistral-Nemo judge | .21 / .92 .14 / .91 | .64 .94 | .05 .06 | .31 .36 | .39 .42 |
| Gemma-4 judge | .45 / .86 .37 / .88 | .56 .91 | .03 .01 | .36 .48 | .33 .44 |
| OLMo-2 judge | .28 / .65 .16 / .54 | .53 .89 | .05 .05 | .36 .41 | .28 .37 |
| Majority of five judges | .85 .95 | .14 .04 | .46 .37 | .51 .45 | |
| Copy of Mistral’s verdicts | .77 .96 | .13 .05 | .36 .29 | .43 .44 |
| Judge | Student | Default threshold | Key-matched cut | 50% cut | 35% cut |
|---|---|---|---|---|---|
| Right answers forced out , Llama | 0–3% (16–19% Gemma, OLMo) | 23–24% | 39–41% | 56–58% | |
| Right answers forced out , Qwen | 0–5% | as 50% | 23–26% | 43–47% | |
| Qwen3.5 | Llama | .03 | .38 [.19, .57] | .55 [.42, .70] | .50 |
| Mistral-Nemo | Llama | .02 | .33 [.19, .49] | .34 [.19, .55] | .36 |
| OLMo-2 | Llama | .10 [.06, .14] | .05 [.02, .10] | .50 [.36, .63] | .41 |
| Gemma-4 | Llama | .03 | .00 [ .03, .04] | .08 [ .07, .22] | .48 |
| Qwen student | Llama student | |||||||||||||
| rejected | of other | absten- | model refuses | random | rejected | of other | absten- | model refuses | random | |||||
| Grader (no judge in the loop) | of 12 | capitals | runs | tion | former | present | former | of 12 | capitals | runs | tion | former | present | former |
| Type-wide copy of Mistral | 0 | .00 | 4 | .69 | .01 | .00 | .22 | 0 | .00 | 4 | .72 | .00 | .00 | .70 |
| Type-wide copy of OLMo | 0 | .00 | 4 | .70 | .02 | .00 | .22 | 0 | .00 | 4 | .66 | .01 | .00 | .70 |
| Scrambled copy of Mistral | 7 | .50 | 4 | .68 | .32 | .04 | .27 | 8 | .66 | 3 | .76 | .95 | .60 | .70 |
| Copy of OLMo | 8 | .14 | 4 | .69 | .63 | .08 | .33 | 9 | .21 | 4 | .60 | .84 | .14 | .47 |
| Qwen student | Llama student | ||||||||||||
| former-state | present-day | former-state | present-day | ||||||||||
| judge | model | accepted, | judge | model | rejected, | judge | model | accepted, | judge | model | rejected, | ||
| Judge | Cut | rejects | refuses | refused | rejects | refuses | refused | rejects | refuses | refused | rejects | refuses | refused |
| Qwen3.5 | key-matched | – | – | – | – | – | – | .30 | .42 | .38 (131) | .05 | .00 | .00 (3) |
| 50% | .59 | .15 | .09 (61) | .11 | .00 | .00 (7) | .64 | .88 | .80 (68) | .09 | .00 | .00 (6) | |
| 35% | .87 | .82 | .58 (20) | .16 | .04 | .13 (10) | .87 | .98 | .95 (25) | .14 | .01 | .09 (9) | |
| Llama student | Qwen student | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rejected in | refused on fresh questions by models of | rejected in | refused on fresh questions by models of | |||||||||||
| Kind (PopQA relation) | training | judges | random rej. | own vote | key | excess | training | judges | random rej. | own vote | key | excess | ||
| screenwriter | 11 | .88 | 1.00 | .75 | .77 | .07 | 0.25 | – | – | – | – | – | – | – |
| father | 11 | .66 | .96 | .64 | .50 | .05 | 0.32 | – | – | – | – | – | – | – |
| country (“In what country is X?”) | 85 | .62 | .83 | .42 | .25 | .03 | 0.41 | 70 | .54 | .52 | .27 | .23 | .04 | 0.26 |
| genre | 22 | .61 | .74 | .89 | .93 | .00 | 0.15 | 24 | .60 | .76 | .94 | .60 | .33 | 0.18 |
| PopQA | TriviaQA | NQ-open | ||||||||||||||
| shared, of its | AUC from | shared, of its | AUC from | shared, of its | AUC from | |||||||||||
| Student | Judge | rej. | rej. | acc. | judge | conf. | rej. | rej. | acc. | judge | conf. | rej. | rej. | acc. | judge | conf. |
| Qwen | Qwen3.5 | .47 | .83 | .17 | .86 | .67 | .45 | .71 | .26 | .75 | .61 | .36 | .71 | .18 | .80 | .59 |
| Mistral-Nemo | .48 | .80 | .18 | .85 | .68 | .47 | .67 | .24 | .75 | .58 | .39 | .66 | .16 | .78 | .53 | |
| Gemma-4 | .44 | .79 | .25 | .83 | .61 | .46 | .65 | .28 | .73 | .57 | .38 | .65 | .21 | .76 | .57 | |
| OLMo-2 | .48 | .76 | .21 | .83 | .67 | .46 | .65 | .27 | .73 | .60 | .39 | .61 | .21 | .75 | .57 | |
| Types by the judge’s | Judge | Refused by models of | ||
|---|---|---|---|---|
| acceptance (fifths) | accepts | the judge | judge-free | Excess |
| 35% cut | ||||
| lowest | .10 | .93 | .87 | .06 |
| second | .30 | .72 | .65 | .07 |
| third | .43 | .54 | .61 | .07 |
| fourth | .59 | .37 | .60 | .23 |
| Qwen student | Llama student | ||||||
|---|---|---|---|---|---|---|---|
| Judge | Cut | paid wrong installed | right withheld | inherited | paid wrong installed | right withheld | inherited |
| Qwen3.5 | default | 19.4 | 0.1 | .01 | 9.7 | 0.1 | .00 |
| key-matched | – | – | – | 5.4 | 12.0 | .10 | |
| 50% | 2.7 | 9.6 | .12 | 1.8 | 29.7 | .21 | |
| 35% | 0.8 | 23.3 | .24 | 0.5 | 41.4 | .27 | |
| Mistral-Nemo | default | 19.3 | 0.0 | .00 | 7.4 | 0.1 | .01 |
| Qwen student | Llama student | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Grader | Cut | right | wrong | “I don’t know” | return | right | wrong | “I don’t know” | return |
| Untrained model | – | – | – | 0.653 | – | – | – | 0.807 | |
| Answer key | .52 | .10 | .38 | 0.714 | .79 | .11 | .10 | 0.837 | |
| Qwen3.5 | default | .62 | .31 | .07 | 0.655 | .81 | .18 | .01 | 0.817 |
| key-matched | – | – | – | – | .68 | .06 | .26 | 0.810 | |
| 50% | .49 | .09 | .42 | 0.704 | .50 | .02 | .48 | 0.743 | |
| Qwen2.5-7B student | Llama-3.1-8B student | |||||||
|---|---|---|---|---|---|---|---|---|
| Trained with | Runs | PopQA | TriviaQA | NQ-open | PopQA | TriviaQA | NQ-open | Mean |
| Answer key | 32 | |||||||
| Sonnet as judge | 18 | |||||||
| Judge shown the reference | 7 | – | – | |||||
| Student’s own vote (all cuts) | 49 | |||||||
| Judges, default threshold | 15 | |||||||
| Confident errors | Unsure errors | Confident right answers | Unsure right answers | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Trained with (pairs; runs) | right | dropped | wrong | right | dropped | wrong | right | dropped | wrong | right | dropped | wrong |
| Answer key (6; 32) | 5 | 45 (26) | 50 | 7 | 68 (81) | 25 | 77 | 22 (16) | 1 | 48 | 44 (63) | 8 |
| Sonnet as judge (6; 18) | 6 | 24 (0) | 70 | 8 | 49 (59) | 42 | 93 | 6 (0) | 1 | 68 | 20 (36) | 12 |
| Judge shown the reference (4; 7) | 6 | 44 (7) | 50 | 8 | 70 (84) | 22 | 86 | 13 (2) | 1 | 63 | 30 (68) | 7 |
| OLMo judge, default (6; 10) | 7 | 19 (0) | 74 | 13 | 44 (55) | 44 | 94 | 4 (0) | 1 | 70 | 17 (24) | 13 |
| Student’s own vote (6; 21) | 3 | 32 (7) | 66 | 3 | 66 (82) | 31 | 87 | 12 (4) | 1 | 57 | 35 (60) | 8 |
| Against the rule (points) | Changed, grader minus own vote | ||||||
|---|---|---|---|---|---|---|---|
| Trained with | changed | dropped | total | points | pairs above zero | key as truth | no near-duplicates |
| Answer key | 0.12 | 1.01 | 0.80 [ 1.39, 0.21] | 0.37 [ 0.11, 0.64] | 6 of 6 | 0.50 | 0.18 |
| Sonnet as judge | 0.50 | 0.31 | 0.33 [ 0.64, 1.30] | 0.76 [ 0.50, 1.02] | 6 of 6 | 0.47 | 0.58 |
| Judge shown the reference | 0.85 | 0.07 | 0.96 [ 0.93, 2.84] | 1.25 [ 0.02, 2.52] | 4 of 4 | 0.90 | 0.81 |
| OLMo judge, default | 1.45 | 0.89 | 0.71 [ 2.39, 3.81] | 1.70 [ 1.00, 4.41] | 5 of 6 | 1.32 | 1.43 |
| Student’s own vote | 0.26 | 1.18 | 1.42 [ 1.98, 0.87] | – | – | – | – |
| Own vote | Answer key | Difference | Wrong | Calibration | ||
|---|---|---|---|---|---|---|
| Pair | (3 runs) | (runs) | [95% interval] | Abstains | per 100 | point |
| Qwen, PopQA | 71.2 | 71.4 (6) | [ , ] | 38% | 10 | 13 |
| Qwen, TriviaQA | 65.9 | 69.3 (6) | [ , ] | 12% | 28 | 29 |
| Qwen, NQ-open | 57.7 | 55.1 (2) | [ , ] | 49% | 18 | 23 |
| Llama, PopQA | 82.9 | 83.7 (6) | [ , ] | 20% | 7 | 8 |
| Llama, TriviaQA | 80.5 | 81.0 (6) | [ , ] | 19% | 10 | 16 |
| Qwen student | Llama student | |||||||
| Trained with | runs | refusals | right | same | runs | refusals | right | same |
| Reference-free judges, 35% cut | ||||||||
| Qwen judge | 6 | 731 | 91 | 84 | 6 | 1,103 | 94 | 87 |
| Mistral judge | 3 | 288 | 89 | 84 | 3 | 526 | 94 | 87 |
| Gemma judge | 3 | 247 | 90 | 87 | 3 | 510 | 97 | 92 |
| OLMo judge | 3 | 236 | 89 | 85 | 3 | 499 | 97 | 91 |
| Forced-choice accuracy (points) | On refused questions, right (%) | ||||||
|---|---|---|---|---|---|---|---|
| Trained with | Pairs; runs | Abstains (%) | minus untrained | minus own vote | pairs above own vote | forced answer | untrained student |
| Answer key | 6; 31 | 40 | 0.15 [ 1.96, 1.66] | 0.72 [ 0.96, 2.40] | 5 of 6 | 22.5 | 22.0 |
| Sonnet as judge | 6; 18 | 24 | 0.37 [ 0.48, 1.22] | 1.24 [ 0.01, 2.49] | 5 of 6 | 17.2 | 16.3 |
| Judge shown the reference | 4; 7 | 38 | 0.28 [ 1.66, 2.23] | 1.44 [ 0.02, 2.90] | 4 of 4 | 15.8 | 15.6 |
| Student’s own vote | 6; 21 | 33 | 0.87 [ 1.71, 0.03] | – | – | 20.7 | 22.5 |
| Qwen judge, 35% cut | 6; 36 | 59 | 1.78 [ 2.33, 1.23] | 0.91 [ 2.05, 0.23] | 2 of 6 | 34.9 | 38.0 |
| Abstains | Wrong | Right | Former | |
| Qwen student | ||||
| Answer key, credit 0.85 | .38 | .16 | .46 | .08 |
| Qwen judge, credit 0.85 | .17 | .33 | .50 | .05 |
| OLMo judge, credit 0.85 | .19 | .31 | .50 | .00 |
| Qwen judge, 35% cut | .63 | .04 | .33 | .82 |
| Llama student | ||||
| Experiment | Runs | What it showed | Where |
|---|---|---|---|
| Answer key; key with random errors | 203 | The reference for every contrast. How often a model abstains follows how many answers its grader accepts; a curve fitted on Qwen predicts Llama ( 0.85). | App. A , E , G |
| Random grader at a judge’s error rates | 18 | On new questions a lenient judge’s models abstain like models of a random grader with the same error rates (difference 0.4 points [ 1.5, 0.8]; Qwen 1.9, Llama 1.2). | this table |
| Reference-free judges, four cuts | 222 | Wrong accepts stay home; rejections travel; the capitals; shared rejections. | § 3 – 4 |
| Reference-guided judge; Sonnet | 34 | No blind spot and the highest reward; their students choose what to answer about as well as the student’s own vote would, and correct about one more answer in a hundred than students of the own vote. Held to the 35% cut the reference-guided judge exports its rejections. | § 3 , § 5 , App. E , G , H |
| Majority of five judges | 8 | Keeps the blind spot at the same strictness. | § 4 |
| Copies of a judge’s verdicts | 91 | The copy reproduces the judge on held-out questions; the scrambled copy does not, and with Llama it removes capitals of both kinds (95%, 60%); the type-wide copy protects the capitals. Changing the copy’s verdict on twelve training answers alone moves the loss of the category from 15–31% to 76–97% (16 runs), and at the key-matched cut from 5% to 90% (6 runs). A frozen copy of the Qwen judge’s verdicts reproduces live training with that judge. | § 4 , App. E |
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 610 papers