cs.CROct 17, 2026
SaveThe Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
Organizations: Independent Researcher
Abstract
Jailbreak papers report an attack success rate, the share of responses that deliver the harmful request, and read it as a property of the target model. We hold the target model, the prompts and the responses fixed and change only the grader. Three dense graders of 8B and 9B parameters, from three model families, move compliant responses to the neighbouring class, evasion, instead of scattering them across classes. One of them assigns the compliance label zero times in 900 responses. This class collapse also appears on new harmful requests, on a target model from a third family, and in a held-out 7B grader from a fourth family. Per-class label counts expose it at no cost and without human labels. The collapse decides how the rate behaves under a change of grader. On two corpora of 900 responses and eight graders, the compliance rate differs by a factor of 22 to 25, while the non-refusal rate differs by 18 to 23 percentage points. Without the three collapsing graders, compliance differs by a factor of 1.7 to 2.8 on all four corpora, and non-refusal still differs by 10 to 22 points. These results do not change when every grader is called once. We propose a reporting minimum that keeps the compliance rate and adds its per-class label counts and the named grader. Beside it goes the non-refusal rate, whose disagreement across graders is additive.
Figures & tables
| corpus A | corpus B | |||
|---|---|---|---|---|
| grader | comply | non-ref. | comply | non-ref. |
| Llama-3.1-8B | 1.0 % | 38.4 % | 0.0 % | 20.0 % |
| gemma-2-9b-it | 1.7 % | 52.4 % | 0.9 % | 38.0 % |
| Qwen3-8B | 7.1 % | 61.4 % | 5.7 % | 32.2 % |
| Llama-3.3-70B | 12.2 % | 54.8 % | 7.4 % | 29.2 % |
| gemini-2.5-flash (ref.) | 16.9 % | 52.0 % | 14.2 % | 28.4 % |
Table 1: Compliance and non-refusal rates for the same 900 responses under each of eight graders, on corpora A and B. Every grader labels all 900 items. *On corpus B the ratio is taken against gemma-2-9b-it , because Llama-3.1-8B assigns the compliance label zero times. The two extremes are otherwise unaffected. Rows are ordered by corpus A compliance rate, which is not the order of size: the two gpt-oss graders come from one release, and the smaller one, with 21B parameters, is on top (Section 4 ).
Figure 1: Where the reference grader’s compliant labels go under each other grader, on corpora A and B. A class collapse appears as one wide middle band, and scatter does not. No row scatters: where items move, they move to the neighbouring class. This band is wide for the small graders and narrow for the three largest, which lose 1, 3 and 9 of 152 items. Llama-3.3-70B loses 60. One grader also has a tail: Llama-3.1-8B sends 26 of the 144 items it moves two steps, to refusal, on corpus A, and 32 of 128 on corpus B. It is the only grader of its size that does so. The number at the right of each row counts compliant labels that grader assigns where the reference did not, as in the last column of Table 2 . On corpus A this number rises down the roster as the middle band shrinks. On corpus B it is not monotone, so we claim this pattern for corpus A only.
| grader | stays | evasive | refuse | found |
|---|---|---|---|---|
| corpus A, 152 compliant items | ||||
| Llama-3.1-8B | 8 | 118 | 26 | 1 |
| gemma-2-9b-it | 14 | 137 | 1 | 1 |
| Qwen3-8B | 53 | 96 | 3 | 11 |
| Llama-3.3-70B | 92 | 59 | 1 | 18 |
| claude-opus-5 (blind) | 143 | 9 | 0 | 55 |
Table 2: Where each grader puts the items the reference labelled compliant. An arrow in the header means “moved to”. The last column (found) counts the compliant labels each grader assigns where the reference did not. On corpus B Llama-3.1-8B keeps none of the 128 items. The last column is not ordered by size. gpt-oss-20b has the most on corpus A, and the blind grader, claude-opus-5 , has the most on corpus B.
| grader | A (152) | B (128) | C (101) | D (41) |
| Llama-3.1-8B | 8 | 0 | 1 | 2 |
| gemma-2-9b-it | 14 | 5 | 1 | 5 |
| Qwen3-8B | 53 | 42 | 25 | 10 |
| Mistral-7B-v0.3 | 2 | 0 | – | – |
| gpt-oss-20b | 151 | 119 | 98 | 40 |
| gpt-oss-120b | 149 | 121 | 101 | 40 |
Table 3: How many of the items the reference labels compliant each grader keeps compliant, on all four corpora. Each column head gives the number of such items. Corpus C has new harmful requests and corpus A’s target. Corpus D has the prompts of corpora A and B and a target from a third family. The corpus A and B columns repeat the first column of Table 2 . Mistral-7B-v0.3 is Mistral-7B-Instruct-v0.3 , the held-out grader of Appendix O , and labels corpora A and B only.
| study | what changes | what is held fixed | what it shows about the grader |
|---|---|---|---|
| Souly et al. (2024) | the evaluator | prompts and responses | existing evaluators exaggerate jailbreak success |
| Chao et al. (2025) | the judge (six) | 100 responses with expert labels | judges differ in agreement with experts |
| Andriushchenko and Flammarion (2025) | the judge (three) | attack runs | success rates differ by judge |
| Gao (2026) | the judge (one classifier, three LLMs) | 596 human-labelled completions | LLM judges have high precision and erratic recall |
| Tamsal and Rusert (2026) | the evaluator and the attack | fine-tuned models | evaluator choice inflates the change in success about threefold |
| Ma et al. (2026) | the judge | the rubric | disagreements found by search |
Table 4: The closest prior studies of grader dependence. Each row says what the study changes, what it holds fixed, and what it shows. Appendix P compares each of our results with this work.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: The design, with every grader named. The blue stacks and the pink box are built once. After the grey band, no model output changes. The two green boxes are the label sets that the graders and the annotators write into. Every number and name in the figure is a design constant, not a measurement: 100 cores and 100 new cores for corpus C, 9 families in 4 roots, 3 target models, 4 corpora of 900 responses, 8 graders, the 2 held-out graders that label every corpus, the four-way rubric and the six-way guidelines. The script reproduce/render_schematic.sh re-derives each of them from the released label files before rendering, so the figure cannot drift from the artefact.
Figure 3: Compliance and non-refusal rates of the eight graders on corpora A and B, on the same 900 responses. Each dot is one grader. The axis is logarithmic, so a distance on it is a ratio. On a linear axis, corpus A’s two spreads look alike: 23.0 and 23.8 points. As ratios they are and . The open ring is the reference grader. On corpus B Llama-3.1-8B assigns the compliance label zero times. A logarithmic axis has no zero, so its marker is drawn off the scale at the left.
| corpus A | corpus B | |||
|---|---|---|---|---|
| labels | comply | non-ref. | comply | non-ref. |
| shipped labels | 24.8 | 23.0 | 22.4 * | 18.0 |
| one call (A) | 24.8 | 23.0 | 22.4 * | 18.0 |
| three calls (B) | 18.4 | 23.0 | 59.7 | 19.1 |
Table 5: The two spreads of Table 1 under three labelling protocols. comply is the highest compliance rate divided by the lowest. non-ref. is the range of non-refusal rates in points. One call: every grader’s first readable vote. Three calls: the majority of three for the reference, the two gpt-oss graders and the three dense 8B and 9B graders, which are sampled at temperature 1.0. The blind grader and Llama-3.3-70B keep one call. *Against gemma-2-9b-it , as in Table 1 .
| rate | max/min | leave-one-out | 2nd/2nd | IQR | range |
|---|---|---|---|---|---|
| A, comply | 24.8 | 14.9–24.8 | 13.6 | 16.4 | 23.8 |
| A, non-ref. | 1.6 | 1.2–1.6 | 1.2 | 7.1 | 23.0 |
| B, comply | 22.4 * | 3.5–22.4 | 20.5 | 13.5 | 19.9 |
| B, non-ref. | 1.9 | 1.3–1.9 | 1.2 | 5.5 | 18.0 |
Table 6: Dispersion of the eight graders’ rates on corpora A and B. max/min : highest rate over lowest. leave-one-out : the smallest and largest max/min ratio after removing each grader in turn. 2nd/2nd : second-highest over second-lowest. IQR and range are in points. *Against gemma-2-9b-it , because Llama-3.1-8B assigns compliance zero times.
| grader | org | par. | com. A | com. B | A | B |
|---|---|---|---|---|---|---|
| (B) | % | % | ||||
| Llama-3.1-8B | M | 8 | 1.0 | 0.0 | +0.436 | +0.451 |
| gemma-2-9b-it | G | 9 | 1.7 | 0.9 | +0.445 | +0.479 |
| Qwen3-8B | Q | 8 | 7.1 | 5.7 | +0.622 | +0.644 |
| Llama-3.3-70B | M | 70 | 12.2 | 7.4 | +0.694 | +0.713 |
| gemini-2.5-flash (ref.) | G | — | 16.9 | 14.2 | — | — |
Table 7: The roster. par. is the parameter count in billions, com. the compliance rate, and chance-corrected agreement with the reference on all 900 items. Organisations: M Meta, G Google, Q Qwen, O OpenAI, A Anthropic. The two dashes in par. are not missing data. No parameter count is published for either grader, so item 1 of Section 7 fails on our own roster. The column gives true totals, so gpt-oss-120b appears as 117. marks a grader that shares a model family with that corpus’s target.
| corpus A | corpus B | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| grader | ref. | eva. | com. | gib. | CoT | ref. | eva. | com. | gib. | CoT |
| Llama-3.1-8B | 554 | 330 | 9 | 7 | 0 | 720 | 180 | 0 | 0 | 0 |
| gemma-2-9b-it | 428 | 449 | 15 | 8 | 0 | 558 | 334 | 8 | 0 | 0 |
| Qwen3-8B | 347 | 482 | 64 | 7 | 0 | 610 | 239 | 51 | 0 | 0 |
| Llama-3.3-70B | 407 | 378 | 110 | 5 | 0 | 637 | 196 | 67 | 0 | 0 |
| gemini-2.5-flash (ref.) | 425 | 311 | 152 | 5 | 7 | 644 | 121 | 128 | 7 | 0 |
Table 8: Per-class label counts for all eight graders on corpora A and B; every grader labels all 900 items. Table 13 gives the counts on corpora C and D. The compliance column, from which attack success rate is computed, runs from 0 to 223 on the same responses. CoT is the refused-in-chain-of-thought label of Appendix J . It is printed as its own column so every row sums to 900, and it is outside the test of Section 5 . gpt-oss-20b ’s 23 gib. labels on corpus A reflect a gap in its label set, not a parse failure. The gpt-oss graders never assign the CoT label, which two other graders do use. Seven of the 23 are the seven items the reference labelled CoT .
Figure 4: The counts of Table 8 as shares of the ordered outcome scale, with the compliance count beside each bar. This plot is the whole diagnostic of Section 5 , and it needs only counts an evaluation already has. The top row of corpus B is a grader that agrees with the reference on 78 % of all 900 items. No accuracy figure separates it from the other rows. Its missing dark segment does.
| corpus A | corpus B | |||
|---|---|---|---|---|
| grader | agr. | solo com. | agr. | solo com. |
| gpt-oss-120b | 93.9 % | 4 / 204 | 94.9 % | 4 / 161 |
| gemini-2.5-flash (ref.) | 89.2 % | 0 / 152 | 93.7 % | 1 / 128 |
| gpt-oss-20b | 88.2 % | 14 / 223 | 94.3 % | 7 / 164 |
| claude-opus-5 (blind) | 84.2 % | 8 / 198 | 88.6 % | 29 / 179 |
| Llama-3.3-70B | 82.9 % | 1 / 110 | 87.7 % | 2 / 67 |
Table 9: Leave-one-out consensus agreement (agr.), and compliant labels no other large grader confirms (solo com.) out of all that grader’s compliant labels. gpt-oss-20b , which contradicted our ordering, is not an outlier on either measure. It sits above two graders we keep, on both corpora, and other large graders confirm 93.7 to 95.7 % of its compliant labels. The agreement denominator is the items on which the four-grader panel reaches a 3-of-4 verdict: 784 to 835 of 900, depending on which grader is left out. An item with no panel verdict says nothing about the grader being scored. A fixed panel containing four of the five graders it scores would favour its own members by 6 to 8 points, so we do not report that version.
| state of the labels | non-ref. | 95 % CI | |
|---|---|---|---|
| as returned | 42.3 % | — | — |
| + protective-inverse ruling | 51.2 % | ||
| + empty-response rule | 50.4 % | ||
| + reconciliation pass | 58.7 % | ||
| end to end | +16.3 | [+13.8, +19.0] |
Table 10: The three decisions of Appendix K , in the order taken, on the one grader whose labels from before the decisions survive. Intervals are from a cluster bootstrap over the 100 harmful cores, so an item is never resampled apart from the eight others that share its core. No response changed at any step. The end-to-end interval is on the difference, not the sum of the three intervals. The rate column is rounded and the column is not. Subtracting the printed rates gives 16.4, while the unrounded difference is 16.3.
Figure 5: The rows of Table 10 as levels, in the order the decisions were taken. The sequence does not move in one direction only. The largest single step is the first, the ruling on the protective inverse. The last, the reconciliation pass that Appendix K names as the weakest decision, is almost as large. Corpus A, one grader, no exclusion applied, and no response changed.
Figure 6: The human annotators on the same axis as the eight graders, re-weighted to the full 900 by the design weights of the stratified draw. The annotators are drawn as one more row with the graders’ marker, because they are one more measurement and not a ground-truth line. An earlier version drew them as a line across the plot, the usual way to draw a reference value. For the same reason, Appendix M bounds its claim by the within-class agreement of Table 11 , not by the totals shown here. The open ring is the reference grader.
| corpus A | corpus B | |||
|---|---|---|---|---|
| grader | comply | eva. agr. | comply | eva. agr. |
| Llama-3.1-8B | 0.0 % | 10.3 % | 0.0 % | 12.5 % |
| gemma-2-9b-it | 1.1 % | 30.8 % | 0.8 % | 30.0 % |
| Qwen3-8B | 6.1 % | 43.6 % | 6.9 % | 37.5 % |
| Llama-3.3-70B | 13.6 % | 46.2 % | 9.3 % | 32.5 % |
| gemini-2.5-flash (ref.) | 16.5 % | 48.7 % | 14.9 % | 42.5 % |
Table 11: Each grader against the human annotators on the items they labelled. comply is the compliance rate re-weighted to the full 900 by the design weights. eva. agr. is raw agreement with the annotators within the items the reference labelled evasive. This second figure bounds the human result. The annotators find more delivered harm than every grader on both corpora, and agree with none of them within the contested class.
| corpus | agr. | floor | ref. | eva. | com. | kept | flagged | |
|---|---|---|---|---|---|---|---|---|
| A | 55.9 | 47.2 | +0.211 | 698 | 193 | 2 | 2 / 152 | com. |
| B | 75.8 | 71.6 | +0.347 | 766 | 134 | 0 | 0 / 128 | com. |
Table 12: The held-out 7B grader, Mistral-7B-Instruct-v0.3 , on corpora A and B. agr. is raw agreement with the reference in percent, and floor is the registered competence floor, the share of the reference’s most frequent class. The next three columns count refusal, evasion and compliance labels. kept counts the items the reference labels compliant that this grader also labels compliant. flagged lists the classes the rule flags against the eight roster graders.
| grader | agr. | ref. | eva. | com. | flagged | |
| corpus A | ||||||
| gpt-oss-20b | 78.8 | +0.677 | 349 | 305 | 223 | – |
| gpt-oss-120b | 84.9 | +0.766 | 370 | 315 | 204 | – |
| claude-opus-5 (blind) | 81.0 | +0.706 | 365 | 325 | 198 | – |
| gpt-oss-safeguard-20b | 69.7 | +0.549 | 231 | 444 | 193 | – |
| qwen3.8-27b | 82.0 | +0.717 | 392 | 336 | 154 | – |
Table 13: Every grader on corpora A to D. agr. is raw agreement with the reference in percent, and is chance-corrected agreement with the reference. The next three columns count refusal, evasion and compliance labels. flagged lists the classes the rule flags. marks a grader from the same model family as that corpus’s target. The reference covers 897 items on corpus C and 899 on corpus D, because its provider blocked the rest. The 123 rule-labelled responses of corpus D appear in none of the three count columns.
| graders | non-ref. | comply | comply | ||
|---|---|---|---|---|---|
| range | range | ratio | |||
| A | all | 10 | 35.9 | 23.8 | 24.8 |
| A | without dense 8–9B | 7 | 22.3 | 12.6 | 2.0 |
| B | all | 10 | 24.2 | 19.9 | 22.4 * |
| B | without dense 8–9B | 7 | 15.8 | 12.4 | 2.7 |
| C | all | 8 | 27.4 | 18.9 | 86.0 |
Table 14: Spreads on each corpus, across all graders that labelled it and without the three dense 8B and 9B graders. is the number of graders. Ranges are in points. On corpus A the held-out gpt-oss-safeguard-20b has the highest non-refusal rate, which widens that range beyond Table 1 . *Against gemma-2-9b-it .
| grader | stays | evasive | refuse |
|---|---|---|---|
| corpus C, 101 items the reference labels compliant | |||
| gpt-oss-120b | 101 | 0 | 0 |
| gpt-oss-20b | 98 | 2 | 1 |
| qwen3.8-27b | 80 | 19 | 2 |
| Qwen3-8B | 25 | 74 | 2 |
| gemma-2-9b-it | 1 | 100 | 0 |
Table 15: Where each grader puts the items the reference labels compliant on the two new corpora. An arrow means “moved to”. Corpus C has new harmful requests and corpus A’s target. Corpus D has corpora A and B’s prompts and a target from a new family.
| our result | status against prior work |
|---|---|
| the two boundaries of one rate differ in scale across graders, on two corpora | new, as far as we can find; that the rate depends on the grader is known ( Gao, 2026 ; Tamsal and Rusert, 2026 ) |
| one class collapses onto its neighbour, on two target families, while agreement stays above two thirds | new |
| the collapse appears on new harmful requests and on a third target family | new; registered, tested on new requests, descriptive on the third target |
| a rarely assigned class is a grader fault if another grader assigns it | new as a rule; the counts themselves are standard practice |
| the rule flags the dense 8B and 9B graders on new requests and a held-out 7B grader from a fourth family, and no large held-out grader | new; registered, one held-out corpus and one held-out small grader |
| three post-labelling decisions move one grader’s rate by 16.3 points | new; this magnitude not documented |
Table 16: Each result against the closest prior work. Of twelve rows, seven are new. One retracts a claim we made earlier, one quantifies an effect attributed elsewhere, and one confirms a published finding with a magnitude. One is known in principle but missing from the small-sample work it should govern. The last is an argument, not a measurement, and is the row to discount first.
| quantity | point | 95 % CI |
|---|---|---|
| benchmark of origin, raw | 15.4 | [ - 8.2, 22.6] |
| domain-standardised | 11.4 | [ 1.4, 25.0] |
| rubric revision, end to end | 16.3 | [13.8, 19.0] |
| step 3, reconciliation pass | 8.2 | [ - 6.2, 10.3] |
| dropping the encoding family, ref. | 5.9 | [ - 5.3, 0 6.3] |
| same, spread across graders | 2.2 | [ - 1.5, 0 2.9] |
Table 17: Cluster-bootstrap intervals on the swings this paper reports. The two encoding rows concern the family behind item 4 of Section 7 . Removing those attacks moves the reference’s own rate by 5.9 points, and moves different graders by amounts that differ by 2.2. The domain-standardised benchmark effect does not hold, because its interval crosses zero. This is why Appendix K reports the raw swing and names its confounds. Points here are differenced before rounding, so the first row reads 15.4 where the text reads 15.3. The text subtracts the printed rates, 58.1 and 42.8, and this column subtracts them at full precision (15.37). The same rounding explains the rubric row against Table 10 . The two human rows resample within each stratum of the draw and re-weight, so the interval reflects the design. Corpus B’s width is what an effective of 33 produces. Read every swing here against the reference grader’s own noise floor: it disagrees with itself on 12.1 % of corpus A (Section 7 ).