What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency
Organizations: M.S. in Computer Science, University of Southern California. · M.S. in Computer Science, Illinois Institute of Technology. · M.S. in Computer Science, New York University.
Abstract
LLM judges often check, against an amendment record, whether the documents an answer cites are still in force. Such records are rarely complete. From a table of replacements, our test generates matched answers citing a replaced document, its successor, the replaced document before its replacement, and an older document that a newer one updated without replacing; it removes links from the record and scores each finding by its stated reason. On 48 replacements from the IETF RFC index, judges given a correct record caught nearly every replaced citation, as a lookup would. On the 24 faults whose link we removed, every judge caught more from the texts alone than with the record: Gemini 3.1 Pro 23/24 against 1/24, GPT-5.5 23/24 against 9/24, Claude Opus 4.5 17/24 against 9/24 and Gemma 4 26B 13/24 against 0/24. The drop recurred on a second draw of removed links. Most misses carried no finding; in an exploratory run with a conflict code, only Gemini used it often. Rewording the gap as "no replacement recorded" did not help; presenting the record as an unverified note, or warning that it was incomplete, restored most lost catches for Gemini, GPT-5.5 and Claude (significant after correction for the first two) but at most 5 of 13 for Gemma; where catches returned, so did false alarms on updates. With both texts, judges mostly told replacements from updates; every update flagged had a near-identical title. A descriptive case study on Indian rooftop-solar rules is appended; we release the harness.
Figures & tables
| Context | The judge is given, besides both texts |
|---|---|
| both-texts | nothing else |
| status | a line from the correct record: the cited document’s status on the date |
| status-damaged | the line from a record with half its links dropped, where a dropped link reads “in force” (authoritative form) |
| norecord-damaged | as above, the line reading “no replacement recorded” (wording) |
| lowauth-damaged | as status-damaged , headed as an unverified note from an automated tagger (authority) |
| caveat-damaged | as status-damaged , with the warning that the record is incomplete and may omit replacements (warning) |
| Faults | Controls | Date controls | Decoys | |||||
|---|---|---|---|---|---|---|---|---|
| Judge | Context | Catch | any | FA | any | FA | FA | any |
| Gemma 4 26B | both-texts | 31/48 | 31/48 | 0/48 | 19/48 | 0/48 | 6/48 | 6/48 |
| status | 47/48 | 47/48 | 0/48 | 20/48 | 0/48 | 0/48 | 0/48 | |
| status-damaged | 23/48 | 23/48 | 0/48 | 20/48 | 0/48 | 0/48 | 0/48 | |
| norecord-damaged | 23/48 | 23/48 | 0/48 | 20/48 | 0/48 | 0/48 | 0/48 | |
| lowauth-damaged | 26/48 | 26/48 | 0/48 | 18/48 | 0/48 | 1/48 | 1/48 | |
| No record | Damaged record, presented as | Holm | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Judge | (texts) | record, “in force” | (bare) | “no replacement recorded” | unverified note | record + warning | H2a | wording | authority | warning |
| Gemma 4 26B | 13/24 | 0/24 | 24 | 0/24 | 2/24 | 5/24 | <.001 | 1.000 | 1.000 | .375 |
| GPT-5.5 | 23/24 | 9/24 | 15 | 10/24 | 24/24 | 23/24 | <.001 | 1.000 | <.001 | .001 |
| Gemini 3.1 Pro | 23/24 | 1/24 | 23 | 6/24 | 18/24 | 22/24 | <.001 | .375 | <.001 | <.001 |
| Claude Opus 4.5 | 17/24 | 9/24 | 13 | 11/24 | 17/24 | 17/24 | .008 | 1.000 | .062 | .062 |
| Seed 0 | Seed 1 | Pooled over unique links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Judge | Caught | Lost/gained | Caught | Lost/gained | Shared | Links | Drop | Lost/gained | |||
| Gemma 4 26B | 13/24 0/24 | 13/0 | <.001 | 17/24 0/24 | 17/0 | <.001 | 12 | 36 | +61 [+44, +78] | 22/0 | <.001 |
| GPT-5.5 | 23/24 9/24 | 14/0 | <.001 | 24/24 10/24 | 14/0 | <.001 | 12 | 36 | +61 [+44, +75] | 22/0 | <.001 |
| Gemini 3.1 Pro | 23/24 1/24 | 22/0 | <.001 | 22/24 4/24 | 18/0 | <.001 | 12 | 36 | +81 [+67, +92] | 29/0 | <.001 |
| Claude Opus 4.5 | 17/24 9/24 | 8/0 | .008 | 19/24 12/24 | 7/0 | .016 | 12 | 36 | +31 [+17, +44] | 11/0 | <.001 |
| Same title | Different title | ||||
|---|---|---|---|---|---|
| Judge | Fault catch | Decoy FA | Fault catch | Decoy FA | Discrimination |
| Gemma 4 26B | 29/33 | 6/33 | 2/15 | 0/15 | +52 [+35, +69] |
| GPT-5.5 | 33/33 | 8/33 | 13/15 | 0/15 | +79 [+67, +90] |
| Gemini 3.1 Pro | 32/33 | 6/33 | 11/15 | 0/15 | +77 [+65, +88] |
| Claude Opus 4.5 | 30/33 | 9/33 | 7/15 | 0/15 | +58 [+42, +73] |
| Condition | Faults | Matched correct answers | Clauses |
|---|---|---|---|
| S-both (old and current clause shown) | 42 | 42 | 14 |
| S-swap (old clause in place of current) | 38 | 7 | 13 |
| C (clean correct answers) | – | 42 | – |
| Condition | Reviewer sees | Reviewer does not see | Compared with; confound | Role |
|---|---|---|---|---|
| R0 | old clause (S-swap), source metadata removed | current clause, dates, issuer | – | ladder |
| R1 | old clause with issuer, dates, title | current clause | R0 : metadata only | ladder |
| R2 | old and current clause (S-both) | any amendment record | R1 : adds the current clause | ladder; Q2 baseline |
| R3 | R2 + amending instrument’s words | status, dates, target | R2 | ladder |
| R4 | R2 + registry status line | – | intact: told the answer for linked clauses | reference |
| R4 ′ | R1 + registry status line | current clause | R1 : status line without the text | ladder |
| Model | Arm | Reviewer sees | TP | FN | FP | TN | Rec. | Spec. | Prec. | Bal. | FA C | Disc. [95% CI] | Cl. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 14B | R2 | both versions | 13 | 29 | 18 | 24 | 31 | 57 | 42 | 44 | 12/42 | 7 [ 38, +20] | 14 |
| Qwen 14B | R4 | status line | 32 | 10 | 10 | 32 | 76 | 76 | 76 | 76 | 11/42 | +55 [+26, +79] | 14 |
| Qwen 14B | R4n | no-record wording | 31 | 11 | 13 | 29 | 74 | 69 | 70 | 71 | 12/42 | +47 [+17, +73] | 14 |
| Lookup | R5 | lookup (ref.) | 31 | 11 | 5 | 37 | 74 | 88 | 86 | 81 | 5/42 | +67 [+43, +88] | 14 |
| Qwen 14B | R4-drop50 | status, drop 50% | 88 | 122 | 54 | 156 | 42 | 74 | 62 | 58 | 55/210 | +21 [ 6, +48] | 14 |
| Qwen 14B | R4n-drop50 | no-record, drop 50% | 95 | 115 | 58 | 152 | 45 | 72 | 62 | 59 | 60/210 | +23 [ 6, +54] | 14 |
| qwen2.5:14b | gemma-4-26b | |||||
|---|---|---|---|---|---|---|
| Registry | R4 | R5 | R4 R5 | R4 | R5 | R4 R5 |
| Intact | +55 | +67 | 11 | +21 | +67 | 45 |
| Drop 25% | +39 | +51 | 12 | +14 | +51 | 37 |
| Drop 50% | +21 | +42 | 21 | +4 | +42 | 38 |
| Spurious 10% | +53 | +62 | 10 | +20 | +62 | 42 |
| Model | Contrast | Clauses | Diff. | 95% CI | (Holm) | S1 | S2 |
|---|---|---|---|---|---|---|---|
| Gemma 4 26B | Q1, intact: R4 R5 discrimination | 14 | 45 | [ 69, 24] | 0.043 | 47 | 50 |
| Gemma 4 26B | Q1, 50% dropped | 14 | 38 | [ 65, 11] | 0.178 | 38 | 50 |
| Gemma 4 26B | Q1, 10% spurious | 14 | 42 | [ 66, 22] | 0.043 | 45 | 45 |
| Gemma 4 26B | Q2: R2 R4-damage catch, dropped links | 9 | +1 | – | 1.000 | +4 | – |
| Gemma 4 26B | R4n R4 catch, dropped links | 9 | +9 | – | 1.000 | +20 | – |
| Qwen 14B | Q1, intact: R4 R5 discrimination | 14 | 11 | [ 33, +9] | 1.000 | 12 | +17 |
| qwen2.5:14b | gemma-4-26b | ||||||
| Rule | R0 | R2 | R4 | R0 | R2 | R4 | |
| Catch | any finding | 40% | 62% | 79% | 50% | 76% | 79% |
| clause named | 40% | 62% | 79% | 48% | 76% | 79% | |
| right reason | 8% | 33% | 79% | 0% | 24% | 64% | |
| False alarm | any finding | 40% | 40% | 23% | 40% | 45% | 43% |
| validity reason | 40% | 23% | 15% | 20% | 0% | 4% | |
| llama3.1:8b | qwen2.5:7b | qwen2.5:14b | gemma-4-26b | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rung | Catch | FA-m | FA-C | Catch | FA-m | FA-C | Catch | FA-m | FA-C | Catch | FA-m | FA-C |
| R0 | 0% | 80% | 51% | 4% | 80% | 20% | 8% | 40% | 26% | 0% | 40% | 34% |
| R1 | 0% | 80% | 49% | 4% | 60% | 12% | 8% | 50% | 27% | 0% | 20% | 36% |
| R2 | 0% | 49% | 49% | 18% | 18% | 12% | 33% | 40% | 27% | 24% | 45% | 36% |
| R3 | 0% | 49% | 49% | 18% | 20% | 19% | 30% | 36% | 27% | 33% | 45% | 36% |
| R4 | 11% | 49% | 51% | 7% | 7% | 5% | 79% | 23% | 26% | 64% | 43% | 39% |
| Model | Planned | Arm | Catch | FA | Any | Disc. / diff. |
|---|---|---|---|---|---|---|
| Qwen 7B | yes | R2 | 5/42 | 7/42 | 14/42 | 1 |
| Qwen 7B | yes | R4 | 2/42 | 2/42 | 3/42 | +0 |
| Qwen 7B | yes | R4-drop50 | 2/42 | 2/42 | 3/42 | +0 |
| Qwen 7B | yes | R4 R5, intact | 67 [ 88, 43] | |||
| Qwen 7B | yes | R4 R5, drop50 | 42 [ 60, 24] | |||
| Qwen 14B | yes | R2 | 13/42 | 18/42 | 26/42 | 7 |
| Replaced clause | Qwen 14B | Gemma 4 26B | ||||||
|---|---|---|---|---|---|---|---|---|
| R2 | R4 | R4n | R5 | R2 | R4 | R4n | R5 | |
| DL-C3-004#cl-3.8 | 2/4 | 4/4 | 4/4 | 8/8 | 0/4 | 4/4 | 4/4 | 8/8 |
| DL-C3-004#cl-9.2b | 2/2 | 2/2 | 2/2 | 4/4 | 2/2 | 2/2 | 2/2 | 4/4 |
| DL-C3-006#cl-3.2 | 2/3 | 3/3 | 3/3 | 6/6 | 3/3 | 3/3 | 3/3 | 6/6 |
| DL-C3-009#cl-3.6 | 2/4 | 4/4 | 4/4 | 8/8 | 1/4 | 4/4 | 4/4 | 8/8 |
| GJ-C3-001#cl-13.5-check | 0/4 | 4/4 | 4/4 | 8/8 | 0/4 | 2/4 | 3/4 | 8/8 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| All faults | Faults whose link was dropped | ||||||
|---|---|---|---|---|---|---|---|
| Reviewer | No record | Correct | Damaged | No record | “In force” | “No replacement” | Bare |
| Qwen 32B | 42/48 | 48/48 | 33/48 | 22/24 | 9/24 | 5/24 | 6 |
| Gemma 4 26B | 16/24 | 24/24 | 12/24 | 10/12 | 0/12 | 0/12 | 12 |
| GPT-OSS 120B | 16/24 | 18/24 | 15/24 | 8/12 | 6/12 | – | 4 |
| Gemini 3.1 Pro | 22/24 | 24/24 | 12/24 | 12/12 | 0/12 | 3/12 | 12 |
| GPT-5.5 | 23/24 | – | 16/24 | 11/12 | 4/12 | 6/12 | 8 |
| Context | Qwen 32B | Gemma 4 26B | GPT-OSS 120B | Gemini 3.1 Pro | GPT-5.5 | Claude Opus 4.5 |
|---|---|---|---|---|---|---|
| cite-only | 38 | +12 | – | +46 | – | +21 |
| cited-text | 69 | – | – | – | – | +25 |
| both-texts | 2 | +67 | +67 | +92 | +96 | +71 |
| both-masked | 19 | +62 | +54 | +83 | +92 | +75 |
| both-explicit | 2 | – | +88 | +100 | +100 | +92 |
| status | +77 | +100 | +71 | +100 | – | +100 |
| Context | Qwen 32B | Gemma 4 26B | GPT-OSS 120B | Gemini 3.1 Pro | GPT-5.5 | Claude Opus 4.5 |
|---|---|---|---|---|---|---|
| cite-only | 31 | +12 | – | +46 | – | +21 |
| cited-text | 69 | – | – | – | – | +21 |
| both-texts | +40 | +67 | +62 | +92 | +96 | +75 |
| both-masked | +35 | +62 | +50 | +83 | +92 | +75 |
| both-explicit | +46 | – | +83 | +100 | +100 | +100 |
| status | +100 | +100 | +75 | +100 | – | +100 |
| Link (RFC) | Gemma 4 26B | GPT-5.5 | Gemini 3.1 Pro | Claude Opus 4.5 |
|---|---|---|---|---|
| T F N U W C | T F N U W C | T F N U W C | T F N U W C | |
| 6312 6342 | ✓ ✓ ✓ | ✓ ✓ | ||
| 8773 9973 | ✓ ✓ ✓ | ✓ | ✓ ✓ ✓ ✓ ✓ | |
| 5046 7145 | ✓ | ✓ ✓ ✓ | ✓ ✓ ✓ | ✓ ✓ ✓ ✓ ✓ ✓ |
| 8407 9907 | ✓ ✓ ✓ | ✓ ✓ ✓ ✓ | ✓ ✓ ✓ | ✓ ✓ ✓ ✓ ✓ ✓ |
| 3205 9205 | ✓ ✓ ✓ | ✓ ✓ ✓ |
| Finding | Llama 8B | Qwen 7B | Qwen 14B | Gemma 4 26B | Flash-Lite | Qwen 32B | Claude Opus 4.5 |
|---|---|---|---|---|---|---|---|
| Right-reason catch, both versions (R2) | 0/42 | 5/42 | 13/42 | 9/42 | 19/42 | 4/42 | 34/42 |
| Any-finding catch, R2 | 24/42 | 14/42 | 26/42 | 32/42 | 35/42 | 35/42 | 42/42 |
| Scoring gap at R2, any minus right reason (points) | +57 | +21 | +31 | +55 | +38 | +74 | +19 |
| Right-reason catch, status line (R4) | 3/42 | 2/42 | 32/42 | 25/42 | 33/42 | 29/42 | 35/42 |
| Correct answers flagged, R4 | 19/42 | 2/42 | 10/42 | 20/42 | 19/42 | 23/42 | 37/42 |
| R4 findings on correct answers not about validity | 13/25 | 0/2 | 4/11 | 28/30 | 19/22 | 21/24 | 44/55 |
LLM Answer Verification
Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.
See all 157 papers