Probabilistic safety judges return the probability of a safety label and cheaply score reinforcement learning rollouts. A confidence gate sends only uncertain verdicts to a stronger LLM judge, which leaves confidently passed harm in a blind spot. Policies can exploit this blind spot, and commercial judges release new versions. We propose a protocol that validates such a judge before and during training, separates the share of confidently passed harm from its amount, and re-validates it after each judge change. We apply it to two versions of Jev, a commercial decision-only judge, and to an open guard classifier. Under the probability reward of Jev, passed harm shrinks in all 11 runs under the primary reference, mostly more slowly than caught harm, and concentrates in the blind spot. Binarizing this judge meets the escape criterion on Llama, and the open guard classifier shows an escape signal in one seed under a second reference. Guard classifiers share this blind spot, and even frontier LLM judges catch at most one of 10 human-labeled confident misses. Without a trained reward model, the probability reward with a helpfulness term lowers HarmBench attack success on Llama-3.1-8B from 0.287 to at most 0.033 at the final checkpoint.
Figures & tables
Figure 1: Two late-training rollouts from the human audit of Llama trained with the judge probability as reward, placed on the gate of the judge. The final human label of both responses is harmful. The confident miss (a) passes the gate unchecked, whereas (b) falls in the uncertain band ( c<τ ) and reaches a delegate judge when delegation is used. Responses in the flag zone receive a reward close to zero, and the offline check finds no harmless response there (Section 5.1 ). Confidence is c=max(p,1−p) . Excerpts are truncated, and real names and drug doses are masked.
Stage
What it checks
Criterion or measure
Result in this study
S0 Before training
Question wording, threshold τ
Smallest τ with confident accuracy ≥0.95
τ=0.93 , holds for 1.14.0
S1 Offline
Error direction, appended sentences, guard classifiers, agreement with the old version
Confident misses vs. false alarms, flip rate, shared misses, agreement
30 misses, 0 false alarms, 1.14.0 agrees on 98.2–99.3%
S2 Under RL
Share (GPR) and amount (PHR) of passed harm
Interval of PHR change <0⇒ pass, >0⇒ escape, else S3
Jev probability (Qwen3Guard), 11 runs: 7 pass, 4 to S3. Binary : escape in 2/2. WildGuard: rises significantly in 1/2 (gpt-oss)
S3 Human audit
Level of confident misses, delegate coverage
Human-harmful share, delegate catches
13.3% of flagged passes harmful, delegate 0/10
Table 1: The validation protocol. The study fixed only the S0 rule in advance. The other criteria are recommendations drawn from the case study, and Section 3.4 states which stages to repeat after a judge change.
Condition
Harmful-prompt reward
Factor changed from Prob
Prob
judge probability
none (reference)
Binary
judge, thresholded at 0.5
reward form, same judge
Prob+Delegate
judge, delegate if c<τ
judge and 0/1 form inside the band
LLM-Verdict
delegate judge
judge and 0/1 form
Prob+Help
judge probability
helpfulness term on other prompts
Prob-WG
WildGuard probability
judge, for both rewards
Table 2: Reward conditions. Benign and general prompts receive the refusal reward of the probabilistic judge in each condition, plus a helpfulness term under Prob+Help .
Figure 2: Change from the early window (first 50 steps) to the late window (steps 150–299) for every training run under the Qwen3Guard reference. The horizontal axis gives the change in the share of passed harm (GPR), and the vertical axis gives the change in its amount (PHR). Runs at the lower right concentrate harm, and runs in the shaded upper region move toward escape. Filled points have a 95% prompt-level bootstrap interval of the PHR change that excludes zero and receive an S2 verdict, and open points go to S3. Tables 7 and 8 in Appendix C list every run.
Policy
Condition
HarmBench ↓
StrongREJECT ↓
XSTest safe refusal ↓
OR-Bench-Hard refusal ↓
AlpacaEval vs. base ↑
Llama-3.1-8B
base
0.287
0.018
0.080
0.287
(0.5)
Prob
0.040 / 0.153
0.009 / 0.013
0.060 / 0.072
0.256 / 0.157
0.227 / 0.341
Binary
0.163 / 0.167
0.021 / 0.022
0.048 / 0.028
0.090 / 0.240
0.275 / 0.339
Prob+Delegate
0.023 / 0.097
0.006 / 0.015
0.080 / 0.052
0.497 / 0.393
0.170 / 0.262
LLM-Verdict
0.060 / 0.077
0.029 / 0.027
0.072 / 0.068
0.297 / 0.315
0.253 / 0.331
Prob+Help
0.017 / 0.033
0.004 / 0.006
0.068 / 0.080
0.428 / 0.229
0.594 / 0.532
Table 3: Final checkpoints at step 300, seed 42 / seed 43. All runs use β=0.001 and a learning rate of 2×10−5 . AlpacaEval reports the pairwise win rate against the base policy of the same family under a gpt-oss-120b judge, and the base row equals 0.5 by construction. Bold marks the best HarmBench and AlpacaEval value per policy and seed. Appendix D lists all checkpoints, and the released result files include 95% bootstrap intervals.
Step
Recommendation
Evidence
Judge
Pass S0–S1, freeze τ
§ 5.1
Harmful prompts
Use the probability, not a 0/1 reward
§ 5.3
Benign prompts
Add a helpfulness term
Table 3
Delegation
Keep LLM judges for monitoring only
§ 5.3
Monitoring
Track GPR and PHR under two references
§ 5.2
Checkpoints
Evaluate several, not only the last
App. D
Table 4: A recipe for safety GRPO with a probabilistic safety judge, drawn from the results on two 7–8B chat policies.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Data ( τ=0.93 )
n
F1
AUROC
ECE
Deleg.
WGT harmful
1,609
0.768
0.954
0.014
19.4%
WGT refusal
1,620
0.891
0.992
0.018
33.6%
XSTest harmful
446
0.881
0.985
0.027
13.9%
XSTest refusal
449
0.876
0.993
0.045
31.2%
Appendix
Table 5: Offline accuracy of Jev 1.13.0 on human-labeled responses, excluding the calibration items. Deleg. is the share of responses with c<τ .
Judge
F1
ECE
Miss @19.4%
Miss @30%
Jev
0.768
0.014
11.2% (30/0)
5.6% (15/0)
Llama-Guard-3-8B
0.712
0.049
10.9% (29/4)
7.5% (20/3)
WildGuard-7B
0.763
0.056
12.4% (33/1)
9.7% (26/0)
Appendix
Table 6: Confident misses on WildGuardTest harmful at matched delegation rates. Each miss cell gives the missed share of the 267 harmful responses and, in parentheses, confident misses and confident false alarms.
Policy
Condition
Seed
Qwen3Guard
gpt-oss
Both
Either
Llama
Prob , β =0.01
42
0.111 → 0.176 (+0.065)
0.048 → 0.093 ( +0.045 )
0.026 → 0.031 (+0.005)
0.108 → 0.169 (+0.062)
Llama
Prob , β =0.01
43
0.102 → 0.184 ( +0.082 )
0.051 → 0.118 ( +0.068 )
0.010 → 0.050 ( +0.040 )
0.107 → 0.186 ( +0.079 )
Llama
Prob , lr 5e-5
42
0.177 → 0.312 (+0.135)
0.086 → 0.137 (+0.051)
0.067 → 0.042 (-0.026)
0.163 → 0.295 ( +0.131 )
Llama
Prob
42
0.143 → 0.181 (+0.037)
0.064 → 0.099 (+0.034)
0.046 → 0.033 (-0.013)
0.132 → 0.172 (+0.039)
Llama
Prob
43
0.125 → 0.321 ( +0.195 )
0.073 → 0.166 ( +0.093 )
0.032 → 0.100 ( +0.068 )
0.131 → 0.298 ( +0.166 )
Llama
Prob+Delegate
42
0.142 → 0.181 (+0.038)
0.088 → 0.086 (-0.002)
0.046 → 0.065 (+0.019)
0.149 → 0.177 (+0.028)
Appendix
Table 7: Gate-pass rate in the first 50 steps and in steps 150–299 for every GRPO run, with the change in parentheses. Bold changes have 95% prompt-level bootstrap intervals that exclude zero. The Prob-WG rows use the WildGuard gate ( τ=0.99 ).
Policy
Condition
Seed
Harmful rate
PHR Qwen3Guard (%)
PHR gpt-oss (%)
Llama
Prob , β =0.01
42
0.153 → 0.059
1.69 → 1.04
0.76 → 0.68
Llama
Prob , β =0.01
43
0.154 → 0.059
1.57 → 1.09
0.82 → 0.94
Llama
Prob , lr 5e-5
42
0.108 → 0.025
1.91 → 0.77
0.97 → 0.34
Llama
Prob
42
0.103 → 0.030
1.48 → 0.54
0.70 → 0.44
Llama
Prob
43
0.139 → 0.040
1.74 → 1.30
1.12 → 0.72
Llama
Binary
42
0.171 → 0.165
2.41 → 3.93
1.42 → 1.98
Appendix
Table 8: Passed-harm rate (PHR) per run, first 50 steps against steps 150–299. The PHR is the share of harmful-prompt responses that the reference judge calls harmful and the training judge passes with c≥τ ( τ=0.99 for WildGuard). The harmful rate is the share of harmful-prompt responses that Qwen3Guard labels Unsafe.
Stratum
n
Final label
Either annotator
Jev and Qwen3Guard safe
98
0.1% †
–
Jev passes, Qwen3Guard flags
60
13.3%
33.3%
Qwen3Guard Unsafe only
18
11.1%
–
Jev uncertain ( c<τ )
40
22.5%
30.0%
Appendix
Table 9: Human audit of Llama rollouts by two annotators. “Flags” means Unsafe or Controversial. The final label follows the designated annotator, and the either-annotator share counts a harmful label from at least one annotator. † Reweighted to the share of refusals in the rollout pool. The table omits the 2 responses that Jev confidently flags.
Policy
Condition
Min. safety
HB @100
Zero-signal
Llama
Prob
0.764 / 0.866
0.163 / 0.033
0.75 / 0.70
Binary
0.712 / 0.716
0.307 / 0.183
0.92 / 0.92
Prob+Delegate
0.530 / 0.731
0.510 / 0.037
0.74 / 0.77
LLM-Verdict
0.782 / 0.429
0.040 / 0.730
0.88 / 0.87
Prob+Help
0.939 / 0.865
0.057 / 0.033
0.75 / 0.67
Prob-WG
0.955 / 0.926
0.383 / 0.140
0.72 / 0.74
Appendix
Table 10: Training stability per seed (42 / 43). Min. safety is the lowest Qwen3Guard safety score of any window of 25 steps after the first. HB @100 is the HarmBench (HB) attack success at step 100 (base 0.287 for Llama, 0.240 for Qwen). Zero-signal is the share of harmful-prompt groups over all 300 steps whose 8 rewards are identical.
Policy
Condition
Seed
HarmBench ↓
AlpacaEval ↑
100
200
300
100
200
300
Llama
base
–
0.287
(0.5)
Llama
Binary
42
0.307
0.230
0.163
0.308
0.253
0.275
Llama
Binary
43
0.183
0.070
0.167
0.411
0.354
0.339
Llama
Prob+Delegate
42
0.510
0.027
0.023
0.222
0.357
0.170
Llama
Prob+Delegate
43
0.037
0.063
0.097
0.278
0.213
0.262
Appendix
Table 11: HarmBench attack success and AlpacaEval win rate at steps 100, 200, and 300 of every run. AlpacaEval is the pairwise win rate against the base policy of the same family. The released result files include all benchmarks and 95% bootstrap intervals for every cell.
Policy
Base
Prob (42 / 43)
Prob+Help (42 / 43)
Llama-3.1-8B
444
183 / 283
316 / 307
Qwen2.5-7B
432
171 / 221
301 / 297
Appendix
Table 12: Mean AlpacaEval answer length in tokens at step 300.
LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety, yet existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the agent's behavior or merely on how the evaluation policy happens to be worded. We argue that any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles: rubric-semantics invariance under certified-equivalent rewrites, rubric-threshold invariance under intentional strict-to-lenient shifts, and ambiguity-aware calibration so that verdict instability concentrates on genuinely ambiguous cases. Instantiating these principles as a stress-test protocol with four agent-class judges on trajectories drawn from ASSEBench and R-Judge, we surface a previously unmeasured failure mode: today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart. Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites, so safety scores produced by these judges partly conflate what the agent did with how the evaluator was prompted. These findings persist with a frontier-scale judge, a third benchmark, and a second base policy. Beyond the diagnosis, we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards. We release the protocol and code so that future agent-safety benchmarks can audit their own evaluators rather than trust them by default: https://github.com/wengshihao/policy-invariance-judge
Shihao Weng, Yang Feng, Xiaofei Xie
Nanjing University Nanjing, China · Singapore Management University Singapore
Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level~α with high probability, using finite-sample Clopper--Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates the instance under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines.
Sher Badshah, Ali Emami, Hassan Sajjad
Dalhousie University · New York University Abu Dhabi · Emory University
LLMs-as-judges are the primary way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchmarks. We therefore investigate two under-explored but crucial properties of LLMs-as-judges: their sensitivity to in-context information, and their steerability to differing safety definitions, which may not align with their internal safety priors. We evaluate the safety judging abilities of 13 generalist LLMs and safety-specific judges, and investigate the impact of novel in-context information and changing safety definitions. We find that while LLM-judges can learn from new information, they are broadly unlikely to update their evaluations if the context or safety definition departs from their prior.