Large language models (LLMs) are now widely used as judges, yet their decisions can change under presentation choices that should be irrelevant. We study one such source of instability: candidate-order sensitivity in listwise factuality evaluation, where several answers can look similarly polished while differing substantially in hallucination risk. We introduce PCFJudge, an inference-time method that reruns the same factuality-first listwise prompt over multiple orderings of the same candidate set and aggregates the resulting scores, ranks, and uncertainty signals into a single consensus decision. On RewardBench 2 Factuality, the final seven-permutation aggregate (K=7) improves top-1 selection accuracy from 86.00% to 91.33% with GPT-5.4 and from 86.33% to 89.67% with Claude Sonnet 4.6. These results suggest that candidate order can be a meaningful source of factuality-judging error and that marginalizing over this nuisance variation can improve the reliability of LLM evaluation.
Figures & tables
Model
N
Direct (%)
PCFJudge (%)
Δ (pp)
Imp./Reg.
Same
GPT-5.4
300
86.00
91.33
+5.33
21 / 5
274
Claude 4.6
300
86.33
89.67
+3.33
17 / 7
276
Micro avg.
600
86.17
90.50
+4.33
38 / 12
550
Table 1: Main RewardBench 2 Factuality results. Accuracy columns are top-1 selection accuracy percentages; Δ is measured in percentage points. Direct uses one canonical candidate order. Each PCFJudge execution uses seven candidate-order permutations with the consensus rule in Eq. 5 . The final K=7 PCFJudge numbers aggregate three separately executed K=7 evaluations on the same fixed 300-example slice. For each item, we average per-candidate consensus scores across the three executions before selecting one final winner; paired counts are then computed against the corresponding direct baseline.
Figure 1: Paired comparison against the direct judge on the final 300-example RewardBench 2 Factuality slices. Bars show only changed examples; unchanged counts are reported in Table 1 .
Model
Protocol
K=3 (%)
K=5 (%)
K=7 (%)
GPT-5.4
repeated canonical
86.67 (+0.67)
85.67 ( − 0.33)
86.00 (+0.00)
PCFJudge
89.33 (+3.33)
88.67 (+2.67)
89.33 (+3.33)
Claude 4.6
repeated canonical
86.00 ( − 0.33)
87.00 (+0.67)
86.67 (+0.33)
PCFJudge
87.00 (+0.67)
87.67 (+1.33)
88.00 (+1.67)
Table 2: Diagnostic cost and mechanism controls on RewardBench 2 Factuality. Entries are exact top-1 accuracy percentages, with absolute percentage-point change over the direct baseline in parentheses. Repeated canonical uses K calls to the same candidate order; PCFJudge uses K candidate-order permutations. This table is a matched single-run diagnostic control, while Table 1 reports the final three-run aggregate for the main K=7 setting.
Variant
ws
wB
wv
wu
GPT-5.4 (%)
Claude 4.6 (%)
Proposed
0.50
0.25
0.20
0.05
89.33
88.00
Uniform
0.25
0.25
0.25
0.25
90.00
87.00
Score only
1.00
0.00
0.00
0.00
88.67
87.00
Rank only
0.00
1.00
0.00
0.00
89.00
88.00
Top vote only
0.00
0.00
1.00
0.00
89.00
88.33
No uncertainty
0.50
0.27
0.23
0.00
89.33
88.33
Table 3: Weight sensitivity on the same diagnostic runs as Table 2 . Entries are exact top-1 accuracy percentages. Columns show the weights on mean score ( ws ), Borda rank ( wB ), top-set frequency ( wv ), and calibrated uncertainty ( wu ). Direct baselines are 86.00% for GPT-5.4 and 86.33% for Claude Sonnet 4.6.
Model
N
Direct (%)
APOCJudge (%)
Δ (pp)
Claude 4.6
100
79.09
82.33
+3.24
GPT-5.4
100
76.21
78.91
+2.70
Table 4: JudgeBench transfer results. N counts unique response pairs, while the reported value is the macro-averaged accuracy percentage over the source buckets present in the 100-pair slice, so the percentages need not be multiples of 1/N . APOCJudge uses our order-swapped pairwise protocol with keyed confirmation before accepting an override.
Figure 2: Development ablation on a fixed 100-example GPT-5.4 RewardBench 2 Factuality slice. Values are exact top-1 accuracy percentages on the development slice. Most of the recoverable error is removed by permutation consensus itself; extra overlay logic helps less than simply trusting the consensus ranker.
Pattern
Observed direct-judge behavior
Observed PCFJudge recovery
Why it matters
Unsupported specificity
Polished answers that add precise settings, dates, or source claims without solid support
Safer answers whose advantage survives across permutations
Illustrates how consensus can reduce reward for hallucinated detail rather than merely smoothing scores
Calibrated narrow correction
Broader but speculative responses that sound more complete
Narrower responses that correctly state that evidence is limited or absent
Aligns the judge with factual reliability rather than informativeness style
Near-tie candidate sets
Small stylistic differences can dominate when all candidates share the same broad factual stance
Limited changes unless an order-stable preference emerges
Explains why improvements are concentrated on genuinely unstable cases
Transfer failure mode
Pairwise estimation or ballpark prompts where an auxiliary checker can overcommit
Final transfer variant suppresses these overrides
Clarifies why JudgeBench gains are positive but smaller than RewardBench 2
Table 5: Representative qualitative patterns from prediction-file inspection. The main benefit of PCFJudge is not extra knowledge; it is greater reluctance to reward answers whose advantage depends on a particular candidate order.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.