Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment. Existing approaches rely on a model's own uncertainty -- such as token entropy or confidence scores -- but these signals fail critically on the most dangerous failure mode: confident errors, where a model is wrong but certain. In this work we introduce cross-model disagreement as a correctness indicator -- a simple, training-free signal that can be dropped into existing production systems, routing pipelines, and deployment monitoring infrastructure without modification. Given a model's generated answer, cross-model disagreement computes how surprised or uncertain a second verifier model is when reading that answer via a single forward pass. No generation from the verifying model is required, and no correctness labels are needed. We instantiate this principle as Cross-Model Perplexity (CMP), which measures the verifying model's surprise at the generating model's answer tokens, and Cross-Model Entropy (CME), which measures the verifying model's uncertainty at those positions. Both CMP and CME outperform within-model uncertainty baselines across benchmarks spanning reasoning, retrieval, and mathematical problem solving (MMLU, TriviaQA, and GSM8K). On MMLU, CMP achieves a mean AUROC of 0.75 against a within-model entropy baseline of 0.59. These results establish cross-model disagreement as a practical, training-free approach to label-free correctness estimation, with direct applications in deployment monitoring, model routing, selective prediction, data filtering, and scalable oversight of production language model systems.
Figures & tables
Figure 1: Cross-model disagreement as a label-free correctness indicator. Given a prompt x , the generator (Llama-3-8B) produces an answer y^ . The verifier (Qwen2.5-7B) performs a single forward pass over (x,y^) with no generation required. The verifier assigns low probability to the token “1942”, the generator’s confident but incorrect answer. Cross-model perplexity ( CMP ) aggregates this surprise signal into a single correctness indicator, and high CMP flags a likely error.
Figure 2: CMP and CME performance across datasets compared to baselines. (A) Mean AUROC over all model pairs. G-Ent and G-PPL measure the generator’s own entropy and perplexity; CME and CMP measure the corresponding signals from a verifier model on the generator’s answer. APGR measures the fraction of the performance gap between weak and strong model that is recovered by routing, normalized so that random routing scores 0 and oracle routing scores 1; pairs with small gaps are excluded because the small denominator in the PGR formula produces unstable estimates. Error bars show standard error across pairs; n indicates the number of pairs per dataset.
Acc.
AUROC ↑
APGR ↑
Dataset
Generator
Verifier
Acc g
Acc v
Gap
G-Ent
G-PPL
CMP
CME
G-Ent
G-PPL
CMP
CME
MMLU
Qwen-0.5B
Qwen-7B
0.42
0.72
0.30
0.588
0.556
0.841
0.583
0.521
0.501
0.793
0.497
Llama-1B
Llama-3-8B
0.43
0.62
0.19
0.654
0.655
0.817
0.679
0.562
0.605
0.869
0.540
Llama-1B
Qwen-7B
0.43
0.72
0.29
0.654
0.655
0.786
0.634
0.569
0.595
0.807
0.558
OLMo-7B †
Qwen-7B
0.56
0.72
0.15
0.503
0.599
0.896
0.697
0.486
0.546
0.914
0.447
TriviaQA
Llama-3-8B
Mistral-7B
0.43
0.64
0.20
0.859
0.885
0.825
0.922
0.822
0.831
0.849
0.789
Table 1: Selected results across all four benchmarks. Bold = best per row separately for AUROC and APGR among G-Ent, G-PPL, CMP, CME. Same-size cross-family pairs marked † . See Appendix for full results.
Figure 3: Top row: per-case signal means. Mean CMP (left axis, blue) and G-Ent (right axis, red) by outcome category. The shaded column highlights the “generator wrong only” case—confident errors the verifier does not share. On MMLU, CMP spikes 9× above the mean of the other three cases while G-Ent is flat ( 1.0× ); on TriviaQA the spike is 18× vs. 1.6× ; on GSM8K both signals rise modestly ( 3× and 1.4× ), reflecting the difficulty of isolating chain-of-thought errors with token-level signals. Bottom row: accuracy by signal quintile. Samples sorted by signal strength (Q1 = lowest, Q5 = highest); bars show weak model accuracy within each bin. On MMLU, CMP produces a 74pp spread versus 23pp for G-Ent and 18pp for G-PPL. On TriviaQA (no context), all three signals are competitive (93pp, 80pp, 81pp). On GSM8K, CMP achieves a 50pp spread while G-PPL nearly collapses to 8pp, confirming that generator self-perplexity is uninformative on chain-of-thought tasks and that cross-model disagreement is doing genuine work.
Mean AUROC ↑
Method
MMLU
TriviaQA
TriviaQA
GSM8K
Verifier work
Cost
(ctx)
(no ctx)
vs. CMP
Single verifier prefill
P(True)
0.518
0.510
0.526
0.603
1 prefill
∼ 1 ×
CME (ours)
0.628
0.759
0.737
0.621
1 prefill
1 ×
CMP (ours)
0.749
0.679
0.761
0.623
1 prefill
1 ×
Table 2: Mean AUROC across model pairs with capability gap > 5pp, comparing CMP and CME against label-free baselines. Methods are grouped by computational cost tier; cost is per-sample overhead relative to a single verifier prefill, excluding shared generator inference. T = answer tokens; k=10 for Semantic Entropy. Bold = best per column within tier; underline = best overall per column. Full per-pair results in Appendix 7 .
Figure 4: CMP AUROC versus capability gap across three benchmarks. On MMLU, AUROC is uncorrelated with gap ( ρ=+0.12 , p=0.71 ) and same-size cross-family pairs (blue) achieve the highest scores, suggesting model diversity drives the signal rather than capability asymmetry. On TriviaQA, gap correlates positively with AUROC ( ρ=+0.63 , p=0.03 ), indicating a stronger verifier helps when errors are knowledge-driven. GSM8K shows no significant trend ( ρ=−0.22 , p=0.43 ).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
AUROC ↑
APGR ↑
Generator
Verifier
Acc g
Acc v
Gap
G-Ent
G-PPL
CMP
CME
G-Ent
G-PPL
CMP
CME
Same-size cross-family
Mistral-7B
Qwen-7B
0.56
0.72
0.16
0.720
0.726
0.711
0.632
0.608
0.662
0.818
0.605
OLMo-7B
Qwen-7B
0.56
0.72
0.15
0.503
0.599
0.896
0.697
0.486
0.546
0.914
0.447
Llama-3-8B
Qwen-7B
0.62
0.72
0.10
0.652
0.684
0.694
0.635
0.617
0.652
0.910
0.603
Mistral-7B
Llama-3-8B
0.56
0.62
0.06
0.720
0.726
0.753
0.716
0.715
0.853
0.941
0.629
Appendix
Table 3: Full results on MMLU. Pairs grouped by type and sorted by capability gap (descending). † marks rows with gap≤0.05 , which produce unstable APGR estimates due to the small PGR denominator and are excluded from the filtered mean. Bold = best per row among G-Ent, G-PPL, CMP, CME.
AUROC ↑
APGR ↑
Generator
Verifier
Acc g
Acc v
Gap
G-Ent
G-PPL
CMP
CME
G-Ent
G-PPL
CMP
CME
Same-size cross-family
Llama-3-8B
Qwen-7B
0.43
0.65
0.21
0.859
0.885
0.772
0.875
0.796
0.808
0.807
0.757
Llama-3-8B
Mistral-7B
0.43
0.64
0.20
0.859
0.885
0.825
0.922
0.822
0.831
0.849
0.789
OLMo-7B
Qwen-7B
0.51
0.65
0.13
0.636
0.429
0.466
0.843
0.621
0.431
0.488
0.723
OLMo-7B
Mistral-7B
0.51
0.64
0.12
0.636
0.429
0.588
0.884
0.659
0.406
0.626
0.724
Appendix
Table 4: Full results on TriviaQA (with context). Pairs grouped by type and sorted by capability gap (descending). † marks rows with gap≤0.05 . Bold = best per row among G-Ent, G-PPL, CMP, CME.
AUROC ↑
APGR ↑
Generator
Verifier
Acc g
Acc v
Gap
G-Ent
G-PPL
CMP
CME
G-Ent
G-PPL
CMP
CME
Same-size cross-family
OLMo-7B
Llama-3-8B
0.39
0.67
0.29
0.774
0.504
0.706
0.807
0.664
0.484
0.628
0.601
OLMo-7B
Mistral-7B
0.39
0.66
0.28
0.774
0.504
0.812
0.883
0.659
0.491
0.729
0.634
OLMo-7B
Qwen-7B
0.39
0.59
0.21
0.774
0.504
0.673
0.849
0.635
0.480
0.616
0.596
Qwen-7B
Llama-3-8B
0.59
0.67
0.08
0.823
0.793
0.748
0.625
0.980
0.946
0.964
0.533
Appendix
Table 5: Full results on TriviaQA (no context). Pairs grouped by type and sorted by capability gap (descending). † marks rows with gap≤0.05 . Bold = best per row among G-Ent, G-PPL, CMP, CME.
AUROC ↑
APGR ↑
Generator
Verifier
Acc g
Acc v
Gap
G-Ent
G-PPL
CMP
CME
G-Ent
G-PPL
CMP
CME
Same-size cross-family
Mistral-7B
Qwen-7B
0.54
0.90
0.36
0.658
0.542
0.662
0.718
0.583
0.522
0.606
0.620
Mistral-7B
OLMo-7B
0.54
0.84
0.30
0.658
0.542
0.722
0.684
0.581
0.523
0.634
0.592
Llama-3-8B
Qwen-7B
0.61
0.90
0.29
0.593
0.577
0.609
0.508
0.549
0.534
0.594
0.492
Llama-3-8B
OLMo-7B
0.61
0.84
0.22
0.593
0.577
0.633
0.527
0.530
0.512
0.592
0.469
Appendix
Table 6: Full results on GSM8K. Pairs grouped by type and sorted by capability gap (descending). No row has gap≤0.05 , so the filtered mean coincides with the full mean. Bold = best per row among G-Ent, G-PPL, CMP, CME.
Dataset
Generator
Verifier
Gap
CMP
CME
P(True)
V-Agree
Sem-Ent (k=10)
MMLU
Gemma-270M
Qwen-7B
+47.5
0.828
0.609
0.572
0.820
0.515
Gemma-270M
Llama-3-8B
+37.4
0.710
0.597
0.503
0.739
0.515
Qwen-0.5B
Qwen-7B
+29.9
0.841
0.583
0.551
0.843
0.633
Llama-1B
Qwen-7B
+28.8
0.786
0.634
0.547
0.821
0.617
Qwen-0.5B
Llama-3-8B
+19.9
0.749
0.639
0.498
0.784
0.633
Llama-1B
Llama-3-8B
+18.8
0.817
0.679
0.475
0.748
0.617
Appendix
Table 7: AUROC by correctness signal across (generator, verifier, dataset) tuples with gap > 5pp. CMP and CME are this paper’s cross-model perplexity / entropy signals (single prefill on the verifier). P(True), Verifier Agreement, and Semantic Entropy Kuhn et al. (2023) are external baselines. Bold = best AUROC per row.
Figure 5: Coverage–accuracy curves (single pair). At each coverage level we abstain on instances whose signal score exceeds a swept threshold and report generator accuracy on the retained subset. CMP (blue, solid) maintains higher accuracy than G-Ent (red, dashed) across nearly all operating points on both MMLU (Qwen-0.5B → Qwen-7B) and GSM8K (Mistral-7B → Llama-3-8B). The dotted line marks full-set accuracy.
Figure 6: Coverage–accuracy curves (averaged over all pairs). Curves are averaged across all 15 model pairs per benchmark; shaded bands show ± 1 SE. CMP consistently achieves higher accuracy than G-Ent at equal coverage on both MMLU and GSM8K.
Figure 7: CMP (filled circles) and within-model entropy (open squares) AUROC versus capability gap. The positive gap effect on TriviaQA is specific to CMP — entropy AUROC is uncorrelated with gap on all three datasets.
Figure 8: CMP advantage over within-model entropy ( Δ AUROC = CMP − G-Ent) versus capability gap. CMP ’s advantage grows significantly with gap on TriviaQA ( ρ=+0.81 , p<0.01 ), while on MMLU CMP leads regardless of gap size. GSM8K shows a positive but non-significant trend ( ρ=+0.39 , p=0.16 ).
AUROC ↑
Baselines
Generator
Verifier
Acc g
Gap
CMP-Full
CMP-Final
P(True)
G-Ent
G-PPL
Qwen-0.5B
Qwen-7B
0.25
0.64
0.629
0.456
0.495
0.616
0.621
Llama-1B
Qwen-7B
0.38
0.51
0.668
0.755
0.463
0.543
0.523
Mistral-7B
Qwen-7B
0.51
0.39
0.668
0.574
0.487
0.655
0.539
Qwen-0.5B
Llama-3-8B
0.25
0.36
0.627
0.525
0.431
0.616
0.621
Llama-3-8B
Qwen-7B
0.61
0.28
0.590
0.584
0.476
0.596
0.577
Appendix
Table 8: GSM8K correctness prediction (AUROC): Full chain-of-thought CMP vs. final-answer-only CMP vs. P(True) verification prompting. CMP-Full and CMP-Final perform comparably, while P(True) is near random. Bold = best among CMP-Full, CMP-Final, P(True).
APGR ↑
Baselines
Generator
Verifier
Acc g
Gap
CMP-Full
CMP-Final
P(True)
G-Ent
G-PPL
Qwen-0.5B
Qwen-7B
0.25
0.64
0.528
0.484
0.499
0.520
0.521
Llama-1B
Qwen-7B
0.38
0.51
0.574
0.621
0.481
0.511
0.503
Mistral-7B
Qwen-7B
0.51
0.39
0.590
0.524
0.506
0.575
0.513
Qwen-0.5B
Llama-3-8B
0.25
0.36
0.530
0.509
0.492
0.544
0.545
Llama-3-8B
Qwen-7B
0.61
0.28
0.614
0.579
0.488
0.551
0.538
Appendix
Table 9: GSM8K routing performance (APGR): Full chain-of-thought CMP vs. final-answer-only CMP vs. P(True). Pairs with gap < 5pp excluded. Bold = best among CMP-Full, CMP-Final, P(True).
Multimodal large language models often generate reasoning chains containing subtle errors that lead to incorrect answers. Current verification approaches have notable limitations. Existing approaches either require expensive labelled supervision with inconsistent cross-task performance or aggregate scores from multiple sources by simple aggregations, missing a key insight: when these scores disagree, that disagreement itself carries important information about whether a reasoning step is truly valid or not. We formalise this as a coupled scoring problem among disparate, frozen verifiers, interpretable as a coordination game with a unique closed-form equilibrium where agreement signals valid steps while disagreement reveals instability. Towards this end, we propose a training-free domain-agnostic step-wise verification approach we call VERDICT: VERification via Disagreement-Informed Coupled Thresholding. To our knowledge, VERDICT is the first training-free verifier that makes the structure of cross-modal disagreement explicit and actionable. It computes consensus scores through a closed-form solution, enabling both disagreement-aware filtering and stability-conscious ranking of reasoning steps. Evaluated across six benchmarks, \method consistently improves over the base model by up to +5.95%, and performs competitively with domain-specific critics that demand extensive supervision, demonstrating that cross-modal agreement provides robust verification signals without task-specific adaptation and Training-Free Verification
Rohit Sinha, Kunal Tilaganji, Tanuja Ganu +3
Indian Institute of Technology, Hyderabad · Microsoft Research
Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction. The result is a dissociation with direct implications for deployment monitoring. Across multi-hop arithmetic chains, probes that detect corruption turn out to be uninformative about final answer correctness; models forced into structured confidence formats collapse to two values with indistinguishable error rates; and probe persistence across hops fails to separate correct from incorrect outcomes, refuting our pre-registered "persistence beats peak" hypothesis. This pattern of knowing but not saying generalises across model families including reasoning models. As a real-time monitor, probe-based interventions are sharply model and error-type dependent: branch-and-pick is net-positive across models and uniquely non-breaking on Llama-3.1-8B (4 rescued, 0 broken), while reprompt and replace-prior break correct traces at roughly the rate they rescue wrong ones. Probe-based monitoring is a necessary complement to verbalised confidence, but no single intervention dominates, and the deployable answer is model-aware, error-type-aware routing.
Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution to more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed as the gap between inter-model and intra-model sequence-semantic similarity. We then define total uncertainty (TU) as the sum of AU and EU. In a comprehensive study across five 7-9B instruction-tuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.