No Universal Signal Predicts Sample-Level LLM Regression under Version Updates

Jia Sheng; Yiwei Lu

cs.AIAug 11, 2026

No Universal Signal Predicts Sample-Level LLM Regression under Version Updates

Authors: Jia Sheng, Yiwei Lu

Organizations: University of Ottawa · Vector Institute

Abstract

Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate. But aggregate gains say little about individual samples: an update can still cause sample-level regression, where a response correct under the old model becomes incorrect under the new one. This paper studies how to predict such regressions from signals available at inference time. We compare single-model signals (confidence, logit margin, attention entropy) against cross-version signals (output KL divergence, likelihood drift, token-level KL, representation drift) under a unified added-value test that isolates each signal's gain over a confidence baseline. Across six benchmarks in three task families (multiple-choice question answering, or MCQ; math reasoning; code generation) and six model update pairs, we find that (1) signal effectiveness is task-dependent: confidence is strongest on MCQ and simpler math, while likelihood/KL signals give the most frequent gains on harder math and code; (2) no signal is universally best across model updates either; and (3) some cross-version signals stay informative even when confidence fails, including without labels, which supports a proof-of-concept selective fallback that routes high-risk samples back to the old model. Practitioners can use these task-level patterns to choose which regression signal to trust for a given update. Code is available at https://github.com/jiashengsally/llm-regression-signals.

Figures & tables

Figure 1: Representative negative flips for Qwen2.5

\rightarrow

3. Three retrospectively selected examples, one per task family.

Figure 2: Method overview. Both versions produce one greedy response per sample. The negative-flip label supervises predictor fitting and evaluation, and the signals enter a baseline-versus-augmented 5-fold cross-validation test reporting

\Delta_{\mathrm{CV\text{-}AUROC}}

Figure 3: Signal added value across tasks. The uncolored first column reports the absolute CV-AUROC of new-model confidence. The colored cells report $\Delta_{\mathrm{CV\text{-}AUROC}}$ after adding each signal; red indicates a gain, blue a decrease, and “n/a” an unavailable signal. MCQ shows old-model margin, attention entropy, and representation drift; generation shows new-trajectory likelihood drift, top- $k$ token KL, and representation drift. Appendix Tables A4 – A6 report all variants and exact values.

Figure 4: Are negative flips hard examples? AUROC using low old-model confidence as flip risk on the $M_{\text{old}}$ -correct subset ( $0.5$ is random ranking). Red indicates that lower confidence ranks flips above kept examples; blue indicates a reversed relationship. The final row is the Llama-2 $\rightarrow$ 3 boundary case.

Figure 5: Selective fallback trade-offs at a fixed 10% budget. Green shows negative flips repaired, red shows positive flips lost, and diamonds show the net accuracy change.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hugging Face checkpoint	Params	Release	Hidden	Vocab
Qwen2.5 $\rightarrow$ Qwen3 (cross-gen, Acc $+4$ pp)
Qwen/Qwen2.5-7B-Instruct	7B	2024.09	$3584$	$152{,}064$
Qwen/Qwen3-8B	8B	2025.04	$4096$	$152{,}064$
Llama-3 $\rightarrow$ Llama-3.1 (minor, Acc $+6$ pp)
meta-llama/Meta-Llama-3-8B-Instruct	8B	2024.04	$4096$	$128{,}256$
meta-llama/Llama-3.1-8B-Instruct	8B	2024.07	$4096$	$128{,}256$

Appendix

Table A1: Model-update pairs. Each sub-heading names a pair (old

\rightarrow

new same-family version) with its update type and average accuracy gain

\Delta_{\text{Acc}}=\mathrm{Acc}_{\text{new}}-\mathrm{Acc}_{\text{old}}

across the three task families; the two rows below give its checkpoints by full Hugging Face repository ( huggingface.co/<name> ). Hidden size and vocabulary are taken from the model configurations.

Signal	Category	Mode	Definition
Confidence ( $\mathrm{conf}_{\text{old}}$ , $\mathrm{conf}_{\text{new}}$ )	single-model	MCQ + gen	max softmax (MCQ) / mean token log-prob (gen)
Logit margin	single-model	MCQ	top-1 $-$ top-2 logits on option tokens
Attention entropy	single-model	MCQ	mean per-head Rényi-2 entropy at the final prompt position
Output KL/JSD (forward / reverse / JSD)	cross-version	MCQ	$\mathrm{KL}/\mathrm{JSD}$ on option distributions
Likelihood drift (old / new traj.)	cross-version	gen	per-token NLL difference, computed on the old or new model’s generated trajectory
Top- $k$ token KL (old / new traj.)	cross-version	gen	approximate token-distribution KL on the old or new trajectory

Appendix

Table A2: Signals evaluated.

MMLU-Pro ( $N{=}3{,}000$ per pair)
Signal	Qwen2.5 $\rightarrow$ 3	Llama-3 $\rightarrow$ 3.1	Mistral v0.2 $\rightarrow$ v0.3	Qwen2 $\rightarrow$ 2.5	Gemma 1.1 $\rightarrow$ 2
Accuracy gain (pp)	$+2.1$	$-0.6$	$+1.8$	$+8.2$	$+14.2$
$\mathrm{conf}_{\text{new}}$ (baseline CV-AUROC)	$0.85$	$0.83$	$0.82$	$0.83$	$0.83$
$\mathrm{conf}_{\text{old}}$	$+0.013$	$+0.036$	$+0.016$	$-0.014$	$+0.001$
Logit margin (old)	$-0.007$	$+0.023$	$-0.002$	$-0.025$	$-0.048$
Logit margin (new)	$+0.001$	$-0.010$	$-0.003$	$-0.003$	$-0.002$

Appendix

Table A4: MCQ-phase results. The baseline row reports CV-AUROC using

\mathrm{conf}_{\text{new}}

alone; all signal rows report the change in CV-AUROC (

\Delta

) after adding that signal to the baseline. The accuracy gain row is in percentage points (pp),

\Delta_{\text{Acc}}=\mathrm{Acc}_{\text{new}}-\mathrm{Acc}_{\text{old}}

, and this convention is used in all signal tables. Output KL and JSD are reported separately as direct measures of output-distribution change. Logit margin and attention entropy are reported for both model versions; the main comparison uses the old-model variants, while the new-model variants largely overlap with the

\mathrm{conf}_{\text{new}}

baseline. NFR is the fraction of the

M_{\text{old}}

-correct subset that becomes incorrect under

M_{\text{new}}

GSM8K ( $N{=}1{,}319$ per pair)
Signal	Qwen2.5 $\rightarrow$ 3	Llama-3 $\rightarrow$ 3.1	Mistral v0.2 $\rightarrow$ v0.3	Qwen2 $\rightarrow$ 2.5	Gemma 1.1 $\rightarrow$ 2
Accuracy gain (pp)	$+3.1$	$+4.8$	$+12.7$	$+9.2$	$+35.6$
$\mathrm{conf}_{\text{new}}$ (baseline CV-AUROC)	$0.76$	$0.65$	$0.69$	$0.78$	$0.62$
$\mathrm{conf}_{\text{old}}$	$-0.000$	$+0.008$	$-0.004$	$-0.009$	$-0.024$
Likelihood drift (new traj.)	$-.005$	$-.007$	$+.037$	$+.003$	$-.065$
Top- $k$ token KL (new traj.)	$-.001$	$-.013$	$+.038$	$-.002$	$-.071$

Appendix

Table A5: Math-phase results. The baseline row reports CV-AUROC using

\mathrm{conf}_{\text{new}}

alone; all signal rows report the change in CV-AUROC (

\Delta

) after adding that signal to the baseline. Each cross-version signal comes in an old-trajectory and a new-trajectory variant; Figure 3 shows the new-trajectory variants used in the main comparison. NFR is the fraction of the

M_{\text{old}}

-correct subset that becomes incorrect under

M_{\text{new}}

HumanEval ( $N{=}164$ per pair)
Signal	Qwen2.5 $\rightarrow$ 3	Llama-3 $\rightarrow$ 3.1	Mistral v0.2 $\rightarrow$ v0.3	Qwen2 $\rightarrow$ 2.5	Gemma 1.1 $\rightarrow$ 2
Accuracy gain (pp)	$+1.2$	$+6.1$	$+13.0$	$+6.1$	$+18.3$
$\mathrm{conf}_{\text{new}}$ (baseline CV-AUROC)	$0.80$	$0.76$	$0.70$	$0.75$	$0.42$
$\mathrm{conf}_{\text{old}}$	$-.003$	$-.009$	$-.160$	$-.009$	$-.060$
Likelihood drift (new traj.)	$-.012$	$+.035$	$-.112$	$+.042$	$+.215$
Top- $k$ token KL (new traj.)	$-.018$	$+.023$	$-.079$	$+.045$	$+.232$

Appendix

Table A6: Code-phase results. The baseline row reports CV-AUROC using

\mathrm{conf}_{\text{new}}

alone; all signal rows report the change in CV-AUROC (

\Delta

M_{\text{old}}

-correct subset that becomes incorrect under

M_{\text{new}}

MMLU-Pro
Old-model signal(s)	Qwen2.5 $\rightarrow$ 3	Llama-3 $\rightarrow$ 3.1	Mistral v0.2 $\rightarrow$ v0.3	Qwen2 $\rightarrow$ 2.5	Gemma 1.1 $\rightarrow$ 2
$\mathrm{conf}_{\text{old}}$	$0.810$	$0.840$	$0.755$	$0.765$	$0.737$
Logit margin	$0.809$	$0.828$	$0.765$	$0.760$	$0.735$
Attention entropy	$0.650$	$0.581$	$0.626$	$0.589$	$0.636$
$\mathrm{conf}_{\text{old}}$ + margin	$0.810$	$0.837$	$0.766$	$0.760$	$0.739$
$\mathrm{conf}_{\text{old}}$ + entropy	$0.756$	$0.837$	$0.683$	$0.752$	$0.702$

Appendix

Table A7: Old-model signals for identifying hard MCQ examples. Values are absolute 5-fold CV-AUROC on the

M_{\text{old}}

-correct subset. All rows use logistic regression fitted within each training fold; they therefore differ from the unfitted standalone AUROC in Figure 4 .

MCQ: MMLU-Pro
Bench	Pair	Best single signal	Best $\Delta$	All-cross $\Delta$	Gain
MMLU-Pro	Qwen2.5 $\rightarrow$ 3	Output KL (JSD)	$+0.146$	$+0.146$	$+0.000$
MMLU-Pro	Llama-3 $\rightarrow$ 3.1	Output KL (JSD)	$+0.126$	$+0.131$	$+0.005$
MMLU-Pro	Mistral v0.2 $\rightarrow$ v0.3	Output KL (JSD)	$+0.177$	$+0.177$	$-0.001$
MMLU-Pro	Qwen2 $\rightarrow$ 2.5	Output KL (JSD)	$+0.161$	$+0.162$	$+0.001$
MMLU-Pro	Gemma 1.1 $\rightarrow$ 2	Output KL (JSD)	$+0.163$	$+0.163$	$-0.000$

Appendix

Table A8: Signal complementarity. Added-value CV-AUROC (

\Delta

) of the best single cross-version signal and the combined cross-version model over

\mathrm{conf}_{\text{new}}

. The out-of-scope Llama-2

\rightarrow

3 rows use unrelated tokenizers, so their token-level cross-version values (likelihood drift, top-

k

token KL) are not directly comparable and are shown only for completeness.

Pair	Task	Cross-version signal	Flips caught	$\Delta_{\text{Acc}}$ (pp)	$\mathrm{conf}_{\text{new}}$	random
Qwen2 $\rightarrow$ 2.5	GSM8K	Top- $k$ token KL (new traj.)	$26.7\%$	$-0.3$	$\mathbf{40.0\%}$	$10.0\%$
Llama-3 $\rightarrow$ 3.1	MBPP	Likelihood drift (new traj.)	$30.8\%$	$+0.0$	$15.4\%$	$10.0\%$
Gemma 1.1 $\rightarrow$ 2	HumanEval	Top- $k$ token KL (old traj.)	$\mathbf{29.4\%}$	$+1.8$	$0.0\%$	$9.8\%$

Appendix

Table A9: Selective fallback with a

10\%

budget. Flips caught is the percentage of repaired negative flips, and

\Delta_{\text{Acc}}

is the accuracy change in percentage points relative to always serving

M_{\text{new}}

. The

\mathrm{conf}_{\text{new}}

column uses the same budget, and the random column reports the expected capture from uniformly routing the same rounded number of inputs. Confidence is the better trigger on Qwen2

\rightarrow

2.5 GSM8K, while the cross-version signal only clearly outperforms confidence on Gemma 1.1

\rightarrow

2 HumanEval, where confidence is unreliable (AUROC

0.42

Figure A1: Selective fallback over the full budget range. The top row reports negative flips repaired, and the bottom row reports net accuracy change. The vertical line marks the fixed

10\%

budget used in the main analysis.

Pair	Task	Signal	$M_{\text{old}}$ -correct	Full
Qwen2.5 $\rightarrow$ 3	MMLU-Pro	$\mathrm{conf}_{\text{old}}$	$0.81\,[0.78,0.83]$	$0.59\,[0.56,0.62]$
Llama-3 $\rightarrow$ 3.1	MMLU-Pro	$\mathrm{conf}_{\text{old}}$	$0.84\,[0.81,0.86]$	$0.57\,[0.54,0.60]$
Mistral v0.2 $\rightarrow$ v0.3	MMLU-Pro	$\mathrm{conf}_{\text{old}}$	$0.75\,[0.72,0.79]$	$0.55\,[0.52,0.59]$
Qwen2 $\rightarrow$ 2.5	MMLU-Pro	$\mathrm{conf}_{\text{old}}$	$0.76\,[0.73,0.80]$	$0.53\,[0.50,0.56]$
Gemma 1.1 $\rightarrow$ 2	MMLU-Pro	$\mathrm{conf}_{\text{old}}$	$0.74\,[0.70,0.77]$	$0.54\,[0.50,0.58]$
Qwen2.5 $\rightarrow$ 3	MMLU-Pro	$\mathrm{conf}_{\text{new}}$	$0.85\,[0.83,0.87]$	$0.63\,[0.61,0.66]$

Appendix

Table A10: Label-free per-cell AUROC. Standalone AUROC of all formally evaluated signals and their main old/new model or trajectory variants on the

M_{\text{old}}

-correct subset and the full evaluation set, with bootstrap

95\%

confidence intervals. Entries marked — are unavailable because representation drift requires equal hidden size, which holds only for three update pairs: Llama-3

\rightarrow

3.1, Mistral v0.2

\rightarrow

v0.3, and Qwen2

\rightarrow

2.5.

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 91 papers

Identifying Influential N-grams in Confidence Calibration via Regression Analysis
Apr 7, 2026Shintaro Ozaki, Wataru Hashimoto, Hidetaka Kamigaito +2Language Model Calibration Confidence Estimation in Language Models
Nara Institute of Science and Technology (NAIST) · The University of Tokyo
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
Mar 17, 2026Yasi Zhang, Tianyu Chen, Mingyuan Zhou +3RL for Language Models Reinforcement Learning
University of California, Los Angeles · The University of Texas at Austin · Google Research
Eliciting Numerical Predictive Distributions of LLMs Without Autoregression
Mar 3, 2026Julianna Piskorz, Katarzyna Kobalczyk, Mihaela van der SchaarLLM Uncertainty Estimation LLM Inference
Department of Applied Mathematics and Theoretical Physics, University of Cambridge

Apr 7, 2026J/K move · Enter open · S save

Identifying Influential N-grams in Confidence Calibration via Regression Analysis

Shintaro Ozaki, Wataru Hashimoto, Hidetaka Kamigaito, Katsuhiko Hayashi+1

Nara Institute of Science and Technology (NAIST) · The University of Tokyo

Language Model Calibration Confidence Estimation in Language Models LLM Reliability Confidence Estimation Causal Interventions in Language Models

Open paper AskPDF arXiv

While large language models (LLMs) improve performance by explicit reasoning, their responses are often overconfident, even though they include linguistic expressions demonstrating uncertainty. In this work, we identify what linguistic expressions are related to confidence by applying the regression method. Specifically, we predict confidence of those linguistic expressions in the reasoning parts of LLMs as the dependent variables and analyze the relationship between a specific

n

-gram and confidence. Across multiple models and QA benchmarks, we show that LLMs remain overconfident when reasoning is involved and attribute this behavior to specific linguistic information. Interestingly, several of the extracted expressions coincide with cue phrases intentionally inserted on test-time scaling to improve reasoning performance. Through our test on causality and verification that the extracted linguistic information truly affects confidence, we reveal that confidence calibration is possible by simply suppressing those overconfident expressions without drops in performance.

Abstract

Language Model Error Detection

Explore similar work

Identifying Influential N-grams in Confidence Calibration via Regression Analysis

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

Eliciting Numerical Predictive Distributions of LLMs Without Autoregression