SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

Muhammad Atif Saeed; Azka Atiq; Adarsh Kumar; Akhtar Jamil; Uzair Iqbal; Roland Aydin

cs.LGOct 25, 2026

SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

Authors: Muhammad Atif Saeed, Azka Atiq, Adarsh Kumar, Akhtar Jamil, Uzair Iqbal, Roland Aydin

Organizations: Department of AI & Data Science, National University of Computer & Emerging Sciences, Islamabad, Pakistan · Data Engineer, University of Arkansas at Little Rock, Arkansas, USA · Department of Computer Science, National University of Computer & Emerging Sciences, Islamabad, Pakistan · Department of Software Engineering, Faculty of Computer Science & Information Technology, University of Malaya, Malaysia ·

Abstract

Large language models (LLMs) can act as decision models that return a probability distribution over a fixed set of options, but deployed decision models face domain, label-prior, emerging-class and candidate-set shift. We study how such models fail under shift and propose SHIFT-Jev, a monitoring, abstention and continual-adaptation procedure for them. It scores the risk of each decision from uncertainty, format and option-consistency signals, certifies abstention thresholds with Learn-then-Test, monitors the decision stream with drift and label-shift detectors, and accepts a model update only when an expected-gain gate finds a significant held-out improvement without loss on earlier regimes. On a benchmark of ten shifted test environments with Qwen3-0.6B and Qwen3-4B (three seeds each) and Phi-4-mini, fine-tuning with KL anchors and temperature scaling lowers calibration error under shift from 0.289 to 0.087. At a target risk of 10%, a learned uncertainty score with a certified threshold exceeds the target in 3 to 5 of 27 environment-seed tests, against 5 to 15 for a max-probability score with the same certification and 9 to 14 for conformal prediction sets; drift-based routing changes little. A chi-squared test on predicted labels detects 95 to 99% of class-prior shifts that confidence- and embedding-based detectors miss. In continual adaptation, SHIFT-Jev reaches 70.5% accuracy on Qwen3-0.6B, above replay, LwF, EWC and fine-tuning baselines in every seed, whereas on Qwen3-4B its conservative gate leaves it 0.4 to 1.4 points behind them.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Shift-Jev . Stage 1 : the LLM-as-Jev decision model scores numbered options through a prefix-free trie and exposes decision probabilities, the probability mass on legal identifier tokens and a state embedding. Stage 2 : in the scoring lane, uncertainty, format and consistency features give the risk score $r(x)$ ; in the routing lane, an option-invariant drift distance assigns the input to a known regime $g$ or to unknown and drives the stream detectors. The model answers only if $r(x)$ is below the threshold certified for that regime, or below the conservative $\tau_{\mathrm{shift}}$ for unknown inputs and after an alarm. Stage 3 : a labelled batch from a new regime drives a severity-scaled LoRA update that is kept only if it significantly improves held-out accuracy without losing more than 5 points on earlier regimes.

Work	Decision-model interface	Shift benchmark	Risk guarantee	Per-regime guarantee	Stream detection	Continual adaptation
LLM-as-Jev [ 7 ]	✓	—	—	—	—	—
JEV-as-a-Judge [ 2 ]	✓	$\circ$	$\circ$	—	—	—
UniCR [ 31 ]	—	—	✓	—	—	—
HG-CRC [ 11 ]	—	$\circ$	✓	✓	—	—
DS-CP [ 10 ]	—	✓	$\circ$	—	—	—
Continual calibration [ 13 ]	—	—	$\circ$	✓	—	✓

Table 1: Positioning. ✓: addressed;

\circ

: partially; —: not addressed.

Group	Environment	Source	Role	$n$ (test)
ID	AG News, SST-2, BoolQ, ARC-C	4 tasks	Tr / Val / Cal / Test	1,200
Proxy shift	SciQ, Emotion	2 tasks	Cal only	—
A. Domain	Banking77 ( $K=77$ ), Twitter-Fin ( $K=20$ ), MMLU ( $K=4$ )	3 tasks	Test (+ adaptation)	$3\times 500$
B. Class prior	AG News, $\rho=10$ and $\rho=100$	resampled	Test	$2\times 300$
C. Emerging classes	CLINC150 + “none of the above” ( $K=16$ )	50% out of scope	Test (+ adaptation)	500
D. Candidate set	rotation, paraphrase, distractor / gold removed	ID test items	Test	153–204 each

Table 2: Evaluation environments and the use of their data: training (Tr), temperature fitting (Val), risk-model fitting and certification (Cal) and testing (Test).

System	BoolQ	MMLU	MMLU-Pro	ARC-C	WinoGr.	SciQ	Macro-6	Banking77	ECE
Base paper, training-free	72.60	46.75	22.25	54.17	53.60	97.75	57.85	22.21	–
This study, training-free (B1)	73.80	45.17	24.00	54.00	51.40	94.25	57.10	20.48	0.246
Base paper, LoRA $\lambda$ =1	72.40	46.00	20.80	57.80	55.00	97.80	58.30	54.90	–
Base paper, LoRA $\lambda$ =0.01	72.80	47.00	21.50	57.20	55.20	97.50	58.53	59.00	–
Base paper, full FT $\lambda$ =1	74.40	47.83	20.50	60.33	53.60	97.75	59.07	60.49	–
This study, LoRA $\lambda$ =1 (B2, 3 seeds)	77.87	46.19	23.71	61.83	53.00	95.08	59.61 $\pm$ 0.23	20.41	0.120

Table 3: Reproduction of LLM-as-Jev with Qwen3-0.6B on the base paper’s external benchmarks and sample sizes (accuracy, %). ECE averages the six macro tasks.

Figure 2: Per-task reproduction of the base paper on Qwen3-0.6B. Hollow markers: reported; filled: this study (b: mean and range over 3 seeds). The Banking77 gap in (b) reflects intent data in the base paper’s training mixture, which our ID mixture excludes by design.

			Qwen3-0.6B				Qwen3-4B-Instruct
Environment	Shift	$n$	Acc B1	ECE B1	Acc B3	ECE B3	Acc B1	ECE B1	Acc B3	ECE B3
In-distribution	—	1200	73.3	0.178	78.5 $\pm$ 1.1	0.018 $\pm$ 0.004	91.1	0.084	91.4 $\pm$ 0.3	0.038 $\pm$ 0.003
Banking77	Domain	500	18.4	0.472	19.2 $\pm$ 0.9	0.108 $\pm$ 0.011	55.4	0.400	56.3 $\pm$ 0.2	0.128 $\pm$ 0.034
Twitter-Fin	Domain	500	27.6	0.582	38.2 $\pm$ 0.7	0.114 $\pm$ 0.012	60.2	0.375	59.5 $\pm$ 0.8	0.167 $\pm$ 0.009
MMLU	Domain	500	43.4	0.346	46.5 $\pm$ 1.1	0.095 $\pm$ 0.017	71.0	0.260	71.3 $\pm$ 0.3	0.077 $\pm$ 0.008
AG prior $\rho$ =10	Class prior	300	81.7	0.136	90.1 $\pm$ 1.0	0.044 $\pm$ 0.015	92.3	0.078	91.3 $\pm$ 0.9	0.044 $\pm$ 0.006

Table 4: RQ1. Accuracy (%) and ECE of the frozen model (B1, raw) and the fine-tuned, temperature-scaled model (B3; mean

\pm

s.d. over 3 seeds) per environment.

Figure 3: Expected calibration error per environment for the frozen model (B1) and the fine-tuned, temperature-scaled model (B3; mean and s.d. over 3 seeds): (a) Qwen3-0.6B, (b) Qwen3-4B-Instruct.

Backbone	System	Seeds	ID acc (%)	ID ECE	Shift acc (%)	Shift ECE	[email protected] (%)	[email protected] (%)	[email protected]
Qwen3-0.6B	B1 frozen	1	73.3	0.178	57.5	0.289	–	–	–
	B2 fine-tuned + KL anchors	3	78.5 $\pm$ 1.1	0.061 $\pm$ 0.004	64.2 $\pm$ 0.6	0.126 $\pm$ 0.007	–	–	–
	B3 = B2 + temperature scaling	3	78.5 $\pm$ 1.1	0.018 $\pm$ 0.004	64.2 $\pm$ 0.6	0.087 $\pm$ 0.003	–	–	–
	B4 = B3 + uncertainty abstention	3	as B3				37.9 $\pm$ 4.9	6.4 $\pm$ 1.9	5/27
	B5 = B3 + Shift-Jev monitor	3	as B3				33.3 $\pm$ 6.6	5.2 $\pm$ 2.8	4/27
	B6 = B5 after continual adaptation	3 / 1	77.6 $\pm$ 0.9	–	see Tables 12 and 14		64.5 †	11.6 †	–

Table 5: Systems B1–B6. Shift metrics average the nine shifted environments; abstention at

\alpha=0.10

\delta=0.1

. The calibration gains of B2 and B3 come from fine-tuning with KL anchors and temperature scaling; the reliability layer (B4, B5) does not change predictions or calibration. B6 is evaluated on the continual evaluation sets: ID accuracy after the last stage (3 seeds) and, marked † , coverage and realized risk on the ID regime with its recertified threshold (seed 1; Table 15 ).

Risk score / threshold rule	Shift AURC $\downarrow$	[email protected] (%)	[email protected] (%)	[email protected] (%)	Violations $\alpha$ =.05/.10/.20	[email protected] per seed	Gold removed: abstain (%)
MSP score, pooled LTT $\tau$	0.2287 $\pm$ 0.0034	28.8 $\pm$ 12.5	43.4 $\pm$ 2.3	16.0 $\pm$ 1.2	12 / 15 / 18	5 / 5 / 5	35.5 $\pm$ 1.5
Energy score, pooled LTT $\tau$	0.4852 $\pm$ 0.0139	0.0	0.0	0.0	0 / 0 / 0	0 / 0 / 0	100.0
Split-conformal LAC (not risk-certified)	0.2287 $\pm$ 0.0034	36.3 $\pm$ 2.0	53.6 $\pm$ 0.9	24.5 $\pm$ 0.4	17 / 14 / 15	5 / 5 / 4	35.6 $\pm$ 0.6
Shift-Jev risk score, pooled LTT $\tau$	0.2265 $\pm$ 0.0036	18.0 $\pm$ 16.7	39.1 $\pm$ 0.9	7.5 $\pm$ 1.0	4 / 8 / 10	2 / 3 / 3	35.8 $\pm$ 3.0
B4: uncertainty risk, pooled LTT $\tau$	0.2280 $\pm$ 0.0027	20.2 $\pm$ 17.6	37.9 $\pm$ 4.9	6.4 $\pm$ 1.9	4 / 5 / 13	2 / 2 / 1	35.8 $\pm$ 2.9
Uncertainty risk, ID-only LTT $\tau$	0.2280 $\pm$ 0.0027	0.0	35.4 $\pm$ 5.2	5.3 $\pm$ 1.7	0 / 4 / 8	2 / 1 / 1	37.9 $\pm$ 3.6

Table 6: Abstention on B2 under shift (Qwen3-0.6B; 3 seeds

\times

9 shifted environments = 27 tests per

\alpha

). “Pooled” thresholds are certified by LTT on ID and proxy-shift calibration data together, “ID-only” on ID data alone; B5 uses the ID threshold and routes unknown inputs to

\tau_{\mathrm{shift}}

. Coverage and realized risk average the nine environments (mean

\pm

s.d. over seeds); a coverage of 0 means that no threshold could be certified and the rule abstains on everything. Violations count (environment, seed) tests whose realized risk exceeds

\alpha

; because the test items are shared across seeds, the counts are descriptive. MSP and LAC rank inputs by the same score and therefore have the same AURC.

Risk score / threshold rule	Shift AURC $\downarrow$	[email protected] (%)	[email protected] (%)	[email protected] (%)	Violations $\alpha$ =.05/.10/.20	[email protected] per seed	Gold removed: abstain (%)
MSP score, pooled LTT $\tau$	0.0902 $\pm$ 0.0021	37.3 $\pm$ 5.4	55.5 $\pm$ 6.8	7.7 $\pm$ 0.6	6 / 5 / 9	2 / 1 / 2	34.6 $\pm$ 2.7
Energy score, pooled LTT $\tau$	0.3417 $\pm$ 0.0057	0.0	0.0	0.0	0 / 0 / 9	0 / 0 / 0	100.0
Split-conformal LAC (not risk-certified)	0.0902 $\pm$ 0.0021	39.3 $\pm$ 4.0	73.8 $\pm$ 4.6	12.1 $\pm$ 0.7	13 / 9 / 7	3 / 3 / 3	27.0 $\pm$ 3.4
Shift-Jev risk score, pooled LTT $\tau$	0.0948 $\pm$ 0.0063	50.1 $\pm$ 1.5	55.9 $\pm$ 1.9	4.7 $\pm$ 0.4	6 / 3 / 3	1 / 1 / 1	48.4 $\pm$ 33.3
B4: uncertainty risk, pooled LTT $\tau$	0.0958 $\pm$ 0.0047	50.5 $\pm$ 0.7	55.8 $\pm$ 2.0	4.8 $\pm$ 0.6	7 / 3 / 4	1 / 1 / 1	17.2 $\pm$ 4.7
Uncertainty risk, ID-only LTT $\tau$	0.0958 $\pm$ 0.0047	1.4 $\pm$ 2.4	61.2 $\pm$ 8.1	6.6 $\pm$ 2.8	1 / 6 / 3	1 / 0 / 5	10.5 $\pm$ 11.1

Table 7: Abstention on B2 under shift (Qwen3-4B-Instruct-2507; 3 seeds

\times

9 shifted environments = 27 tests per

\alpha

). “Pooled” thresholds are certified by LTT on ID and proxy-shift calibration data together, “ID-only” on ID data alone; B5 uses the ID threshold and routes unknown inputs to

\tau_{\mathrm{shift}}

. Coverage and realized risk average the nine environments (mean

\pm

s.d. over seeds); a coverage of 0 means that no threshold could be certified and the rule abstains on everything. Violations count (environment, seed) tests whose realized risk exceeds

\alpha

; because the test items are shared across seeds, the counts are descriptive. MSP and LAC rank inputs by the same score and therefore have the same AURC. The 4B model has no consistency signal, so B5 without the format signal reduces to the uncertainty risk with routing and is omitted.

Model	Risk score	Threshold rule	Shift cov. .05/.10/.20 (%)	[email protected] (%)	ID [email protected] (%)	Violations .05/.10/.20	Sum
0.6B	uncertainty	pooled (ID + proxy)	20.2 / 37.9 / 66.9	6.4	58.8	4 / 5 / 13	22
		ID only	0.0 / 35.4 / 60.7	5.3	54.6	0 / 4 / 8	12
		ID + routing to $\tau_{\mathrm{shift}}$ (B5 rule)	0.0 / 35.4 / 60.7	5.3	54.6	0 / 4 / 8	12
		$\tau_{\mathrm{shift}}$ for all inputs	0.0 / 35.0 / 60.3	5.2	53.9	0 / 3 / 8	11
	Shift-Jev	pooled (ID + proxy)	18.0 / 39.1 / 63.3	7.5	60.8	4 / 8 / 10	22
		ID only	0.0 / 33.3 / 68.0	5.5	51.1	0 / 4 / 14	18

Table 8: Isolating the threshold rule from the risk score (both Qwen backbones, 3 seeds, 27 tests per

\alpha

). All thresholds are re-certified offline from the stored calibration splits of each run. Routing changes a threshold only when

\tau_{\mathrm{proxy}}<\tau_{\mathrm{ID}}

, which never happened at

\alpha=0.05

; “

\tau_{\mathrm{shift}}

for all inputs” applies the stricter of the ID and proxy thresholds everywhere.

Figure 4: (a, b) Selective-risk violations out of 27 (environment, seed) tests at each target risk level. At

\alpha=0.05

, B5 abstains on (almost) all inputs (coverage 0.0% and 1.4%), so its zero count there is vacuous; see Tables 6 and 7 . (c) Mean coverage against mean realized risk at

\alpha=0.10

(error bars: s.d. over seeds); the dashed line marks the target.

Figure 5: Risk–coverage curves on six shifted environments (Qwen3-0.6B, seed 1) for MSP, the uncertainty risk score (B4) and the Shift-Jev risk score (B5). The dotted line marks

\alpha=10\%

Model	Perturbation	Type	Acc B1 (%)	Acc B2 (%)	Rotation agreement (%)	B4 abstain / risk (%)	B5 abstain / risk (%)
0.6B	Option rotation	harmless	70.6	77.5	87.7	38.7 / 9.6	49.5 / 8.7
0.6B	Label paraphrase	harmless	70.6	77.1	88.9	35.9 / 10.2	47.7 / 8.8
0.6B	Near-duplicate distractor	harmless	59.3	72.1	77.0	47.1 / 6.5	61.8 / 2.6
0.6B	Gold option removed	task-altering	–	–	81.9	32.8 / 100.0	42.2 / 100.0
4B	Option rotation	harmless	92.6	90.2	–	9.3 / 5.9	10.8 / 4.4
4B	Label paraphrase	harmless	89.5	90.2	–	9.2 / 7.2	5.9 / 6.2

Table 9: Candidate-set perturbations (seed 1). For gold-removed items every answer is wrong, so abstention should be high. Rotation agreement is computed only at 0.6B (the consistency signal is not used at 4B).

Model	Detector	Domain shift	Emerging classes	Class-prior shift	Candidate perturbation	False alarm (%)
0.6B	CUSUM $\cdot$ max-probability	98 (6) $\cdot$ 99	98 (11) $\cdot$ 100	0 $\cdot$ 0	34 (81) $\cdot$ 18	2
0.6B	CUSUM $\cdot$ entropy	33 (19) $\cdot$ 25	0 $\cdot$ 0	0 $\cdot$ 0	9 (88) $\cdot$ 5	4
0.6B	CUSUM $\cdot$ $k$ NN drift	97 (12) $\cdot$ 98	98 (7) $\cdot$ 98	0 $\cdot$ 1	1 (90) $\cdot$ 1	6
0.6B	CUSUM $\cdot$ Shift-Jev risk	99 (6) $\cdot$ 98	100 (7) $\cdot$ 98	0 $\cdot$ 0	36 (57) $\cdot$ 17	7
0.6B	KS window $\cdot$ Shift-Jev risk	97 (13) $\cdot$ 98	100 (16) $\cdot$ 100	0 $\cdot$ 0	31 (90) $\cdot$ 15	3
0.6B	$\chi^{2}$ $\cdot$ predicted labels (BBSE)	97 (10) $\cdot$ 96	94 (10) $\cdot$ 96	95 (37) $\cdot$ 80	70 (96) $\cdot$ 27	6

Table 10: Stream-level shift detection: detection rate (%) at severity 100% (mean delay in items)

\cdot

at 50%, 50 streams per environment and severity. Every alarm threshold is calibrated to

\leq

5% false alarms on 200 simulated no-shift streams; the last column is the observed false-alarm rate on 50 fresh no-shift streams per pool.

Figure 6: Detection rate against shift severity for four detectors (top: Qwen3-0.6B; bottom: Qwen3-4B; 50 streams per point). Only the predicted-label

\chi^{2}

test detects class-prior shift.

Environment	0.6B B4 cov / risk	0.6B Shift-Jev cov / risk	0.6B violating streams B4 / Shift-Jev (%)	4B B4 cov / risk	4B Shift-Jev cov / risk	4B violating streams B4 / Shift-Jev (%)
Banking77	0.0 / 0.0	0.0 / 0.0	0 / 0	0.0 / 0.0	0.0 / 0.0	0 / 0
Twitter-Fin	0.0 / 0.0	0.0 / 0.0	0 / 0	0.0 / 0.0	0.0 / 0.0	0 / 0
MMLU	18.2 / 27.3	9.2 / 9.9	98 / 44	66.5 / 16.0	59.3 / 12.5	100 / 84
AG prior $\rho$ =10	87.3 / 5.4	75.8 / 4.1	0 / 0	93.2 / 6.1	88.7 / 4.5	4 / 0
AG prior $\rho$ =100	89.0 / 7.1	78.0 / 6.5	6 / 4	93.5 / 8.6	86.3 / 5.5	20 / 0
CLINC-OOS	14.0 / 7.2	4.6 / 8.6	20 / 38	0.0 / 0.0	0.0 / 0.0	0 / 0

Table 11: Selective risk after a change point (50 streams per environment,

\alpha=0.10

, seed 1). Shift-Jev switches to

\tau_{\mathrm{shift}}

after a drift-CUSUM or label-

\chi^{2}

alarm.

Method (3 seeds)

Avg acc (%)

BWT (pts)

Forgetting (pts)

Banking77

Twitter-Fin

CLINC

Stages adapted

Train (min)

Shift-Jev

-

row (pts)

Seeds Shift-Jev better

p

(items)

Naive fine-tuning

66.65

\pm

1.17

-

1.1

\pm

1.0

1.1

\pm

1.0

76.9

\pm

0.9

42.7

\pm

2.9

60.5

\pm

2.2

86.5

\pm

2.0

4.0/4

9.3

+3.83

\pm

1.88

3/3

1.1\times 10^{-9}

EWC

66.48

\pm

2.10

0.4

\pm

1.6

-

0.3

\pm

1.7

77.2

\pm

0.8

42.0

\pm

4.6

59.9

\pm

3.5

86.8

\pm

1.6

4.0/4

9.6

+4.00

\pm

2.88

3/3

6.3\times 10^{-8}

Seq-FT + KL anchors (base paper)

66.33

\pm

1.92

0.3

\pm

1.0

-

0.2

\pm

1.1

77.1

\pm

0.9

41.8

\pm

4.1

59.4

\pm

2.8

87.0

\pm

1.3

4.0/4

9.5

+4.15

\pm

2.74

3/3

1.2\times 10^{-8}

Experience replay

66.67

\pm

1.71

-

0.3

\pm

2.7

1.4

\pm

2.5

76.2

\pm

1.1

46.2

\pm

3.2

56.5

\pm

5.7

87.8

\pm

1.0

4.0/4

13.4

+3.81

\pm

2.53

3/3

6.7\times 10^{-7}

LwF (tree-local)

68.10

\pm

1.71

0.3

\pm

1.3

-

0.3

\pm

1.3

77.2

\pm

1.3

43.6

\pm

2.5

63.0

\pm

2.4

88.6

\pm

2.7

4.0/4

9.4

+2.38

\pm

2.52

3/3

8.5\times 10^{-4}

Shift-Jev (expected-gain gate)

70.48

\pm

0.88

3.0

\pm

0.9

-

2.1

\pm

1.9

77.6

\pm

0.9

54.7

\pm

1.9

61.3

\pm

1.9

88.3

\pm

0.6

3.0/4

12.4

–

Table 12: Continual adaptation on Qwen3-0.6B. Stream: Banking77

\rightarrow

control (no shift)

\rightarrow

Twitter-Fin

\rightarrow

CLINC-OOS, starting from B2. Final accuracy (%) on 400 held-out items per regime; mean

\pm

s.d. over 3 seeds. The last three columns compare Shift-Jev with each row: the per-seed paired difference in average accuracy (mean

\pm

s.d.), the number of seeds in which Shift-Jev is better, and a two-sided sign test on the 1,600 unique evaluation items using each item’s correctness averaged over the three seeds (the evaluation items are shared across seeds, so seeds are not pooled as independent pairs).

Figure 7: Continual adaptation on Qwen3-0.6B. (a) Final average accuracy and forgetting (mean $\pm$ s.d., 3 seeds; shaded: Shift-Jev ablations). (b) Final accuracy per regime: the gain over the baselines comes from Banking77, and the earlier gate loses CLINC150, the regime it never adapts to.

Model	Regime	Accepted	Stopped by probe	Gain (pts)	Sign-test $p$	v4 trigger	v4 gate adapted
Qwen3-0.6B	Banking77	3/3	0/3	18.8 to 32.5	$1.3\times 10^{-6}$ to $2.0\times 10^{-3}$	drift+perf, perf	3/3
	Control (no shift)	0/3	3/3	$-$ 3.8 to 0.0	0.66 to 0.97	none	0/3
	Twitter-Fin	3/3	0/3	11.2 to 15.0	$2.1\times 10^{-3}$ to 0.025	perf	3/3
	CLINC-OOS	3/3	0/3	17.5 to 26.2	$4.8\times 10^{-7}$ to $6.6\times 10^{-4}$	none	0/3
Qwen3-4B	Banking77	3/3	0/3	10.0 to 12.5	$3.2\times 10^{-3}$ to 0.029	perf	–
	Control (no shift)	0/3	2/3	$-$ 2.5 to 1.2	0.5 to 1	none	–

Table 13: Expected-gain gate decisions of Shift-Jev (3 seeds per backbone). Gain = held-out accuracy gain (points) of the best candidate on 80 new-regime items, with its one-sided sign-test

p

; an update is kept iff

p<0.10

and old-regime accuracy drops by

\leq

5 points. “Trigger” = the v4 shift test’s outcome (logged only); the last column shows how often the v4 shift gate adapted.

Figure 8: All expected-gain gate decisions: held-out gain of the best candidate against its one-sided sign-test

p

-value (log scale). Filled markers: accepted; open markers: rolled back. On Qwen3-4B the Twitter-Fin gains of 2.5–7.5 points do not reach

p<0.10

on 80 items.

Method (3 seeds)	Avg acc (%)	BWT (pts)	Forgetting (pts)	ID	Banking77	Twitter-Fin	CLINC	Stages adapted	Train (min)	Shift-Jev $-$ row (pts)	Seeds Shift-Jev better	$p$ (items)
Naive fine-tuning	81.22 $\pm$ 0.55	0.8 $\pm$ 1.4	$-$ 0.4 $\pm$ 1.4	89.2 $\pm$ 1.1	66.9 $\pm$ 1.8	73.0 $\pm$ 1.0	95.8 $\pm$ 0.5	4.0/4	15.0	$-$ 1.36 $\pm$ 0.59	0/3	0.31
EWC	81.00 $\pm$ 1.01	1.0 $\pm$ 1.2	$-$ 0.5 $\pm$ 0.9	89.3 $\pm$ 1.2	67.0 $\pm$ 1.2	72.0 $\pm$ 2.0	95.7 $\pm$ 0.3	4.0/4	15.2	$-$ 1.14 $\pm$ 0.10	0/3	0.055
Seq-FT + KL anchors (base paper)	80.28 $\pm$ 1.12	0.7 $\pm$ 1.3	$-$ 0.3 $\pm$ 1.0	89.8 $\pm$ 1.6	64.7 $\pm$ 2.2	71.3 $\pm$ 1.5	95.3 $\pm$ 0.3	4.0/4	15.1	$-$ 0.42 $\pm$ 0.22	0/3	0.94
Experience replay	80.33 $\pm$ 0.76	$-$ 0.4 $\pm$ 1.1	1.6 $\pm$ 0.6	89.9 $\pm$ 0.8	67.7 $\pm$ 1.2	68.7 $\pm$ 1.8	95.1 $\pm$ 1.3	4.0/4	18.6	$-$ 0.47 $\pm$ 1.43	1/3	0.94
LwF (tree-local)	81.00 $\pm$ 0.60	1.0 $\pm$ 0.7	$-$ 0.6 $\pm$ 0.7	89.7 $\pm$ 1.2	66.0 $\pm$ 0.9	72.9 $\pm$ 1.0	95.4 $\pm$ 0.8	4.0/4	15.0	$-$ 1.14 $\pm$ 0.50	0/3	0.14
Shift-Jev (expected-gain gate)	79.86 $\pm$ 1.10	1.9 $\pm$ 1.0	$-$ 1.8 $\pm$ 1.0	89.0 $\pm$ 0.9	68.6 $\pm$ 1.0	65.9 $\pm$ 3.4	96.0 $\pm$ 0.3	2.0/4	17.9	–	–	–

Table 14: Continual adaptation on Qwen3-4B-Instruct-2507 (3 seeds; 300 held-out items per regime). The last three columns are computed as in Table 12 , with 1,200 unique evaluation items; negative differences mean that Shift-Jev is worse.

Figure 9: Qwen3-4B-Instruct: accuracy on each regime after every stage (mean of 3 seeds). Shift-Jev keeps ID and Banking77 stable and is best on CLINC150, but stays at its starting level on Twitter-Fin after rejecting that update.

Model	Method	Eval set	Acc (%)	ID-only $\tau$	B4 pooled $\tau$	B4 regime $\tau$	Shift-Jev (B5)
0.6B	Shift-Jev	ID	76.5	53.8 / 6.5	62.0 / 8.9	59.8 / 11.3	64.5 / 11.6
		Banking77	56.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0
		Twitter-Fin	60.2	0.0 / 0.0	0.0 / 0.0	1.2 / 40.0	1.2 / 40.0
		CLINC-OOS	89.0	0.0 / 0.0	0.0 / 0.0	43.2 / 4.0	44.5 / 4.5
	Experience replay	ID	77.0	45.2 / 5.0	62.7 / 9.6	51.5 / 9.7	57.5 / 10.0
		Banking77	45.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0

Table 15: Reliability after continual adaptation (B6, seed 1): coverage / realized risk (%) at

\alpha=0.10

on the continual evaluation sets after the last stage. Each adapted regime receives its own certified threshold.

Figure 10: Scale and model family. (a) Frozen accuracy on the base paper’s external benchmarks. (b) ECE of the frozen models under shift. (c) Abstention when the correct option is removed (bars: mean of 3 seeds; dots: individual seeds).

Anchor weight	ID acc (%)	ID ECE	Banking77	MMLU	SciQ	KL start	KL answer	Text acc (%)	Text $\equiv$ score (%)	Runaway replies
$\lambda$ = 1	79.8	0.057	18.6	45.6	93.0	0.0002	0.0034	81	100	0/100
$\lambda$ = 0.1	79.8	0.066	18.2	45.6	92.7	0.0008	0.0043	80	100	0/100
$\lambda$ = 0	80.2	0.066	18.8	46.6	92.7	0.6204	0.2329	81	100	85/100

Table 16: Base-paper anchor-weight ablation on Qwen3-0.6B (seed 1).

$K$	Acc (%)	AURC B4	AURC B5	ms/item
5	81.2	0.056	0.052	49
20	62.0	0.193	0.198	115
77	18.6	0.695	0.693	383

Table 17: Sweeps on Qwen3-0.6B (seed 1). Left: candidate-set size on Banking77. Middle: labelled data for one Shift-Jev stage on Banking77 (the gate rejects the 75-item update). Right: adapter configuration (shift accuracy over nine environments; AURC of B5).

Figure 11: (a) KL divergence to the base model at the reply start and after the answer, by anchor weight (log scale). (b) Runaway text-mode replies out of 100. (c) Accuracy and latency against candidate-set size on Banking77.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

System	BoolQ	MMLU	MMLU-Pro	ARC-C	WinoGr.	SciQ	Macro-6	Banking77	ECE (6)
Base paper, Qwen3.5-4B, training-free	88.00	77.08	47.12	93.33	66.20	99.75	78.58	69.03	–
Base paper, Qwen3.5-4B, LoRA $\lambda$ =1	88.20	77.00	46.90	92.70	68.80	100.00	78.93	75.40	–
Ours, Qwen3-4B-Instruct-2507, B1	87.20	70.25	45.00	90.33	62.80	99.00	75.76	57.30	0.220
Ours, Qwen3-4B-Instruct-2507, B2 (seed 1)	88.20	69.42	45.00	90.17	62.20	99.25	75.71	57.10	0.105
Ours, Phi-4-mini-instruct, B1	85.60	65.17	39.25	84.33	63.60	98.00	72.66	55.80	0.111

Appendix

Table 18: External benchmarks for the larger backbones (accuracy, %; Banking77 = fixed random 1,000 of 3,080 items). The paper’s 4B model is Qwen3.5-4B.

Risk score / threshold rule	Shift AURC $\downarrow$	[email protected] (%)	[email protected] (%)	[email protected] (%)	Violations $\alpha$ =.05/.10/.20	[email protected] per seed	Gold removed: abstain (%)
MSP score, pooled LTT $\tau$	0.1119	31.8	49.0	6.3	1 / 0 / 3	0	30.9
Split-conformal LAC	0.1119	43.2	68.4	14.3	4 / 4 / 1	4	30.9
Uncertainty risk, pooled $\tau$	0.1291	32.9	47.9	5.2	1 / 2 / 1	2	25.0
Uncertainty risk, ID-only $\tau$	0.1291	0.0	49.0	5.6	0 / 2 / 1	2	19.6
Drift inside risk	0.1609	33.5	45.7	12.1	5 / 3 / 4	3	22.1
Shift-Jev layer (B5 rule)	0.1595	0.0	48.7	7.9	0 / 2 / 1	2	14.2

Appendix

Table 19: Frozen Phi-4-mini-instruct with abstention layers (one run; 9 shifted environments, 9 tests per

\alpha

). Columns as in Table 6 .

Quantity	Value
Scoring latency, mean over environments, 0.6B / 4B / Phi-4-mini	72 / 92 / 89 ms/item
Peak GPU memory, scoring 0.6B / 4B	2.87 / 8.37 GB
Peak GPU memory, LoRA training 0.6B / 4B	3.23 / 9.09 GB
LoRA parameters ( $r$ = 16, all linear), 0.6B / 4B	10.1 M / 33.0 M
Adapter storage (fp32), 0.6B / 4B	40.4 / 132.1 MB
B2 training + reference cache, 0.6B (2,000 examples)	8.8 + 2.3 min

Appendix

Table 20: Computational cost on Kaggle Tesla T4 GPUs.

Parameter	Value
LoRA rank / $\alpha$ / dropout / targets	16 / 32 / 0 / all linear layers
Optimizer, learning rate, weight decay	AdamW, $10^{-4}$ , 0.01
Schedule	10% warm-up, linear decay to 10%, 1 epoch, batch 8
Gradient clipping, max length	1.0, 1024 tokens
Anchor weight $\lambda$ , truncation	1, top-64 + tail
Risk levels $\alpha$ , LTT $\delta$	$\{0.05,0.10,0.20\}$ , 0.1

Appendix

Table 21: Hyperparameters.

Online Language Model Adaptation

See all 11 papers

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
Oct 1, 2026Yinheng Li, Justin WagleLLM Inference LLM Fine-Tuning
Microsoft
Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors
Oct 19, 2026Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir SaleemLanguage Model Calibration Model Calibration
Faculty of Computer Science and Information Technology, The Superior University, Lahore, Pakistan · Independent Researcher
Conformalized Large Language Models under Configuration Shift
Aug 2, 2026Yuqicheng Zhu, Jialin Yu, Lin Li +7Conformal Prediction Conformal Prediction for Language Models
University of Stuttgart · Robert Bosch GmbH · University of Oxford +4

Oct 1, 2026J/K move · Enter open · S save

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Yinheng Li, Justin Wagle

Microsoft

LLM Inference LLM Fine-Tuning Language Model Calibration LLM Decision-Making

Open paper AskPDF arXiv

Jev-style decision models have gained surging traction by making decision-making dramatically faster and more efficient, returning categorical probability distributions over predefined options without generating free-form text. However, because Jev's architecture and training methodologies remain proprietary, how to replicate these capabilities effectively remains an open question. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework offering both a training-free inference recipe and a fine-tuning objective with KL-divergence anchors. Across Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: off the shelf, frozen models match or exceed existing training-free readouts on the same 4B backbone, scale to arbitrary option counts, and natively handle multimodal decisions. They also seamlessly inherit the backbone's thinking mode: while Jev was introduced as a System 1 model, "Thinking Jev" leverages budgeted test-time reasoning to boost accuracy on challenging tasks without training. Fine-tuning provides targeted gains for smaller models and complex routing, while our KL anchors prevent behavioral degradation in general text generation.

Abstract

Online Language Model Adaptation

Explore similar work

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors

Conformalized Large Language Models under Configuration Shift