Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev

Dushyant Rajput; Nirdesh Chauhan; Siddharth Kosaraju

cs.AIOct 12, 2026

Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev

Authors: Dushyant Rajput, Nirdesh Chauhan, Siddharth Kosaraju

Organizations: AltSlate Labs LLP

Abstract

Decision APIs that return a probability for each option are routinely used as if those probabilities were properties of the options themselves. That reading is valid only if the model satisfies independence of irrelevant alternatives (IIA): adding an option that is neither the right answer nor its strongest rival should leave the odds between those two unchanged. We test this in a pre-registered study of jev-1.13.0, a hosted decision model, using 299,463 calls on intent classification (BANKING77, CLINC150) and topic classification (AG News). Every added option moves the log-odds between the gold label and the strongest wrong label: the root-mean-square systematic shift is 0.20-0.52 nats, 1.2 to 3 times the run-to-run variation, even when the added option takes at most 0.1% of the probability mass. The shifts are directional. An option similar to the wrong answer draws probability from it (+0.10 nats toward the gold label). A paraphrase of the wrong answer produces an intermediate pattern: the gold label's probability is almost conserved, but the paraphrase receives only about a third of the wrong answer's share rather than the half that strict substitution implies. The distortion grows steadily with the logarithm of the number of options, reaching about 1 nat at 255 options, and off-domain padding pushes the odds against the gold label. In contrast, questions packed into one call do not interact (bias < 0.001). Scoring each option with an independent yes/no question removes the option-set dependence but picks the right answer 8.7 points less often. Jev sits between two local baselines, an option-independent scorer and a prompt readout. Its probabilities are best read as relative to the option list they were computed over.

Figures & tables

Figure 1: Study workflow. Items, decoys, padding, companions and analysis code were frozen (sha256) and pre-registered before any measurement call. All experiments ran in one shuffled pass with the model version pinned; the base set was re-run at the end as a drift check. Local baselines ran Exp 1–2 on contested items (§3.2).

Experiment	Manipulation	Calls
Exp 1: one added option (2,500 items)	Base $\{A,B\}$ (both orders $\times$ 3 repeats) vs $\{A,B,C\}$ for five decoys $C$ , each in all 6 orderings plus a repeat of the two $C$ -last orderings	115,000
Exp 1n: neutral names (200 items)	Options renamed Option NN , label text moved into the description	4,800
Exp 2: option count (1,000 items)	$K\in\{2,4,\dots,128,255\}$ with off-domain or in-domain padding; $A,B$ in positions 1–2 (primary) or randomly permuted	96,000
Exp 3: question isolation (1,000 items)	Choice $\{A,B\}$ alone vs with 1–128 unrelated yes/no questions, a related question about $A$ or $B$ , last position, or a leading question id; 3 repeats each	30,000
Exp 4: yes/no per option (1,000 items)	One Noul question per option, all in one call, over the Exp 1 and in-domain $K$ option sets; plus a 300-item one-question-per-call reference	38,663

Table 1: The four pre-registered experiments (284,463 calls) and the neutral-name probe. All jobs were shuffled together and run in one pass; the Exp 1 base set was then re-run as a drift check (15,000 calls).

H	Claim	Decision rule
H1	Adding one option moves $L$ beyond noise	Mean $y>0$ , Holm-adjusted [ 21 ] $p<.05$ , in any of 5 decoys $\times$ 2 main sets
H2	The shift is directional	Signed mean $\Delta L_{c}$ 95% CI excludes 0 (near-B $>0$ : similarity; $<0$ : attraction)
H2-clone	Luce vs EBA	Luce: $\Delta L_{\text{pair}}$ 90% CI within $\pm 0.10$ and $\Delta L_{\text{clone}}$ 95% CI covers $-\ln 2$ ; EBA: the mirror image; otherwise intermediate
H3	The shift grows with $K$	Per-item OLS slope of $y_{K}$ on $\log_{2}K$ ; 95% CI lower bound $>0$
H4	Questions in one call are isolated	Each condition: mean $\Delta P(A)$ 90% CI within $\pm 0.02$ and per-item RMS shift $<0.01$ (two one-sided tests, TOST [ 22 ] )
H5	Yes/no per option fixes it at no cost	(a) RMS shift $<0.10$ for all decoys and (b) accuracy difference 95% CI lower bound $>-0.02$

Table 2: Pre-registered hypotheses and decision rules. BANKING77 and CLINC150 are the main sets; AG News is a reported control outside the Holm family.

H	Verdict	Key estimate (95% CI)
H1	Supported, 10/10 cells	RMS shift 0.20–0.52 nats; RMS run-to-run noise 0.15–0.17
H2	Supported (near-B, clone, far, random)	near-B $+0.099$ $[0.073,0.124]$ ; near-A $+0.008$ $[-0.019,0.035]$
H2-clone	Intermediate	$\Delta L_{\text{pair}}=+0.365$ $[0.335,0.396]$ ; $\Delta L_{\text{clone}}=-0.110$ $[-0.163,-0.060]$ ; $\lambda=0.53^{a}$
H3	Supported	squared-excess slope 0.134 $[0.113,0.157]$ nats 2 per doubling of $K$
H4	Supported, 9/9 conditions	all mean $\Delta P(A)$ within $\pm 0.0006$ (90% CIs within $\pm 0.0013$ ); per-item RMS $<0.003$
H5	Not supported	(a) RMS $<0.02$ , passes; (b) accuracy $-0.087$ $[-0.105,-0.070]$ , fails

Table 3: Verdicts under the pre-registered rules.

{}^{a}\lambda=\Delta L_{\text{pair}}/\ln 2

, the fraction of the EBA prediction.

Figure 2: One added option. (a) RMS systematic shift in the gold-vs-rival log-odds per decoy and dataset, with 95% CIs; the dashed line is the RMS run-to-run noise between identical base calls. (b) Signed shift (positive = toward the gold label) by the added option’s position, BANKING77 and CLINC150 pooled.

Figure 3: Mean probabilities before and after adding a paraphrase of the wrong answer

B

(547 primary contested items, BANKING77 and CLINC150, clone listed last, both A/B orders), with the Luce and EBA predictions computed from the base split.

Figure 4: Option count, 283 contested items,

A

and

B

listed first. (a) RMS systematic shift against

K

(95% CI bands) for Jev and the two baselines. (b) Accuracy on the same items.

Figure 5: Mean change in

P(A)

when other questions share the call, with 90% CIs, against the pre-registered

\pm 0.02

equivalence margin.

Figure 6: One yes/no question per option. (a) RMS shift in the ratio of yes-probabilities under added options and padding, against the same quantity for Choice. (b) Accuracy of Choice and of the argmax yes-probability on the same option sets (y-axis starts at 0.6).

Figure 7: RMS shift from one added option, by system (BANKING77 and CLINC150 contested items).

	Jev-1.13.0	Kev-4B	Label-prob
Run-to-run noise (median $\|\Delta L\|$ )	0.09–0.11	0	0
RMS shift, one added option	0.20–0.52 (AG 0.56)	0.01–0.03	0.70–1.10
Signed shift, near-B / clone	$+0.10$ / $+0.36$	$+0.004$ / $+0.005$	$+0.18$ / $+0.02$
Clone $\Delta L_{\text{pair}}$ / $\Delta L_{\text{clone}}$ a	$+0.37$ / $-0.11$	$+0.005$ / $-0.56$	$-0.01$ / $-0.43$
Contested accuracy, $K=2$	0.729	0.747	0.549
$K=255$ off-domain / in-domain	0.627 / 0.678	0.578 / 0.592	0.497 / 0.510

Table 4: Jev against the two baselines on the same contested items. Accuracy here is on contested items only and is not comparable with the all-item figures above. a On the 547 primary clone items.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Added option	RMS shift	Signed $\Delta L$	C first	Flips	Mass to C
	b77 / clinc / ag	(95% CI)		(b77)	(b77)
far topic	0.20 / 0.23 / 0.17	$+0.046$ $[0.024,0.068]$	$-0.009$	5.6%	0.1%
random in-set	0.20 / 0.21 / 0.24	$+0.034$ $[0.013,0.055]$	$+0.053$	5.8%	0.1%
near-A	0.31 / 0.26 / 0.24	$+0.008$ $[-0.019,0.035]$	$+0.104$	5.3%	1.0%
near-B	0.32 / 0.24 / 0.25	$+0.099$ $[0.073,0.124]$	$+0.179$	5.4%	1.3%
B-clone	0.52 / 0.43 / 0.56	$+0.360$ $[0.330,0.389]$	$+0.572$	14.7%	10.4%

Appendix

Table 5: Exp 1 by condition. Signed shifts pool BANKING77 and CLINC150 (added option last). “C first” is the signed shift with the added option listed first. Flips: top choice differs from the base. a Median

|\Delta L|

between identical calls.

$\mathbf{K}$	4	8	16	32	64	128	255
RMS shift, off-domain	0.26	0.39	0.53	0.68	0.83	0.96	1.02
RMS shift, in-domain	0.26	0.35	0.42	0.54	0.60	0.69	0.80
Signed $\Delta L$ , off-domain	$+0.06$	$+0.01$	$-0.04$	$-0.15$	$-0.24$	$-0.32$	$-0.31$
Accuracy, off-domain	.852	.853	.852	.847	.840	.830	.814
Accuracy, in-domain	.855	.856	.856	.855	.859	.849	.839
Mass to padding, off-domain	0.2%	0.2%	0.4%	0.8%	1.8%	3.8%	6.8%

Appendix

Table 6: Exp 2,

A

and

B

listed first. Shifts on 283 contested items; accuracy on all 1,000 items (

K=2

: 0.853 off-domain, 0.854 in-domain).

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

JevOut: Natural Context Can Flip Decision Models
Sep 24, 2026Zixiang Xu, Zirui Song, Chiyu Zhang +4Adversarial Attacks on LLMs LLM Decision-Making
University of Southern California
Labels Override Definitions in Jev-Style Typed Decision Models
Oct 1, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramLanguage Model Bias Evaluation Prompt Sensitivity
University of Southern California (USC), USA
When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models
Oct 21, 2026Rushab MunotLLM Decision-Making LLM Reliability
Independent Researcher

Sep 24, 2026J/K move · Enter open · S save

JevOut: Natural Context Can Flip Decision Models

Zixiang Xu, Zirui Song, Chiyu Zhang, Xiuying Chen+3

University of Southern California

Adversarial Attacks on LLMs LLM Decision-Making Language Model Robustness

Open paper AskPDF arXiv

An ordinary-looking background detail can turn a correct model decision into a confident mistake. We demonstrate this fragility in four decision systems, including Jev, across seven datasets covering knowledge, reasoning, and tool routing. Within 64 accepted target evaluations per decision, we uncover short context additions that redirect 61.4%-73.2% of each system's initially correct decisions toward a wrong option fixed in advance. The additions supply background or procedural information rather than explicit answer-selection instructions, leaving the original question and choices intact. We construct them through probability-guided context optimization, which uses shifts in the option distribution to refine surrounding text under naturalness and answer-preservation constraints. Redirection affects initially confident decisions, often produces high-confidence wrong choices, and transfers across models. In blinded human evaluation, 91.6% of 250 sampled successful contexts are judged natural, answer-preserving, and free of decisive answer-changing evidence by a majority of three independent annotators. These findings expose a weakness in current decision models: context that looks entirely compatible with an input can redirect the choices that agents, routers, and evaluators rely on.

Abstract

LLM Decision-Making

Explore similar work

JevOut: Natural Context Can Flip Decision Models

Labels Override Definitions in Jev-Style Typed Decision Models

When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models