When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models

Rushab Munot

cs.AIOct 21, 2026

When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models

Authors: Rushab Munot

Organizations: Independent Researcher

Abstract

Decision models return normalized probability distributions over candidate choices, which downstream applications can use to select among choices. Although their providers emphasize the reliability of these probabilities, calibration on tasks with a single correct choice does not establish their reliability when multiple choices are correct. We evaluate three decision models (Jev, Liquid d1, and GPT-6 Luna Decisions) on three diverse datasets (MetaTool, HotpotQA, and ECtHR) to study probability allocation and thresholding. Even when models rank every correct choice above every distractor, in 28% to 50% of those lists, at least one correct choice receives a probability below the uniform probability of 1/n, where n is the number of choices. We also present each model with five identical copies of a correct choice and observe probability disparities, with mean gaps ranging from 17 to 79 percentage points. We further show how these allocation disparities affect the recovery of multiple correct choices under probability thresholding. Decision-model probabilities can rank correct choices, but a probability threshold does not identify the set of correct choices.

Figures & tables

Question: who is older, Maria Sharapova or Chanda Rubin?

Instruction: Choose every passage in the list that supports the answer to the question.

Chanda Rubin ( born February 18, 1976 ) is an American tennis …

Maria Yuryevna Sharapova (… born April 19, 1987 ) is a Russian professional …

Table 1: An example from HotpotQA asks which of the two tennis players is older, and lists 10 passages. The two passages below are necessary and sufficient together. However, the models concentrate almost all probability on one passage. The full list of passages is in Appendix A .

	MetaTool	HotpotQA	ECtHR
Query	Query	Question	Case facts
Choice	Tool	Passage	Article
Choices per list	5 to 15	2 to 10	10
One correct choice	2,795	–	655
Two or more	497 (2)	7,405 (2)	192 (2 to 4)
No correct choice	995	–	153

Table 2: The three datasets, with the number of examples by their number of correct choices, given in brackets for two or more. A list has 10 choices, except the MetaTool scenario lists, with 5, 10 or 15 tools (100, 1,400 and 300 queries), and 60 HotpotQA lists, with 2 to 9 passages.

	MetaTool, 2 correct			HotpotQA, 2 correct			ECtHR, 2 to 4 correct
Model	Gap	Ranked	Below $1/n$	Gap	Ranked	Below $1/n$	Gap	Ranked	Below $1/n$
Jev	74	73%	50%	60	82%	31%	58	79%	36%
Liquid d1	55	70%	29%	55	85%	28%	57	76%	42%
GPT-6 Luna Decisions	55	52%	35%	57	70%	29%	61	73%	45%

Table 3: For lists with several correct choices, Gap is the mean probability of the highest correct choice minus that of the lowest, in percentage points, Ranked is the share of lists with every correct choice above every distractor, and Below

1/n

is the share of those lists with a correct choice below 1/n for a list of n choices.

1/n=0.1

for every list except 39 HotpotQA lists of 3 to 9 passages. The two shares use the released order of the ECtHR lists and leave out the lists with no distractor.

	MetaTool	HotpotQA	ECtHR
Model	Gap	Gap	Gap
Jev	18	26	17
Liquid d1	77	69	37
GPT-6 Luna Decisions	79	75	51

Table 4: Each list has 5 copies of one correct choice and is sent in its 5 cyclic permutations. The table shows the mean probability of the highest copy minus that of the lowest copy, in percentage points ( Gap ). Equal probabilities give 0.

	MetaTool		HotpotQA		ECtHR
Model	First	Last	First	Last	First	Last
Jev	36%	25%	66%	16%	10%	56%
Liquid d1	98%	1%	98%	1%	97%	1%
GPT-6 Luna Decisions	98%	1%	98%	1%	76%	1%

Table 5: The share of answers that predict the copy in the first and in the last position, on the lists of Table 4 . An equal share is 20%.

	MetaTool		HotpotQA		ECtHR
Model	List	Yes/no	List	Yes/no	List	Yes/no
Jev	0.57	0.31	0.62	0.29	0.52	0.44
Liquid d1	0.46	0.38	0.64	0.27	0.48	0.54
GPT-6 Luna Decisions	0.34	0.24	0.49	0.27	0.54	0.38

Table 6: The highest exact set accuracy over the thresholds 0, 0.01, …, 1. Exact set accuracy is the share of examples in which every correct choice is at or above the threshold and every distractor is below it. List uses the probabilities of the list and Yes/no the probability of yes of each choice.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Question	who is older, Maria Sharapova or Chanda Rubin?
Instruction	Choose every passage in the list that supports the answer to the question.
Title	Passage
1997 IGA Classic – Doubles	Chanda Rubin and Brenda Schultz-McCarthy were the defending champions but did not compete that year.
1997 EA-Generali Ladies Linz	The 1997 EA-Generali Ladies Linz was a women’s tennis tournament played on indoor hard courts at the Intersport Arena in Linz in Austria that was part of Tier III of the 1997 WTA Tour. …
1997 Australian Open – Women’s Doubles	Chanda Rubin and Arantxa Sánchez Vicario were the defending champions, but competed this year with different partners. …
1997 State Farm Evert Cup – Doubles	Chanda Rubin and Brenda Schultz-McCarthy were the defending champions but only Rubin competed that year with Mary Joe Fernandez.

Appendix

Table 7: The opening HotpotQA question, the instruction to choose every passage that supports the answer, and the 10 passage titles and texts in the order sent.

Instruction	Model	Rubin	Sharapova	Distractor
Supports	Jev	0.93	0.07	0.00
	Liquid d1	0.94	0.06	0.00
	Luna	0.93	0.02	0.01
Needed	Jev	0.78	0.22	0.00
	Liquid d1	0.91	0.09	0.00
	Luna	0.95	0.01	0.01

Appendix

Table 8: Mean Choice probability. Distractor is the highest probability among the other eight passages. Luna is GPT-6 Luna Decisions.

Instruction	Model	Rubin	Sharapova	Distractor
Supports	Jev	0.31	0.14	0.05
	Liquid d1	0.16	0.07	0.05
	Luna	0.47	0.00	0.01
Needed	Jev	0.61	0.40	0.05
	Liquid d1	0.53	0.41	0.03
	Luna	1.00	0.87	0.80

Appendix

Table 9: Mean probability of yes, with Distractor as in Table 8 .

Question	who is older, Jane or John?
Question	who is younger, Jane or John?
Instruction	Choose every passage in the list that supports the answer to the question.
Title	Passage
1998 Harlow Classic – Doubles	John and Petra Lindqvist were the defending champions but did not compete that year.
1998 Varden Indoor Open	The 1998 Varden Indoor Open was a tennis tournament played on indoor hard courts at the Kestrel Arena in Varden in Switzerland that was part of Tier III of the 1998 Meridian Tour. …
1998 Corvale Open – Doubles	John and Ilse Marquart were the defending champions, but competed this year with different partners. …

Appendix

Table 10: The questions and passages with invented names. John stands for Chanda Rubin and Jane for Maria Sharapova, and the passages are in the order of Table 7 .

Question	Passage	Jev	Liquid d1	Luna
Who is older?	John	0.91	0.94	0.06
	Jane	0.09	0.06	0.79
Who is younger?	John	0.02	0.15	0.03
	Jane	0.98	0.85	0.89

Appendix

Table 11: Choice probability of John’s and Jane’s passages. Luna is GPT-6 Luna Decisions.

	MetaTool		HotpotQA		ECtHR
Model	Before	After	Before	After	Before	After
Jev	0.11	0.71	0.17	0.66	0.19	0.71
Liquid d1	0.16	0.62	0.19	0.68	0.16	0.68
GPT-6 Luna Decisions	0.11	0.53	0.16	0.60	0.15	0.68

Appendix

Table 12: Mean probability of the second correct choice in the full list ( Before ) and after the removal ( After ).

	MetaTool		HotpotQA		ECtHR
Model	Renorm.	After ∗	Renorm.	After ∗	Renorm.	After ∗
Jev	0.77	0.72	0.80	0.67	0.72	0.71
Liquid d1	0.64	0.61	0.80	0.70	0.70	0.67
GPT-6 Luna Decisions	0.40	0.49	0.61	0.60	0.59	0.68

Appendix

Table 13: Mean probability of the second correct choice after renormalisation ( Renorm. ) and after the removal ( After ∗ ), on the examples in which the choices other than the removed one hold 0.1 or more.

Figure 1: Reliability diagram of the yes/no questions on examples with two or more correct choices, in bins of width 0.1. Each point is at the mean probability of yes of its bin. The top row gives the observed frequency of correct choices, and the bottom row the observed frequency minus the mean probability, in percentage points. The bars give the share of the answers in each bin, on a log scale.

Position	ID	Jev	Liquid d1	GPT-6 Luna Decisions
1	7bg8	0.43	0.66	0.82
2	c9jq	0.11	0.08	0.04
3	3nwa	0.15	0.06	0.03
4	6hp7	0.09	0.07	0.02
5	2haw	0.22	0.13	0.09
	Gap	34	60	80

Appendix

Table 14: Probability of each copy in one run, in the order sent. The highest and the lowest probability of each model are in bold, and Gap is the highest minus the lowest, in percentage points.

	One correct choice
	MetaTool	ECtHR
Model	ECE	ECE
Jev	8	3
Liquid d1	9	1
GPT-6 Luna Decisions	14	1

Appendix

Table 15: ECE in percentage points of the confidence on lists with one correct choice.

	Yes/no questions
	MetaTool		HotpotQA		ECtHR
Model	All bins	From 0.1	All bins	From 0.1	All bins	From 0.1
Jev	3	9	4	12	9	12
Liquid d1	3	10	8	17	4	8
GPT-6 Luna Decisions	9	18	9	15	3	8

Appendix

Table 16: ECE in percentage points of the probability of yes on the yes/no questions, over all bins ( All bins ) and without the bin from 0 to 0.1 ( From 0.1 ).

Figure 2: Exact set accuracy, the share of examples in which every correct choice is at or above the threshold

t

and every distractor is below it. Above 0.50 the list curves are 0 by construction.

	Random choices		The example’s own
	MetaTool	HotpotQA	MetaTool	HotpotQA	ECtHR
Model	Conf.	Conf.	Conf.	Conf.	Conf.
Jev	0.78	0.58	0.80	0.61	0.83
Liquid d1	0.74	0.49	0.77	0.67	0.87
GPT-6 Luna Decisions	0.79	0.71	0.78	0.65	0.87

Appendix

Table 17: Mean confidence on the answered lists with no correct choice and no “none of the above” choice ( Conf. ). Appendix I counts the refused lists.

	Random choices		The example’s own
	MetaTool	HotpotQA	HotpotQA	ECtHR
Model	Selected	Selected	Selected	Selected
Jev	95%	100%	84%	37%
Liquid d1	98%	100%	88%	7%
GPT-6 Luna Decisions	93%	100%	79%	17%

Appendix

Table 18: Share of predictions that select “none of the above”, the only correct choice ( Selected ).

Model	First	Tenth	Change
Jev	0.78	0.80	+2
Liquid d1	0.69	0.74	+4
GPT-6 Luna Decisions	0.80	0.67	$-13$

Appendix

Table 19: Accuracy with the correct tool first and tenth, and the change in percentage points.

List	MetaTool	HotpotQA	ECtHR
Several correct choices	497	7,405	192 (1,920)
Five identical copies	3,292 (16,460)	1,000 (5,000)	847 (4,235)
One correct choice	2,795 (16,770)	–	655 (6,550)
No correct choice, random choices	3,292 (16,460)	1,000 (5,000)	–
No correct choice, the example’s own	995 (3,980)	7,377	153 (1,530)
Random choices and “none of the above”	995 (4,975)	1,000 (5,000)	–

Appendix

Table 20: Examples by dataset and list type, with the number of lists or yes/no questions per model in parentheses where it differs. GPT-6 Luna Decisions refused some of them (Appendix I ).

Model	Dataset	Setting	Threshold	Ranked	Exact among ranked	Precision	Recall	Best F1
Jev	MetaTool	List	0.02	0.73	0.78	0.86	0.87	0.87 (0.02)
	MetaTool	Yes/no	0.35	0.52	0.59	0.70	0.83	0.78 (0.25)
Liquid d1	MetaTool	List	0.06	0.70	0.66	0.80	0.84	0.83 (0.05)
	MetaTool	Yes/no	0.60	0.61	0.63	0.76	0.81	0.79 (0.47)
GPT-6 Luna Decisions	MetaTool	List	0.05	0.52	0.65	0.73	0.77	0.75 (0.05)
	MetaTool	Yes/no	0.10	0.42	0.57	0.68	0.74	0.71 (0.12)

Appendix

Table 21: The threshold with the highest exact set accuracy on the examples of Table 6 ( Threshold ), the share of examples with every correct choice above every distractor ( Ranked ), and the share of those examples whose set is exact at that threshold ( Exact among ranked ), precision and recall at that threshold, and the highest F1 over all thresholds, with its threshold in parentheses ( Best F1 ). Precision, recall and F1 pool the choices of all examples. List uses the probabilities of the list, and Yes/no the probability of yes of each choice. The highest and the lowest value of each setting in the columns Threshold and Exact among ranked are in bold.

MetaTool
Figure	Jev	Liquid d1	GPT-6 Luna Decisions
Gap	74 [71, 76]	55 [52, 57]	55 [53, 58]
Ranked	73 [69, 77]	70 [66, 74]	52 [47, 56]
Below $1/n$	50 [45, 55]	29 [24, 34]	35 [30, 41]
Gap of the copies	18 [18, 18]	77 [76, 77]	79 [78, 79]
Exact set, list	0.57 [0.52, 0.61]	0.46 [0.42, 0.51]	0.34 [0.29, 0.38]

Appendix

Table 22: The figures of Tables 3 , 4 and 6 with their 95% intervals. Gap and Gap of the copies are in percentage points, and Ranked and Below

1/n

in percent.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?
Sep 27, 2026Keyi Li, Yihao He, Quanyi LiReasoning Consistency in Language Models
Software College, Northeastern University Shenyang, China
Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev
Oct 12, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLLM Decision-Making LLM Evaluation
AltSlate Labs LLP
JevOut: Natural Context Can Flip Decision Models
Sep 24, 2026Zixiang Xu, Zirui Song, Chiyu Zhang +4Adversarial Attacks on LLMs LLM Decision-Making
University of Southern California

Sep 27, 2026J/K move · Enter open · S save

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Keyi Li, Yihao He, Quanyi Li

Software College, Northeastern University Shenyang, China

Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Typed-decision models such as TypeSafe's Jev answer a declared yes/no or multiple-choice question about a state with a probability instead of text, and their evaluations report accuracy and calibration. Neither requires that the probabilities a model gives to logically related questions fit together item by item, which is what a system that acts on those probabilities needs. We test this property, coherence, with a battery of logically linked questions that needs no labels. For 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, we ask whether the label is X, whether it is not X, whether it is one of the other two labels, and which label applies. On 480 negation pairs, Jev's probabilities for "the label is X" and "the label is not X" miss summing to one by 0.064 on average (95% CI 0.055 to 0.072). Qwen3.8-27B, run from its official BF16 weights, misses by 0.293 with first-token probabilities and by 0.122 with verbalized probabilities. The gap to the first-token readout persists on pairs where both systems give similar probabilities, without double-negation labels, and after averaging Jev's repeated calls. Jev is not coherent either: its violations are about five times its repeat noise, and it over-endorses statements about single labels, so that its three single-label probabilities sum to 1.14 on average. The two systems also fail differently. Qwen3.8-27B's first-token readout under-endorses the complement of a label whether or not the question contains "not", rejecting both a statement and its negation in 196 of 480 pairs, and it does not become more coherent where it is more confident, whereas Jev's violations concentrate where its answer is uncertain. Because the checks need no labels, they expose biases that appear only when question forms are compared, and inconsistencies within items.

Abstract

LLM Decision-Making

Explore similar work

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev

JevOut: Natural Context Can Flip Decision Models