Consider, Then Choose: Order-Invariant Decisions in System One Models

Jianguo Huang; Yao Long Teng; Yanchen Deng; Bo An

cs.AIOct 13, 2026

Consider, Then Choose: Order-Invariant Decisions in System One Models

Authors: Jianguo Huang, Yao Long Teng, Yanchen Deng, Bo An

Organizations: Nanyang Technological University, Singapore

Abstract

System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose $\textbf{Runoff}$ , which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.

Figures & tables

Panel of Figure 1 — Figure 1: Accuracy and flip rate against the number of options, Qwen3-8B. Flip rate: share of answers that change when the options are shuffled.

Panel of Figure 2 — Figure 2: Order dependence of Listwise and its relation to the pointwise distribution for Qwen3-8B. Panels (b) and (c) pool Banking20 and Newsgroups. (a) Percentage of questions whose top k pointwise options hold 99.9% of the probability, at K=20. (b) Mean top-1 probability of Listwise and its flip rate at K=20, with questions binned by their pointwise top-1 probability, which does not depend on the order. (c) Percentage of uncertain (pointwise top-1 probability below 0.9) and certain (at least 0.999) questions as K grows.

Qwen3-8B
	Both	One	Neither
Banking20	43.1	47.1	9.8
Newsgroups	80.4	17.9	1.8
Gemma 4 12B
Banking20	69.6	30.4	0.0
Newsgroups	72.1	20.9	7.0

Table 1: Share of flips (%) with both, one, or neither of the two answers among the three candidates,

K{=}20

Open-source
		Banking20		Newsgroups		MMLU-Pro		Typed Decisions		JevBench		Avg
Model	Method	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$
Qwen3-8B	Listwise	73.0	17.0	68.0	18.7	47.6	44.4	62.7	12.2	66.2	21.6	63.5	22.8
	AnyJev	78.0	10.3	68.0	11.3	53.1	24.7	65.2	3.0	69.8	8.6	66.8	11.6
	Runoff (ours)	76.7	0.0	68.3	0.0	51.6	0.0	64.2	0.0	68.3	0.0	65.8	0.0
Gemma 4 12B	Listwise	83.7	7.7	67.3	14.3	57.1	31.9	70.0	11.5	87.1	6.5	73.0	14.4

Table 2: Main results on five decision benchmarks. Acc: accuracy (%) with the options in their original order. Flip: share of questions (%) whose answer changes when the options are shuffled. Native: the model’s own choice answer. Avg: mean over the five datasets. Best per model in bold.

	Native		AnyJev		Runoff
Model	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$
Torchcast 12B	70.2	15.6	73.3	5.4	72.2	0.0
Kev 9B	72.7	12.5	73.2	7.2	72.8	0.0
Intern-Decision 4B	70.6	14.7	72.9	8.1	73.0	0.0
Tev1 4B	68.4	12.3	70.2	7.0	70.2	0.0
Laya	40.7	30.6	42.3	15.4	42.4	0.0

Table 3: Results of open-weight models fine-tuned for System One decisions: accuracy and flip rate (%), averaged over the five datasets (per-dataset results in Table 9 ). Best per model in bold.

	$K$	Qwen3-8B	Gemma 4 12B
Banking20	20	308 $\to$ 77 ( 4.0 $\times$ )	525 $\to$ 121 ( 4.3 $\times$ )
Newsgroups	20	248 $\to$ 82 ( 3.0 $\times$ )	401 $\to$ 140 ( 2.9 $\times$ )
MMLU-Pro	10	125 $\to$ 78 ( 1.6 $\times$ )	216 $\to$ 135 ( 1.6 $\times$ )
Typed	4–5	79 $\to$ 85 (0.9 $\times$ )	140 $\to$ 145 (1.0 $\times$ )
JevBench	3–6	67 $\to$ 77 (0.9 $\times$ )	116 $\to$ 130 (0.9 $\times$ )
Avg	–	165 $\to$ 80 ( 2.1 $\times$ )	280 $\to$ 134 ( 2.1 $\times$ )

Table 4: Latency of AnyJev

\to

Runoff in ms per record, and the speedup of Runoff. A record is one choice question. Mean over the original and shuffled orders. Avg: mean latency over the five datasets.

Figure 4: OSWorld tasks solved by a System One computer-use agent with the native decisions and Runoff.

Variant	Acc $\uparrow$	Flip $\downarrow$
Consider only	63.3	0.0
+ Canonical ranking, one listing	67.3	0.0
+ one listing (input order)	68.3	15.8
+ Choose (Runoff)	69.0	0.0

Table 5: Ablations: accuracy and flip rate (%), means over the ten model–dataset cells of Table 2 . Consider only: the pointwise distribution,

P=\pi

. One listing: Choose makes one listwise decision on the consideration set, with its members in the order of the input. Cyclic shifts: Choose makes listwise decisions on the cyclic shifts of the canonical ranking.

Setting	Bank.	News.	MMLU	Typed	JevB.	$m$
$\alpha=0.9$	76.7	68.3	50.6	64.2	67.6	3.2
$\alpha=0.95$	76.7	68.3	51.6	64.2	67.6	3.3
$\alpha=0.99^{\dagger}$	76.7	68.3	51.6	64.2	68.3	3.5
$\alpha=0.999$	75.7	68.3	52.1	64.2	68.3	3.9
$k_{\min}=1$	76.0	68.3	50.2	62.2	64.7	2.1
$k_{\min}=2$	75.7	68.7	50.7	61.8	66.2	2.7

Table 6: Accuracy (%) per dataset, Qwen3-8B, and the mean size

m

of the consideration set over the five datasets.

\dagger

: the default setting of Runoff.

	Listwise		AnyJev		Runoff
$K$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$
2	95.8	4.1	96.2	0.3	96.3	0.0
4	89.8	5.7	90.3	2.5	90.4	0.0
8	84.4	8.5	84.3	3.8	84.2	0.0
12	79.3	10.6	80.5	6.2	79.2	0.0
16	77.0	14.0	77.8	6.3	77.8	0.0

Table 7: Accuracy and flip rate (%) against the number of options

K

, on the questions of Figure 1 : means over Qwen3-8B and Gemma 4 12B on Banking20 and Newsgroups, 300 questions per model, dataset and

K

(per model and dataset in Table 10 ). Best per

K

in bold.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen3-8B			Gemma 4 12B
Shuffled	All	$<0.9$	$\geq 0.9$	All	$<0.9$	$\geq 0.9$
Candidates	14.0	28.9	9.4	7.2	11.6	5.4
Others	15.3	30.3	10.7	6.0	9.8	4.4
All options	17.8	38.0	11.6	11.0	20.2	7.3
Questions	600	142	458	600	173	427

Appendix

Table 8: Flip rate (%) of Listwise under controlled shuffles,

K{=}20

, Banking20 and Newsgroups pooled, over all questions and split by the pointwise top-1 probability. Candidates: only the top three options of the pointwise distribution are permuted, among the positions they hold. Others: only the other options are permuted, every candidate keeping its position and letter. All options: a full shuffle.

		Banking20		Newsgroups		MMLU-Pro		Typed Decisions		JevBench		Avg
Model	Method	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$
Torchcast 12B	Native	79.0	11.7	64.7	14.0	54.7	34.7	66.2	13.5	86.3	4.3	70.2	15.6
	AnyJev	84.3	2.3	66.7	6.0	60.1	13.8	68.2	3.7	87.1	1.4	73.3	5.4
	Runoff (ours)	82.7	0.0	65.7	0.0	56.5	0.0	68.2	0.0	87.8	0.0	72.2	0.0
Kev 9B	Native	90.3	7.0	70.7	6.3	55.4	31.4	64.2	11.2	82.7	6.5	72.7	12.5
	AnyJev	90.3	2.0	70.7	7.3	58.5	19.4	64.5	3.7	82.0	3.6	73.2	7.2

Appendix

Table 9: Results of the fine-tuned decision models per dataset (Table 3 gives the means); columns as in Table 2 . Best per model in bold.

Panel of Figure 7 — Figure 7: Figure 1 for Gemma 4 12B: accuracy and flip rate of Listwise and AnyJev against the number of options.

Panel of Figure 8 — Figure 8: Figure 2 for Gemma 4 12B, with the same panels, thresholds, and bins.

			$K{=}2$		$K{=}4$		$K{=}8$		$K{=}12$		$K{=}16$		$K{=}20$
Model	Dataset	Method	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$	Acc $\uparrow$	Flip $\downarrow$
Qwen3-8B	Banking20	Listwise	95.0	4.7	90.7	8.7	85.0	12.7	81.7	11.3	79.0	15.3	73.0	17.0
		AnyJev	95.3	0.3	92.0	2.0	87.0	5.7	82.7	6.3	81.3	6.3	78.0	10.3
		Runoff	95.3	0.0	92.0	0.0	86.0	0.0	82.3	0.0	81.0	0.0	76.7	0.0
Qwen3-8B	Newsgroups	Listwise	95.0	5.3	86.3	5.0	79.0	9.3	73.3	14.3	71.0	18.3	68.0	18.7
		AnyJev	95.3	0.7	86.7	3.7	78.3	4.7	76.3	9.0	71.7	9.7	68.0	11.3

Appendix

Table 10: Accuracy and flip rate (%) against the number of options

K

, per model and dataset (Table 7 gives the means), on the questions of Figure 1 , 300 per

K

. Best per model, dataset, and

K

in bold.

Input Order Sensitivity in Language Models

See all 14 papers

Prism-1: Position- and Rival-Independent Scoring for System One Models
Oct 15, 2026Yao Long Teng, Jianguo Huang, Jiayi Cai +2LLM Decision-Making Permutation-Invariant Learning
Nanyang Technological University, Singapore
System One models Are Not Better Rerankers
Oct 23, 2026Michael Dinzinger, Kanishka Ghosh Dastidar, Michael GranitzerLLM Reranking Information Retrieval
University of Passau, Passau, Germany · Interdisciplinary Transformation University Austria (IT:U), Linz, Austria
Jeva: System One Models from First Principles
Oct 13, 2026Qi Li, Xinchao WangLLM Decision-Making Efficient Language Model Inference
National University of Singapore

Oct 15, 2026J/K move · Enter open · S save

Prism-1: Position- and Rival-Independent Scoring for System One Models

Yao Long Teng, Jianguo Huang, Jiayi Cai, Yanchen Deng+1

Nanyang Technological University, Singapore

LLM Decision-Making Permutation-Invariant Learning

Open paper AskPDF arXiv

Given a state and a set of options, such as intents, tools or interface elements, System One decision models return a distribution over the options in one forward pass, without generated reasoning. Web systems increasingly build them by reading decisions from language models. Two axioms of rational choice are natural requirements for such decisions, because in web systems the listing and the option set are determined by other components and can change or be manipulated: the order in which the options are listed should not matter, and adding an option should not change the relative odds of the others (Luce's choice axiom). We test the standard reads, namely single-choice question (SCQ) prompting, with shuffled-order training, order averaging or sorted options, and invariant set architectures, and find that each violates at least one axiom: one shuffle changes 12-20% of SCQ answers, sorting only freezes an arbitrary convention, and a well-chosen added option steers 4-34% of decisions. By Luce's theorem, a read satisfies both axioms only if it is a softmax over scores that each depend on one option alone, so in a transformer they hold by construction only if no token that scores an option sees another. We implement this as Prism-1 (Position- and Rival-Independent Scoring for System One Models): every option is read in its own block at shared positions, with the model's usual left-to-right attention, in one forward pass. On five benchmark sets with Qwen3-8B and Gemma 4 12B, SCQ prompting trained on the same data still changes 7% of its answers under one shuffle, and an added irrelevant option steers 12% of its decisions. Prism-1 stays within one point of that baseline's accuracy and is 2-4 points more accurate once only answers that survive a reordering are counted, while no shuffle changes any of its answers and no added irrelevant option moves its choice among the others.

Abstract

Input Order Sensitivity in Language Models

Explore similar work

Prism-1: Position- and Rival-Independent Scoring for System One Models

System One models Are Not Better Rerankers

Jeva: System One Models from First Principles