Prism-1: Position- and Rival-Independent Scoring for System One Models

Yao Long Teng; Jianguo Huang; Jiayi Cai; Yanchen Deng; Bo An

cs.AIOct 15, 2026

Prism-1: Position- and Rival-Independent Scoring for System One Models

Authors: Yao Long Teng, Jianguo Huang, Jiayi Cai, Yanchen Deng, Bo An

Organizations: Nanyang Technological University, Singapore

Abstract

Given a state and a set of options, such as intents, tools or interface elements, System One decision models return a distribution over the options in one forward pass, without generated reasoning. Web systems increasingly build them by reading decisions from language models. Two axioms of rational choice are natural requirements for such decisions, because in web systems the listing and the option set are determined by other components and can change or be manipulated: the order in which the options are listed should not matter, and adding an option should not change the relative odds of the others (Luce's choice axiom). We test the standard reads, namely single-choice question (SCQ) prompting, with shuffled-order training, order averaging or sorted options, and invariant set architectures, and find that each violates at least one axiom: one shuffle changes 12-20% of SCQ answers, sorting only freezes an arbitrary convention, and a well-chosen added option steers 4-34% of decisions. By Luce's theorem, a read satisfies both axioms only if it is a softmax over scores that each depend on one option alone, so in a transformer they hold by construction only if no token that scores an option sees another. We implement this as Prism-1 (Position- and Rival-Independent Scoring for System One Models): every option is read in its own block at shared positions, with the model's usual left-to-right attention, in one forward pass. On five benchmark sets with Qwen3-8B and Gemma 4 12B, SCQ prompting trained on the same data still changes 7% of its answers under one shuffle, and an added irrelevant option steers 12% of its decisions. Prism-1 stays within one point of that baseline's accuracy and is 2-4 points more accurate once only answers that survive a reordering are counted, while no shuffle changes any of its answers and no added irrelevant option moves its choice among the others.

Figures & tables

Figure 1. SCQ violates both axioms, after training and after sorting (mean over five sets). (a) Share of answers that change under one shuffle after each epoch of training on shuffled orders, and when the options move but keep their own letters. (b) After one epoch of training, the answers that change when the options are shuffled, zero by construction with sorted options because sorting undoes the shuffle, the answers that move to a different original option when one irrelevant option is added, and, for options sorted by text, the answers that change when they are sorted Z–A instead of A–Z.

Figure 2. Prism-1 . (a) Every option is a block that starts at the same position

P

after the shared context, and all blocks are read in one forward pass of the pretrained decoder, fine-tuned with LoRA. The score

s_{k}=\log P(\textit{yes}\mid h,b_{k})-\log P(\textit{no}\mid h,b_{k})

is read at the last token of each block (

\blacksquare

), a softmax over the blocks gives the decision, and training minimises

-\log R(o_{y}\mid S)

for the correct option

o_{y}

. (b) The attention mask: the context attends causally to itself, and each block attends to the context and causally to itself, never to another option. No option’s score depends on the other options or on the listing.

Closed-source models, through their APIs
	Banking20				Newsgroups				BEV				Typed				JevBench				Avg
Method	Acc	Flip	Dr.	St.	Acc	Flip	Dr.	St.	Acc	Flip	Dr.	St.	Acc	Flip	Dr.	St.	Acc	Flip	Dr.	St.	Acc	Rob.	Flip	Dr.	St.
Jev 1.13 §	87.0	4.0	1.6	3.3	75.3	3.7	2.8	8.0	76.4	6.3	3.2	13.9	72.8	8.0	5.1	22.0	88.5	1.4	1.6	7.2	80.0	77.9	4.7	2.9	10.9
Mercury Decide §	91.3	2.3	1.6	7.0	73.3	7.7	2.6	8.3	75.1	12.5	4.7	27.2	67.3	10.5	6.8	30.5	89.9	5.0	1.4	7.2	79.4	76.7	7.6	3.4	16.0
Open-source models: Gemma 4 12B
SCQ	91.9	5.4	1.8	3.0	71.8	9.4	4.2	13.2	77.5	10.1	5.2	20.3	75.9	5.3	4.8	11.2	84.9	5.6	2.8	10.8	80.4	77.8	7.1	3.8	11.7

Table 1. Main results. Acc: accuracy (%). Flip: answers changed by one shuffle (%). Dr.: probability moved among the original options when one irrelevant option is added (points). St.: questions on which an added irrelevant option can move the answer to another original option (%). Rob.: order-robust accuracy, right in both the source and the shuffled order (%). Flip, Dr. and St. are zero for a rule that satisfies both axioms. For SCQ with sorted options, Rob. compares A–Z with Z–A sorting. § : published model, not trained on our data. Bold: best per model, excluding published models.

Figure 3. BEV by domain: order-robust accuracy of SCQ (grey) and Prism-1 (blue), which for Prism-1 equals its accuracy. Prism-1 is ahead wherever its dot lies to the right.

Figure 4. Injected options: answers (%) that an added option naming a wrong option as correct moves to that option, per set.

Figure 5. The number of options and the choice axiom, with every method trained for one epoch on the same data. (a, b) Order-robust accuracy, right both in the source order and in the shuffled order, on the 300 Newsgroups test posts with

K

nested topics as options. Shaded: SCQ’s answers that are right in the source order but not after one reshuffle, up to its accuracy in the source order alone. (c) Switch rate: answers that move to a different original option when one irrelevant option is added, averaged over five sets.

	Gemma 4 12B		Qwen3-8B
Variant	Acc	Switch	Acc	Switch
Prism-1	79.6	0.0	77.7	0.0
options attend to each other	74.4	3.5	74.8	1.3
+ score read after the set	76.4	3.8	72.0	3.4
+ likelihood of the option’s text	74.7	3.6	72.1	5.1
Prism-1 + answer slot	79.7	0.0	76.1	0.0

Table 2. Ablation of Prism-1 : mean accuracy and switch rate when an option is added (%). Each row changes one more design choice, and the fourth row is the design of Set-LLM.

Figure 6. The price of the choice axiom: accuracy relative to Prism-1 against the answers an added option moves to another original option. Grey: run-to-run noise, two standard deviations.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7. What bidirectional attention among options costs. (a) Validation accuracy before and after one epoch of training, for a bidirectional set prompt (Set-LLM’s design) and for isolated causal blocks with the answer slot. (b) Accuracy of the trained bidirectional set prompt minus trained SCQ, in points.

Set	$K$	Eval	Train	State (chars)
Banking20 ( Casanueva et al., 2020 )	20	300	2,450	55 / 71
Newsgroups ( Lang, 1995 )	20	300	2,450	542 / 750
BEV ( avbiswas, 2026 )	2–10	1,182	2,450	232 / 787
Typed Decisions ( LocalLLaMA, 2026 )	4–5	600	1,750	708 / 697
JevBench ( Standhartinger, 2026 )	3–6	139	–	82 / 2,563

Appendix

Table 3. The benchmark. Train: training questions, after holding out 50 per set for validation. State: length in characters, median / mean.

		Gemma 4 12B			Qwen3-8B
Variant	S	Acc	JB	Sw.	Acc	JB	Sw.
Options compare (not of the required form)
gated channel, rivals’ last tokens	1	$-0.2$	$-1.4$	0.7	$-0.2$	$0.0$	1.0
gated channel, all rival tokens	1	$+0.1$	$+1.4$	0.7	–	–	–
gated channel, gates forced open	1	$-1.2$	$+1.4$	–	–	–	–
comparison head, all options	2	$+0.9$	$+3.2$	0.0	$+0.7$	$+3.6$	0.0

Appendix

Table 4. The mechanisms of Section 5.6 : accuracy minus Prism-1 ’s with the same seeds (points), overall and on JevBench (JB), and switch rate (Sw., %). S: seeds; –: not run.

	Tokens		Attn. pairs (k)		ms
Method	B20	NG	B20	NG	B20	NG
SCQ	282	417	40	100	60	63
Set-LLM	322	413	60	168	68	66
Prism-1	221	356	21	76	58	60

Appendix

Table 5. Cost per decision with 20 options on Banking20 (B20) and Newsgroups (NG): tokens processed, attention pairs (query–key pairs the mask allows, thousands) and milliseconds on one H100 GPU, Gemma 4 12B, medians over the first 100 questions of each set.

Model	Flip	Dr.	St.	Sw.
Jev 1.13	4.7	2.9	10.9	3.4
identical copies	1.3	1.3	5.1	1.3
Mercury Decide	7.6	3.4	16.0	3.5
identical copies	0.7	0.5	1.5	0.7

Appendix

Table 6. The closed-source models against their noise floor, five-set means. Identical copies: the same question repeated in one request, scored as the added-option variants are. Sw.: switch rate (%).

	Gemma 4 12B		Qwen3-8B
Read	Target	Switched	Target	Switched
SCQ	23.1	25.3	9.6	13.0
SCQ + cyclic avg.	10.0	10.8	4.7	6.1
SCQ, sorted	11.2	13.0	2.9	5.1
Set-Based Prompting	54.8	58.5	31.4	41.8
Set-LLM	22.7	23.6	15.3	18.1

Appendix

Table 7. Injected options: answers (%) moved to the named option (Target) or to any other original option (Switched).

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 237 papers

Oct 13, 2026

Consider, Then Choose: Order-Invariant Decisions in System One Models

System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose

\textbf{Runoff}

, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.

Jianguo Huang, Yao Long Teng, Yanchen Deng +1

Nanyang Technological University, Singapore

Input Order Sensitivity in Language Models Multiple-Choice Question Answering Permutation-Invariant Learning Language Model Bias Evaluation

Ask PDF

Oct 13, 2026

Jeva: System One Models from First Principles

System one models like Jev enable direct, low-latency decision-making. Yet recent open-source practice of predicting an answer from serialized text does not, by itself, respect the structure of a decision problem. We argue that current designs underemphasize four important structural requirements. (i) null-aware: Models should distinguish answer absence from uncertainty among candidates; (ii) permutation-invariant: Reordering alternatives should not change the decision; (iii) option-scalable: Growing candidate sets should not require redefining the output space; (iv) cacheable: Request-independent computation over recurring alternatives should be reusable across requests. We introduce Jeva, a decision-native architecture that addresses these requirements jointly. A shared encoder independently represents the request and each option, while a compact decision stack refines only the request state against immutable option memory. Symmetric aggregation yields permutation-equivariant probabilities; shared scoring accommodates variable-size candidate sets. A separate coverage estimate distinguishes answer absence from candidate competition, enabling explicit null outcomes. Request-independent option representations and layer projections support reuse across newly assembled sets. These mechanisms unify Choice, Noul, and Score outputs. Across evaluations against Jev and open source models, Jeva combines strong typed-decision performance with near-perfect null detection and no decision flips under any tested permutations. It scales to thousands of candidates and substantially reduces latency through cross-request reuse without changing decisions.

Qi Li, Xinchao Wang

National University of Singapore

LLM Decision-Making Efficient Language Model Inference Selective Prediction Permutation-Invariant Learning

Ask PDF

Oct 15, 2026

System1-Arena: Which Decisions Can Be Left to System 1?

Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.

Tianyu Chen, Xinyang Liu, Ruyi An +2

The University of Texas at Austin

LLM Decision-Making LLM Evaluation Efficient Language Model Inference

Ask PDF

Abstract

LLM Decision-Making

Explore similar work

Consider, Then Choose: Order-Invariant Decisions in System One Models

Jeva: System One Models from First Principles

System1-Arena: Which Decisions Can Be Left to System 1?