SanSi: A Looped Typed Decision Model for System 1.5 Thinking

Shuyu Gan; Young-Jun Lee; Dongyeop Kang

cs.CLOct 6, 2026

SanSi: A Looped Typed Decision Model for System 1.5 Thinking

Authors: Shuyu Gan, Young-Jun Lee, Dongyeop Kang

Organizations: University of Minnesota

Abstract

Typed decision models answer a declared question without generating text: a decision head returns a probability for each of the declared options in a single forward pass. A single pass is fast, intuitive System 1 thinking. We study what lies between one pass and generated reasoning: looping, in which the same layers are recursively applied several times before one typed readout. Each loop lets the model revise its hidden state before it commits to an answer, without generating a token; we call this System 1.5 thinking. We propose SanSi, which turns a pre-trained looped language model into a typed decision model. The option probabilities are read after every loop, and every loop is trained with a proper scoring rule, so that one model serves every budget from one loop to eight in a single run. On 10,027 test decisions from 59 sources, SanSi reaches 72.0% accuracy: 13.5 points above a non-looped model of the same shape trained with the same recipe, 5.3 points above a newer non-looped model of its size, and 1.8 points below one with three times the parameters. On two depth-controlled tasks, loops extend the solvable depth beyond the depths seen in training, where the larger single-pass model fails. Used as the judge for policy optimization with reinforcement learning, without gold answers, SanSi raises the generator's F1 by 7.7 points.

Figures & tables

Figure 1: Top: three ways to spend computation on a decision. Bottom: Accuracy on the 10,027 test decisions against backbone size, for the models trained on our data. Single-pass models gain accuracy by adding parameters; SanSi gains it by looping the same parameters. Kev-4B (our data) is Qwen3.5-4B trained with Kev’s recipe. Dashed line is the Jev API, which was not trained on our data.

Panel of Figure 2 — Figure 2: SanSi . (a) Training: one stack of 24 layers is applied $T$ times; after every loop the option probabilities are read and trained towards the target. (b) Test time: a test item from PAWS ( Zhang et al., 2019 ) followed through the loops; the first loop prefers the wrong option, the later loops the correct one.

		Test
Item type	Train	In-dist.	Near	Far
A Classification	2,400	321	–	768
B Multi-step reasoning	5,500	800	991	1,281
C Uncertain evidence	2,300	478	640	440
D Long documents	1,000	357	–	544
E Sentence pairs	1,600	160	240	968

Table 1: The decision suite: number of items by type and by distance from the training data. Type C contains the unanswerable and crowd-labelled items. The test set also contains the 231 public JevBench items (10,027 test items in total). Sources and an example of every type are in Appendix A .

Panel of Table 2 — Table 2: Main results on the 10,027 test items: mean $\pm$ standard deviation over three training seeds (bold: best). All models are trained on the same data, with our recipe except Kev-4B (our data), which uses Kev’s. SanSi -2.6B: the same recipe on Ouro-2.6B. Cost: GPU time of one pass over the test set, relative to Ouro-1.4B with one loop; all models are timed on one RTX A6000 with the same setting (Table 6 in Appendix B ; loop 3: three eighths of the eight-loop pass). Appendix D gives the full table. Shaded rows: SanSi .

Panel of Figure 3 — Figure 3: What looping costs and where it helps. (a) Accuracy against GPU time of one pass over the test set (unit: one loop of Ouro-1.4B). SanSi and SanSi -2.6B are read after each loop (bands: $\pm$ 1 s.d., three seeds); Ouro-1.4B trained with one loop coincides with loop 1 of SanSi . Kev-4B (our data): Qwen3.5-4B trained with Kev’s recipe. All models timed on one RTX A6000 (Table 6 ). (b) Accuracy by item type, ordered by the gain of SanSi at loop 8 over SmolLM2-1.7B (thick line; number on the right: gain in points). Error bars: $\pm$ 1 s.d., three seeds; values in Table 12 .

Panel of Figure 4 — Figure 4: SanSi read after each loop on the 10,027 test items (mean $\pm$ s.d., three seeds); horizontal lines: single-pass models; Kev-4B (our data) is not drawn, as its values almost coincide with those of Qwen3.5-4B (Table 2 ). (a) Accuracy, with the gains of loops 2 and 3. (b) Mean confidence on right and wrong answers. (c) ECE on answerable items. (d) Evidence AUROC. Values in Table 15 .

Panel of Figure 5 — Figure 5: How SanSi ’s answers change across loops (9,645 answerable items; mean $\pm$ s.d., three seeds). (a) Answers fixed or broken by loop $T$ relative to loop 1. (b) Answers that the loops after $T$ will still fix or break. (c) Mean settling loop by how many single-pass models (SmolLM2-1.7B, Qwen3.5-2B, Qwen3.5-4B) answer correctly. (d) Share of loop-8 answers that are right, by settling loop; above bars: share of items (%). Values in Tables 17 , 18 and 19 .

Panel of Figure 6 — Figure 6: Accuracy by depth $k$ , with SanSi read after 1, 2, 4 and 8 loops. Shaded: depths unseen in training ( $k>8$ ). Dashed: single-pass Qwen3.5-4B trained on the same data. Mean of three seeds; bands: $\pm$ 1 s.d. for loop 8 and Qwen3.5-4B.

Figure 7: Four test items: SanSi ’s probability for each option after every loop, and the three single-pass models (seed 0). Correct option starred and outlined; column maxima in bold. Item 3 has no correct option: its deciding sentence (struck out) was removed. Full prompts in Appendix K .

Figure 8: Running beyond the trained loop size: accuracy from loop 2 on for SanSi (trained with 8 loops, run for 16) and a model trained with 4 (run for 8). Hollow: untrained loops; bands:

\pm

1 s.d., three seeds. Values in Table 34 .

Loops that carry the loss
	Accuracy (%) read at loop				ECE at loop
Training	1	2	4	8	4	8
SanSi : every loop, cross-entropy + Brier, $T{=}8$	58.4 $\pm$ 0.7	66.9 $\pm$ 0.6	71.6 $\pm$ 0.5	72.0 $\pm$ 0.7	.085 $\pm$ .009	.093 $\pm$ .012
Loops 1, 2, 4, 8 only	59.2 $\pm$ 0.8	67.0 $\pm$ 0.3	71.0 $\pm$ 0.2	71.2 $\pm$ 0.2	.090 $\pm$ .013	.098 $\pm$ .014
Last loop only	35.6 $\pm$ 4.7	51.9 $\pm$ 7.9	67.9 $\pm$ 1.5	70.9 $\pm$ 0.4	.069 $\pm$ .017	.073 $\pm$ .009
Every loop, $T{=}4$	59.7 $\pm$ 0.5	67.6 $\pm$ 0.7	70.8 $\pm$ 0.4	69.6 $\pm$ 0.1	.094 $\pm$ .005	.106 $\pm$ .005

Table 3: Training choices: accuracy on the 10,027 test items and ECE, mean

\pm

standard deviation over three seeds. Every variant differs from SanSi in one choice. For

T=4

, loops 5–8 were never trained. For “loops 1, 2, 4, 8 only” (loss weight

1/4

at each of these loops) and “last loop only”, the loops without a loss are read with the frozen language-model head. “Cross-entropy only”: the loss of Equation 3 without the Brier term. “Reinforcement learning”: the training with a reward of Appendix H.2 .

Figure 9: GRPO on 2WikiMultiHopQA with SanSi ’s probability as the only reward: test F1 (3,000 questions) by the loop at which SanSi is read (filled: mean of three seeds; hollow: seeds; dashed: before training). Values in Table 38 .

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Item type	Sources (selection)	Example decision	Train	Test
A Classification	AG News, DBpedia, Yelp ( Zhang et al., 2015 ) , IMDB ( Maas et al., 2011 ) , SST-5 ( Socher et al., 2013 ) , TREC ( Li and Roth, 2002 ) , Banking77 ( Casanueva et al., 2020 ) , TweetEval ( Barbieri et al., 2020 )	“Meh. I was unimpressed.” Would this reviewer recommend the business? (A) no (B) yes $\rightarrow$ A	2,400	1,089
B Multi-step reasoning	CLUTRR ( Sinha et al., 2019 ) , ProofWriter ( Tafjord et al., 2021 ) , MuSiQue ( Trivedi et al., 2022 ) , FOLIO ( Han et al., 2022 ) , BBH ( Suzgun et al., 2023 )	“No homework is fun. Some reading is homework.” Is the statement “Some reading is fun.” true, false, or unknown? (A) true (B) false (C) unknown $\rightarrow$ C	5,500	3,072
C Uncertain evidence	ChaosNLI ( Nie et al., 2020b ) , Sys1Cal ( Porcedda, 2026 ) , question pairs with the evidence removed (from MuSiQue)	“A hockey fight.” Hypothesis: “fighting on the ice”. (A) entailment (B) neutral (C) contradiction $\rightarrow$ the annotators’ label distribution	2,300	1,558
D Long documents	HotpotQA ( Yang et al., 2018 ) , ContractNLI ( Koreeda and Manning, 2021 ) , QuALITY ( Pang et al., 2022 )	[two encyclopedia paragraphs] Are Anja Salomonowitz and Rod Lurie both directors? (A) no (B) yes $\rightarrow$ B	1,000	901
E Sentence pairs	MNLI ( Williams et al., 2018 ) , BoolQ ( Clark et al., 2019 ) , ANLI ( Nie et al., 2020a ) , WANLI ( Liu et al., 2022 ) , PAWS ( Zhang et al., 2019 ) , QNLI ( Wang et al., 2019 )	“Revco was acquired in 1997 by CVS.” Does this sentence mean the same thing: “Revco was subsequently acquired by CVS in 1997.” (A) no (B) yes $\rightarrow$ B	1,600	1,368
F Knowledge	ARC ( Clark et al., 2018 ) , CommonsenseQA ( Talmor et al., 2019 ) , MMLU ( Hendrycks et al., 2021 ) , SciQ ( Welbl et al., 2017 )	Coal is formed from (A) seas that have evaporated … (D) plant remains decomposed under pressure $\rightarrow$ D	–	1,808

Appendix

Table 4: Item types, a selection of their sources, and one test item of each type (state, question , options and target). The test total includes the 231 public items of JevBench.

Item type	Seen in training (training / in-distribution test items)	Near transfer	Far transfer
A Classification	2,400 / 321 : AG News † 300/41; Amazon † 300/40; Banking77 † 300/40; DBpedia † 300/40; IMDB † 300/40; SST-5 † 300/40; TREC † 300/40; Yelp † 300/40	–	768 : Emotion 240; TweetEval 240; Yahoo Answers 160; Emotion ‡ 64; Offensive tweets ‡ 64
B Multi-step reasoning	5,500 / 800 : ProofWriter 1,500/240; CLUTRR, 2–4 hops 1,200/160; Kev rules † 1,110/120; MuSiQue, 2–3 hops 1,000/160; Kev policies † 690/120	991 : CLUTRR, 5–10 hops 480; ProofWriter, paraphrased 319; MuSiQue, 4 hops 192	1,281 : BBH 480; HoVer 360; FOLIO 320; Kev policies, held out ‡ 64; Kev rules, held out ‡ 57
C Uncertain evidence	2,300 / 478 : MuSiQue pairs, 2–3 hops 1,000/158; SQuAD 2.0 800/240; ChaosNLI (SNLI, $\alpha$ NLI) 500/80	640 : ChaosNLI (MNLI) 400; MuSiQue pairs, 4 hops 240	440 : Sys1Cal 292; Kev unknowable pairs ‡ 148
D Long documents	1,000 / 357 : MuSiQue with distractors 600/237; HotpotQA 400/120	–	544 : ContractNLI 240; QuALITY 240; Kev buried evidence ‡ 64
E Sentence pairs	1,600 / 160 : BoolQ † 800/80; MNLI † 800/80	240 : MNLI, mismatched genres 240	968 : ANLI 240; PAWS 200; QNLI 200; WANLI 200; PAWS ‡ 64; QNLI ‡ 64
F Knowledge	–	–	1,808 : MMLU 480; ARC-Challenge 320; CommonsenseQA 240; MMLU-Pro 240; MMLU-Pro ‡ 160; ARC-Easy 120; SciQ 120; MMLU ‡ 64; SciQ ‡ 64

Appendix

Table 5: The sources of every item type in training and in the three test groups, with their numbers of items (bold: the total of the type). For a source seen in training, the two numbers are its training items and its in-distribution test items; near transfer consists of harder or reworded versions of these sources, far transfer of sources that are never trained on. † : from Kev’s training suite; ‡ : from Kev’s transfer suite. The 231 public JevBench items (48 easy, 72 standard, 111 hard) form a fourth test group.

	Backbone	Trained	Training	Test
Model	params	params	(GPU-min)	(GPU-s)
SmolLM2-1.7B	1.71B	72.6M	39 †	382
Ouro-1.4B, one loop	1.43B	60.8M	33	334
SanSi , $T=8$	1.43B	60.8M	313	2,571
SanSi -2.6B, $T=8$	2.67B	121.4M	558 §	4,963
Qwen3.5-2B	1.88B	67.5M	48 ‡	387

Appendix

Table 6: Sizes and measured cost. Test: one pass over the 10,027 test items on one RTX A6000, the same setting for all models. Training: 1,000 steps on RTX A6000 cards (wall-clock time times the number of cards), except † RTX A5500 and ‡ two RTX A5000; the times include five evaluations on the development set ( § two). ∥ : Kev’s recipe, two epochs (3,200 steps), without evaluations.

Panel of Table 7 — Table 7: Full results on the 10,027 test items (mean $\pm$ standard deviation over three seeds; the untuned backbones, the released Kev-4B and the Jev API are single runs; bold: best fine-tuned model). All fine-tuned models share the training data, recipe and seeds; Loops is the number of loops run at test time. Cost: GPU time of one pass over the test set, relative to Ouro-1.4B with one loop (Table 6 ; not measured for the untuned backbones and the released models). Hard ans.: share of the 382 unanswerable items that receive a hard answer. Evidence AUROC: separation of answerable from unanswerable items by confidence (786 items). The low hard-answer rate of the untuned SmolLM2-1.7B reflects its low confidence on all items. The released Kev-4B is read at the temperature it ships with. Shaded rows: looped models ( SanSi ); squares: model colours of the figures.

Difference (points)	All	In-dist.	Near	Far	JevBench
SanSi L8 $-$ SmolLM2-1.7B	$+13.5$ [ $+12.5$ , $+14.5$ ]	$+10.5$ [ $+8.8$ , $+12.2$ ]	$+10.0$ [ $+7.4$ , $+12.6$ ]	$+15.8$ [ $+14.5$ , $+17.0$ ]	$+14.0$ [ $+8.5$ , $+19.5$ ]
SanSi L8 $-$ Ouro one loop	$+13.4$ [ $+12.4$ , $+14.4$ ]	$+9.3$ [ $+7.7$ , $+11.0$ ]	$+7.9$ [ $+5.7$ , $+10.1$ ]	$+16.7$ [ $+15.3$ , $+18.0$ ]	$+12.6$ [ $+7.5$ , $+17.6$ ]
Ouro one loop $-$ SmolLM2-1.7B	$+0.1$ [ $-0.7$ , $+1.0$ ]	$+1.2$ [ $-0.1$ , $+2.5$ ]	$+2.2$ [ $+0.2$ , $+4.2$ ]	$-0.9$ [ $-2.2$ , $+0.3$ ]	$+1.4$ [ $-3.2$ , $+6.1$ ]
SanSi L1 $-$ Ouro one loop	$-0.2$ [ $-0.6$ , $+0.3$ ]	$-0.9$ [ $-1.9$ , $+0.0$ ]	$+1.2$ [ $+0.1$ , $+2.4$ ]	$-0.5$ [ $-1.1$ , $+0.1$ ]	$+4.8$ [ $+1.9$ , $+7.8$ ]
SanSi $T{=}4$ (L4) $-$ SmolLM2-1.7B	$+12.4$ [ $+11.4$ , $+13.4$ ]	$+10.3$ [ $+8.5$ , $+12.1$ ]	$+6.8$ [ $+3.8$ , $+9.8$ ]	$+15.0$ [ $+13.8$ , $+16.3$ ]	$+11.0$ [ $+5.3$ , $+16.2$ ]
Qwen3.5-4B $-$ SanSi $T{=}4$ (L4)	$+3.0$ [ $+2.4$ , $+3.6$ ]	$+2.1$ [ $+1.0$ , $+3.2$ ]	$+4.5$ [ $+3.0$ , $+6.1$ ]	$+2.8$ [ $+1.9$ , $+3.7$ ]	$+3.9$ [ $+0.4$ , $+7.5$ ]

Appendix

Table 8: Accuracy differences in points with 95% bootstrap intervals. L

t

: read at loop

t

. Colours: teal , the first term is significantly better (the interval excludes 0); red , significantly worse; grey , the interval includes 0.

Table 18

Panel of Table 11 — Table 11: Accuracy (%) on each of the 59 test sources: mean $\pm$ standard deviation over three seeds (the Jev API is called once). Type: A–F as in Table 1 ; Tier: in-distribution (ID), near transfer, far transfer, JevBench (JB). SanSi is read at loop 8. Shaded column: SanSi ; red : below SmolLM2-1.7B.

Panel of Table 12 — Table 12: Accuracy (%) by item type, all groups pooled: mean $\pm$ standard deviation over three seeds. $\Delta$ : SanSi at loop 8 minus SmolLM2-1.7B, paired by seed. Shaded columns: SanSi .

Panel of Table 13 — Table 13: Accuracy (%) by who was trained on the source of an item: mean $\pm$ standard deviation over three seeds (the released Kev-4B and the Jev API are single runs). Kev-4B (our data) is trained on our items with Kev’s recipe and read at temperature 1; the released Kev-4B is read at the temperature it ships with.

Panel of Table 14 — Table 14: Accuracy (%) on the public JevBench items by tier: mean $\pm$ standard deviation over three seeds (the Jev API is a single run).

	Accuracy (%)					ECE			Confidence		AUROC	Evid.	Hard	Answers
Loop	All	In-dist.	Near	Far	JevB.	In-dist.	Near	Far	right	wrong	r/w	AUROC	(%)	changed (%)
1	58.4	76.3	61.1	50.8	64.5	.020	.053	.156	.769	.567	.760	.835	27.1	–
	$\pm$ 0.7	$\pm$ 0.9	$\pm$ 1.3	$\pm$ 0.5	$\pm$ 0.7	$\pm$ .006	$\pm$ .008	$\pm$ .019	$\pm$ .020	$\pm$ .026	$\pm$ .005	$\pm$ .011	$\pm$ 2.1
2	66.9	82.8	64.2	61.9	67.0	.044	.110	.095	.822	.619	.777	.902	19.1	29.2
	$\pm$ 0.6	$\pm$ 0.4	$\pm$ 1.2	$\pm$ 0.8	$\pm$ 1.5	$\pm$ .005	$\pm$ .013	$\pm$ .016	$\pm$ .014	$\pm$ .015	$\pm$ .008	$\pm$ .002	$\pm$ 1.4	$\pm$ 0.3
3	70.4	85.6	67.0	65.9	71.1	.047	.119	.086	.847	.634	.788	.921	18.8	14.9

Appendix

Table 15: SanSi after every loop (10,027 test items; means of three seeds, with the standard deviation over the seeds on the line below). Confidence right/wrong: mean top probability on items answered correctly/wrongly at that loop. AUROC r/w: right against wrong by confidence. Answers changed: share of answerable items whose answer differs from the previous loop.

Panel of Figure 10 — Figure 10: Accuracy of SanSi after every loop, by distance from the training data (10,027, 2,116, 1,871 and 5,809 items; line: mean of three seeds; band: one standard deviation). Horizontal lines: the single-pass models. Grey: Ouro-1.4B before fine-tuning.

Panel of Figure 11 — Figure 11: Accuracy of SanSi after every loop for each item type, over all test groups (1,089, 3,072, 1,558, 901, 1,368 and 1,808 items; line: mean of three seeds; band: one standard deviation). Horizontal lines: the single-pass models.

Loops	changed	wrong $\to$ right	right $\to$ wrong	changed	wrong $\to$ right	right $\to$ wrong
	Every loop trained ( SanSi )			Last loop only
1 $\to$ 2	29.2 $\pm$ 0.3	15.3 $\pm$ 0.8	6.8 $\pm$ 0.2	58.4 $\pm$ 8.9	29.2 $\pm$ 5.5	11.7 $\pm$ 2.2
2 $\to$ 3	14.9 $\pm$ 0.4	7.6 $\pm$ 0.2	3.9 $\pm$ 0.1	31.5 $\pm$ 13.3	17.9 $\pm$ 7.9	6.0 $\pm$ 2.1
3 $\to$ 4	7.8 $\pm$ 0.4	3.6 $\pm$ 0.4	2.4 $\pm$ 0.1	15.4 $\pm$ 4.9	8.5 $\pm$ 3.2	3.4 $\pm$ 0.7
4 $\to$ 5	4.5 $\pm$ 0.3	1.9 $\pm$ 0.2	1.7 $\pm$ 0.1	8.2 $\pm$ 1.8	4.0 $\pm$ 0.8	2.3 $\pm$ 0.5
5 $\to$ 6	3.3 $\pm$ 0.3	1.4 $\pm$ 0.2	1.1 $\pm$ 0.2	5.3 $\pm$ 0.8	2.5 $\pm$ 0.5	1.6 $\pm$ 0.2

Appendix

Table 16: Share of answerable items (%) whose answer changes between consecutive loops: mean

\pm

standard deviation over three seeds.

	Since loop 1 (%)			Settled	Up to loop 8 (%)
Loop $T$	changed	fixed	broken	by $T$ (%)	already final	to be fixed	to be broken
1	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	56.7 $\pm$ 0.2	64.5 $\pm$ 0.8	21.0 $\pm$ 0.3	7.3 $\pm$ 0.3
2	29.2 $\pm$ 0.3	15.3 $\pm$ 0.8	6.8 $\pm$ 0.2	74.2 $\pm$ 1.2	78.7 $\pm$ 0.7	11.1 $\pm$ 0.7	5.9 $\pm$ 0.2
3	33.2 $\pm$ 0.4	19.0 $\pm$ 0.8	6.9 $\pm$ 0.1	83.6 $\pm$ 1.1	85.8 $\pm$ 0.8	6.4 $\pm$ 0.7	4.8 $\pm$ 0.2
4	34.6 $\pm$ 0.4	20.5 $\pm$ 0.5	7.1 $\pm$ 0.1	89.1 $\pm$ 0.9	90.2 $\pm$ 0.9	4.0 $\pm$ 0.5	3.6 $\pm$ 0.4
5	35.0 $\pm$ 0.5	20.8 $\pm$ 0.3	7.2 $\pm$ 0.2	92.5 $\pm$ 0.8	93.1 $\pm$ 0.7	2.7 $\pm$ 0.4	2.6 $\pm$ 0.3

Appendix

Table 17: How the answers of SanSi move over the loops (9,645 answerable test items; mean

\pm

standard deviation over three seeds). Since loop 1: share of items whose answer at loop

T

differs from that at loop 1, and the shares that went from wrong to right (fixed) and from right to wrong (broken). Settled by

T

: the answer does not change from loop

T

on. Up to loop 8: share of items whose answer at loop

T

is already that of loop 8, and the shares that the later loops will still fix or break.

Settles at loop	Items (%)	Right at loop 8 (%)	Confidence at loop 8
1	56.7 $\pm$ 0.2	82.2 $\pm$ 1.1	.894 $\pm$ .014
2	17.5 $\pm$ 0.9	68.7 $\pm$ 1.2	.795 $\pm$ .020
3	9.4 $\pm$ 0.2	60.1 $\pm$ 0.5	.714 $\pm$ .017
4	5.5 $\pm$ 0.3	50.7 $\pm$ 3.3	.645 $\pm$ .031
5	3.4 $\pm$ 0.2	43.6 $\pm$ 4.3	.581 $\pm$ .032
6	2.8 $\pm$ 0.3	44.1 $\pm$ 2.5	.540 $\pm$ .032

Appendix

Table 18: Items by the loop at which the answer of SanSi settles (9,645 answerable test items; mean

\pm

standard deviation over three seeds). Shading of the accuracy column: darker is more often right.

Single-pass	Items	Settles at loop	Never changes	Accuracy (%) at loop
models right	(%)	(mean)	(%)	1	8
3	45.5 $\pm$ 0.1	1.41 $\pm$ 0.02	80.8 $\pm$ 0.6	86.3 $\pm$ 0.9	95.0 $\pm$ 0.2
2	22.9 $\pm$ 0.4	2.39 $\pm$ 0.06	41.4 $\pm$ 1.5	52.3 $\pm$ 1.6	77.4 $\pm$ 1.5
1	16.0 $\pm$ 0.3	3.01 $\pm$ 0.14	27.5 $\pm$ 1.9	28.0 $\pm$ 0.6	47.1 $\pm$ 2.0
0	15.6 $\pm$ 0.3	2.80 $\pm$ 0.10	39.0 $\pm$ 2.1	13.8 $\pm$ 0.1	19.8 $\pm$ 2.3

Appendix

Table 19: Items by the number of single-pass models that answer them correctly (9,645 answerable test items; mean

\pm

standard deviation over three seeds). Never changes: SanSi keeps the answer of loop 1 through all eight loops.

	Share	Mean confidence at loop
Items (single gold answer)	(%)	1	2	4	8
Never changed, right at loop 8	48.0 $\pm$ 0.5	.805 $\pm$ .019	.883 $\pm$ .013	.918 $\pm$ .013	.924 $\pm$ .011
Never changed, wrong at loop 8	9.5 $\pm$ 0.7	.669 $\pm$ .031	.756 $\pm$ .022	.779 $\pm$ .024	.782 $\pm$ .022
Changed at least once, right at loop 8	24.7 $\pm$ 0.4	.567 $\pm$ .024	.647 $\pm$ .015	.734 $\pm$ .018	.763 $\pm$ .017
Changed at least once, wrong at loop 8	17.7 $\pm$ 0.3	.530 $\pm$ .023	.559 $\pm$ .020	.575 $\pm$ .020	.595 $\pm$ .021

Appendix

Table 20: SanSi : four groups of items by whether the answer ever changed after loop 1 and whether it is right at loop 8, with their mean confidence (mean

\pm

standard deviation over three seeds).

ECE (answerable items) $\downarrow$
Difference (first $-$ second)	First	Second	Difference [95% interval]
SanSi loop 3 $-$ loop 1	0.082	0.105	$-0.023$ [ $-0.032$ , $-0.013$ ]
SanSi loop 8 $-$ loop 3	0.093	0.082	$+0.010$ [ $+0.005$ , $+0.016$ ]
SanSi loop 1 $-$ Ouro one loop	0.105	0.137	$-0.032$ [ $-0.038$ , $-0.027$ ]
SanSi loop 8 $-$ Ouro one loop	0.093	0.137	$-0.045$ [ $-0.054$ , $-0.034$ ]
SanSi loop 8 $-$ SmolLM2-1.7B	0.093	0.069	$+0.024$ [ $+0.015$ , $+0.033$ ]

Appendix

Table 21: Differences in the probability metrics with 95% bootstrap intervals (means of three seeds; the statistic is recomputed on every resample of the groups of related test items). First, second: the two values that are compared. Colours: teal , the first term is significantly better (taking the direction of the metric into account; the interval excludes 0); red , significantly worse; grey , the interval includes 0.

Loop	Accuracy (%)	Mean confidence	Confidence $-$ accuracy	ECE
1	57.9 $\pm$ 0.7	.683 $\pm$ .022	$+0.105$	.105 $\pm$ .020
2	66.3 $\pm$ 0.6	.750 $\pm$ .015	$+0.086$	.088 $\pm$ .009
3	70.0 $\pm$ 0.6	.780 $\pm$ .016	$+0.080$	.082 $\pm$ .010
4	71.2 $\pm$ 0.6	.793 $\pm$ .016	$+0.081$	.085 $\pm$ .009
5	71.4 $\pm$ 0.7	.798 $\pm$ .016	$+0.084$	.087 $\pm$ .008
6	71.7 $\pm$ 0.7	.802 $\pm$ .016	$+0.085$	.089 $\pm$ .010

Appendix

Table 22: SanSi after every loop on the 9,645 answerable test items: accuracy, mean confidence, their difference, and the ECE (mean

\pm

standard deviation over three seeds).

Panel of Table 23 — Table 23: Probability quality: means of three seeds, with the standard deviation over the seeds on the line below (the Jev API is a single run). Overconf.: mean confidence minus accuracy. Conf. errors: wrong with top probability $\geq 0.8$ . TV crowd: total variation distance to the crowd distribution (766 items).

Panel of Table 24 — Table 24: Four models as trained and with one temperature fitted on the development split, the last step of Kev’s recipe (10,027 test items; mean $\pm$ standard deviation over three seeds; fitted temperature: mean of the seeds).

Panel of Figure 12 — Figure 12: ECE of SanSi after every loop, by distance from the training data (answerable items; line: mean of three seeds; band: one standard deviation). Horizontal lines: the single-pass models, whose standard deviations are in Table 23 ; the lines of the two Qwen3.5 models nearly coincide in (a) and (b).

Panel of Table 25 — Table 25: Unanswerable items by group (number of items in parentheses) and evidence AUROC: mean $\pm$ standard deviation over three seeds. The Jev API (a single run) was not trained to give a uniform distribution on such items.

$k$	Item	Answer
Liar chains (options: yes, no)
1	Lee is honest. Wes is honest. According to Flo, Lee tells the truth. According to Abe, Wes lies. Does Abe tell the truth?	no
2	Uma always tells the truth. Seth always tells the truth. Fred says that Seth lies. Kurt says Fred is a liar. Bert says that Uma lies. Ben says Bert is a liar. Is Ben telling the truth?	yes
4	Omar is honest. Ege always lies. According to Bert, Omar tells the truth. Yves says that Eli lies. Ben says that Bert lies. Liv says that Nate lies. Eli says that Liv tells the truth. Jon says Vera is honest. Vera says that Ben lies. According to Nate, Ege tells the truth. Is Yves telling the truth?	no
8	Omar is a liar. Seth is a liar. Vera says Iris is a liar. Bea says Flo is a liar. According to Eli, Wade tells the truth. According to Yul, Ida tells the truth. Nia says that Nate lies. According to Bert, Omar lies. Ida says that Nia lies. Iris says Bert is honest. According to Wade, Meg lies. Nate says Vera is honest. According to Meg, Bo tells the truth. Ben says that Seth tells the truth. According to Kim, Bea tells the truth. Flo says Ben is a liar. According to Ola, Yul lies. Bo says that Kim lies. Is Eli telling the truth?	no
Object swaps (options: the five people)

Appendix

Table 26: One test item of each depth-controlled task at depths

k=1,2,4,8

(the item of median length at that depth). A liar chain contains a second chain of the same length that does not matter for the question; an object-swap item contains

k

swaps of the queried object and

k

further swaps.

Liar chains (chance 50%)
	Holds	Accuracy (%)		Difference to Qwen3.5-4B (points)
Model	depth $k$	$k\leq 8$	$k>8$	$k\leq 8$	$k>8$
SanSi , loop 1	3	69.0	49.9	$-5.3$ [ $-7.3$ , $-3.4$ ]	$-0.2$ [ $-1.6$ , $+1.2$ ]
SanSi , loop 2	6	88.0	51.6	$+13.7$ [ $+11.7$ , $+15.6$ ]	$+1.6$ [ $-1.1$ , $+4.3$ ]
SanSi , loop 4	11	96.6	70.9	$+22.3$ [ $+20.5$ , $+24.1$ ]	$+20.8$ [ $+18.9$ , $+22.9$ ]
SanSi , loop 8	11	97.0	72.6	$+22.7$ [ $+20.8$ , $+24.7$ ]	$+22.6$ [ $+20.5$ , $+24.7$ ]

Appendix

Table 27: The two depth-controlled tasks in summary (means of three seeds; 120 test items per depth). Holds depth

k

: the accuracy is at least 75% at every depth up to

k

k\leq 8

: depths seen in training;

k>8

: unseen depths. Differences with 95% bootstrap intervals; loop 16 is beyond the trained loops. Differences: teal , SanSi is significantly better (the interval excludes 0); red , significantly worse; grey , the interval includes 0.

Liar chains (chance 50%)
	Depths seen in training								Unseen depths
$k$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
SanSi , loop 1	100.0	98.6	79.2	60.0	59.4	52.8	50.6	51.4	48.6	50.6	51.7	50.6	50.8	48.3	49.2	49.2
	$\pm$ 0.0	$\pm$ 1.3	$\pm$ 9.8	$\pm$ 11.0	$\pm$ 3.4	$\pm$ 8.0	$\pm$ 7.1	$\pm$ 1.7	$\pm$ 3.8	$\pm$ 1.7	$\pm$ 4.4	$\pm$ 1.3	$\pm$ 2.5	$\pm$ 1.4	$\pm$ 2.2	$\pm$ 1.4
SanSi , loop 2	100.0	100.0	99.7	96.7	91.4	80.8	69.7	65.6	55.8	51.7	51.1	52.5	52.5	50.6	51.1	47.5
	$\pm$ 0.0	$\pm$ 0.0	$\pm$ 0.5	$\pm$ 2.2	$\pm$ 2.7	$\pm$ 6.0	$\pm$ 7.7	$\pm$ 5.1	$\pm$ 3.6	$\pm$ 3.6	$\pm$ 4.3	$\pm$ 0.8	$\pm$ 3.8	$\pm$ 5.7	$\pm$ 1.9	$\pm$ 3.0

Appendix

Table 28: Accuracy (%) at every depth

k

on the two depth-controlled tasks (120 test items per depth; means of three seeds, with the standard deviation over the seeds on the line below). Qwen3.5-4B makes a single pass. Cells are shaded by the accuracy above chance (blue: SanSi ; green: Qwen3.5-4B).

Liar chains
$k$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Settles at loop	1.0	1.0	1.2	1.5	1.6	1.9	2.1	2.3	2.7	2.9	3.1	3.4	3.9	4.0	4.2	4.2
Changed, loop 1 to 8 (%)	0	1	21	40	41	47	50	52	54	52	52	54	55	56	53	52
Object swaps
Settles at loop	1.0	1.1	1.4	1.9	2.0	2.3	2.7	3.0	3.0	3.3	3.5	3.6	4.0	4.1	4.0	4.5
Changed, loop 1 to 8 (%)	1	3	32	53	51	65	59	70	64	68	66	66	68	64	64	64

Appendix

Table 29: SanSi on the depth-controlled tasks, by depth

k

: mean loop at which the answer settles, and share of items whose answer at loop 8 differs from that at loop 1 (means of three seeds). Darker cells: later settling and more changed answers.

SmolLM2-1.7B, recipe of Section 3 (2,000 steps)
$k$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Seed 0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0
Seed 1	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0
SmolLM2-1.7B, half the learning rate and 4,000 steps
Seed 0	50.0	49.2	54.2	52.5	48.3	49.2	57.5	53.3	49.2	52.5	46.7	55.0	55.8	50.0	46.7	45.0
Seed 1	51.7	52.5	57.5	54.2	52.5	50.8	59.2	50.0	46.7	46.7	54.2	54.2	49.2	50.0	46.7	52.5

Appendix

Table 30: Single-pass models of SanSi ’s size on the liar chains: accuracy (%) at every depth

k

for each seed (120 test items per depth; chance is 50%).

Liar chains (chance 50%)
$k$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Accuracy (%)	100.0	86.4	62.5	56.9	58.6	44.7	48.6	47.8	44.7	48.3	51.1	53.3	47.5	43.9	42.5	51.1
Mean confidence	.99	.82	.69	.66	.63	.63	.61	.62	.62	.60	.61	.61	.60	.59	.60	.60
Object swaps (chance 20%)
Accuracy (%)	77.5	42.5	23.6	20.6	24.2	25.0	19.4	21.4	18.6	20.3	26.9	24.2	21.4	31.1	23.3	27.8
Mean confidence	.92	.69	.53	.47	.42	.39	.37	.36	.32	.33	.31	.31	.31	.29	.29	.29

Appendix

Table 31: The Jev API on the two depth-controlled tasks: accuracy and mean confidence at every depth

k

(1,920 items per task, three calls per item). Cells are shaded by the accuracy above chance.

	Accuracy (%) at loop				Accuracy at loop 8			ECE		Hard	Evid.	AUROC
Training	1	2	4	8	In	Near	Far	all	near	ans. (%)	AUROC	r/w
Loss (cross-entropy + Brier)	58.4	66.9	71.6	72.0	86.6	67.7	68.0	.093	.149	17.5	.935	.795
	$\pm$ 0.7	$\pm$ 0.6	$\pm$ 0.5	$\pm$ 0.7	$\pm$ 0.6	$\pm$ 2.7	$\pm$ 0.1	$\pm$ .012	$\pm$ .031	$\pm$ 1.3	$\pm$ .004	$\pm$ .005
Reinforcement learning	58.1	66.5	71.3	71.7	86.1	66.8	67.9	.076	.129	21.6	.919	.793
	$\pm$ 0.2	$\pm$ 0.4	$\pm$ 0.4	$\pm$ 0.1	$\pm$ 0.1	$\pm$ 2.7	$\pm$ 0.7	$\pm$ .015	$\pm$ .047	$\pm$ 4.5	$\pm$ .008	$\pm$ .002

Appendix

Table 32: SanSi trained with the supervised loss of Equation 3 and with reinforcement learning (Equation 8 ): 10,027 test items, means of three seeds with the standard deviation over the seeds on the line below. ECE, hard-answer rate and AUROC are those of loop 8.

Accuracy (%)
Difference (first $-$ second)	First	Second	Difference [95% interval]
SanSi : loop 8 $-$ loop 4	72.0	71.6	$+0.4$ [ $-0.0$ , $+0.7$ ]
SanSi : loop 12 $-$ loop 8	71.2	72.0	$-0.8$ [ $-1.1$ , $-0.5$ ]
SanSi : loop 16 $-$ loop 8	70.1	72.0	$-1.8$ [ $-2.2$ , $-1.5$ ]
Four-loop model: loop 8 $-$ loop 4	69.6	70.8	$-1.3$ [ $-1.7$ , $-0.9$ ]
Four-loop model $-$ SanSi , both at loop 4	70.8	71.6	$-0.8$ [ $-1.2$ , $-0.4$ ]

Appendix

Table 33: The number of loops: differences with 95% bootstrap intervals (means of three seeds), and the share of answers that change, are fixed and are broken between two loops. Colours: teal , the first term is significantly better (taking the direction of the metric into account; the interval excludes 0); red , significantly worse; grey , the interval includes 0.

	SanSi (8 trained loops)		Trained with 4 loops
Loop	Acc. (%)	ECE	Acc. (%)	ECE
1	58.4 $\pm$ 0.7	.105 $\pm$ .020	59.7 $\pm$ 0.5	.123 $\pm$ .010
2	66.9 $\pm$ 0.6	.088 $\pm$ .009	67.6 $\pm$ 0.7	.096 $\pm$ .005
3	70.4 $\pm$ 0.6	.082 $\pm$ .010	70.4 $\pm$ 0.5	.093 $\pm$ .007
4	71.6 $\pm$ 0.5	.085 $\pm$ .009	70.8 $\pm$ 0.4	.094 $\pm$ .005
5	71.9 $\pm$ 0.7	.087 $\pm$ .008	70.9 $\pm$ 0.4 †	.103 $\pm$ .009

Appendix

Table 34: Accuracy and ECE at every loop of SanSi , run for 16 loops, and of the model trained with four loops, run for eight (10,027 test items; mean

\pm

standard deviation over three seeds). † : loop beyond the trained ones. Grey values: loops beyond the trained ones.

Panel of Figure 13 — Figure 13: Accuracy of SanSi at every depth $k$ after every loop on the two depth-controlled tasks, up to 16 loops (white: chance; means of three seeds). Dashed: the depths and the loops seen in training. Black line: the depth that the model holds. Table 27 gives the summary.

	Accuracy (%) at loop
Model	1	4	8
SmolLM2-1.7B, not fine-tuned	38.8	–	–
+ loop as in Ouro	38.8	32.0	31.5
SmolLM2-1.7B, fine-tuned	58.4 $\pm$ 0.7	–	–
+ loop as in Ouro	35.4 $\pm$ 4.2	32.5 $\pm$ 0.6	33.5 $\pm$ 0.4
+ loop through a linear map	58.1 $\pm$ 1.1	58.3 $\pm$ 1.0	58.3 $\pm$ 1.0

Appendix

Table 35: A loop added to SmolLM2-1.7B after pre-training (10,027 test items; mean

\pm

standard deviation over the seeds: three for SmolLM2-1.7B and SanSi , two for the fine-tuned rows with an added loop; the untuned rows are single runs). “As in Ouro”: the previous state replaces the token embeddings. “Through a linear map”: the previous state is added to the token embeddings through a rank-64 map that is zero at initialisation.

	Accuracy (%)					ECE			Confidence		AUROC	Evid.	Hard	Answers
Loop	All	In-dist.	Near	Far	JevB.	In-dist.	Near	Far	right	wrong	r/w	AUROC	(%)	changed (%)
1	62.4	78.8	61.9	56.6	60.6	.032	.081	.118	.789	.596	.756	.839	31.5	–
	$\pm$ 0.4	$\pm$ 0.5	$\pm$ 1.7	$\pm$ 0.8	$\pm$ 0.7	$\pm$ .009	$\pm$ .022	$\pm$ .014	$\pm$ .011	$\pm$ .014	$\pm$ .004	$\pm$ .006	$\pm$ 3.5
2	71.8	85.2	70.0	67.6	70.7	.049	.103	.078	.852	.650	.780	.914	22.6	27.5
	$\pm$ 0.7	$\pm$ 0.2	$\pm$ 3.0	$\pm$ 0.2	$\pm$ 1.7	$\pm$ .002	$\pm$ .023	$\pm$ .010	$\pm$ .008	$\pm$ .014	$\pm$ .003	$\pm$ .007	$\pm$ 3.0	$\pm$ 0.9
3	74.7	87.8	72.5	70.6	74.6	.049	.117	.073	.873	.660	.795	.935	18.1	12.2

Appendix

Table 36: SanSi -2.6B after every loop (10,027 test items; means of three seeds, with the standard deviation over the seeds on the line below). Columns as in Table 15 .

All test items; no calibration data
	Acc.	ECE $\downarrow$
	(%)	all	near	far
Loop 8	72.0 $\pm$ 0.7	.093 $\pm$ .012	.149 $\pm$ .031	.092 $\pm$ .019
Mean of loops 1–8	71.8 $\pm$ 0.7	.044 $\pm$ .009	.084 $\pm$ .028	.045 $\pm$ .010
Temperature fitted on in-distribution items
Loop 8	70.2 $\pm$ 0.7	.098 $\pm$ .013	.151 $\pm$ .032	.092 $\pm$ .019

Appendix

Table 37: Averaging the option probabilities of the eight loops, and temperature scaling. In the two lower blocks a random half of the groups of related items in the in-distribution test set, or in the whole test set, serves as calibration data, and the rows are evaluated on the remaining test items (means of ten splits). All numbers are mean

\pm

standard deviation over three seeds; temp.: temperature scaling. Shaded rows: the average of the eight loops.

Reward	F1 after training	Change in F1 [95% interval]	Reward AUROC, first 20 steps
Not trained	39.5	–	–
SanSi read at loop 1	29.1 $\pm$ 2.4	$-10.5$ [ $-11.9$ , $-9.0$ ]	.779 $\pm$ .050
SanSi read at loop 2	39.5 $\pm$ 1.8	$-0.0$ [ $-1.6$ , $+1.6$ ]	.857 $\pm$ .030
SanSi read at loop 4	45.8 $\pm$ 4.2	$+6.3$ [ $+4.6$ , $+8.0$ ]	.914 $\pm$ .032
SanSi read at loop 8	47.3 $\pm$ 1.1	$+7.7$ [ $+6.1$ , $+9.4$ ]	.905 $\pm$ .025
Loop 8 $-$ loop 4	–	$+1.4$ [ $+0.8$ , $+2.0$ ]	–

Appendix

Table 38: The verifier case study in summary (3,000 test questions; mean

\pm

standard deviation over three training seeds). Change in F1: against the generator before training, with a 95% bootstrap interval. Reward AUROC: how well the reward separates the sampled answers that match the gold answer from those that do not, in the first 20 training steps. Colours: teal , the first term is significantly better (the interval excludes 0); red , significantly worse; grey , the interval includes 0.

	All questions			F1 by question type				Answer
Reward	F1	EM	Contains	Comparison	Bridge comparison	Compositional	Inference	words
Not trained	39.5	32.5	34.5	50.6	48.2	30.7	28.6	2.6
SanSi read at loop 1	29.1 $\pm$ 2.4	9.6 $\pm$ 3.0	38.5 $\pm$ 1.0	34.0 $\pm$ 3.2	32.7 $\pm$ 4.9	28.6 $\pm$ 2.8	21.0 $\pm$ 2.7	5.5 $\pm$ 0.5
SanSi read at loop 2	39.5 $\pm$ 1.8	19.6 $\pm$ 5.4	45.0 $\pm$ 1.9	38.9 $\pm$ 2.9	42.4 $\pm$ 7.5	36.3 $\pm$ 1.5	40.5 $\pm$ 3.4	4.3 $\pm$ 0.5
SanSi read at loop 4	45.8 $\pm$ 4.2	29.4 $\pm$ 10.7	49.7 $\pm$ 0.9	49.2 $\pm$ 8.2	48.5 $\pm$ 6.2	37.3 $\pm$ 1.9	48.4 $\pm$ 1.1	3.9 $\pm$ 0.8
SanSi read at loop 8	47.3 $\pm$ 1.1	32.2 $\pm$ 3.8	49.7 $\pm$ 2.0	49.4 $\pm$ 5.1	51.6 $\pm$ 0.7	38.3 $\pm$ 1.5	49.7 $\pm$ 0.9	3.6 $\pm$ 0.3

Appendix

Table 39: The generator on the 3,000 test questions of 2WikiMultiHopQA (750 per question type) after 400 GRPO steps, by the loop at which the rewarding SanSi is read. Mean

\pm

standard deviation over three training seeds (Table 40 lists them); the untrained generator is a single run. Contains: share of answers in which the gold answer occurs as a sequence of whole words, after normalisation. Answer words: mean length of the generated answer.

Figure 14: F1 of the generator before training, and its change after training, by question type and by the loop at which SanSi is read (3,000 test questions; means of three seeds).

	F1			EM			Contains			Answer words
Reward, seed	0	1	2	0	1	2	0	1	2	0	1	2
SanSi read at loop 1	27.4	27.9	31.9	8.2	7.5	13.0	37.5	38.4	39.5	5.9	5.6	4.9
SanSi read at loop 2	40.2	37.4	40.9	24.0	13.6	21.3	43.0	45.4	46.7	3.9	4.8	4.3
SanSi read at loop 4	48.3	48.2	41.0	36.3	34.8	17.1	48.9	49.5	50.7	3.4	3.5	4.7
SanSi read at loop 8	48.5	46.5	46.8	36.5	30.7	29.3	48.7	48.3	52.0	3.4	3.5	4.0

Appendix

Table 40: The verifier case study seed by seed: the generator on the 3,000 test questions after 400 GRPO steps. Before training it has F1 39.5, EM 32.5, 34.5% of answers that contain the gold answer, and 2.6 words per answer.

Figure 15: F1 on the 500 development questions during the training of the generator, by the loop at which the verifier is read (thick lines: means of three training seeds; thin lines: the seeds).

Share of the items (%)
SanSi wrong at loop 8	27.2
Qwen3.5-4B wrong	25.7
SmolLM2-1.7B wrong	41.1
SanSi and Qwen3.5-4B both wrong	18.5
Only SanSi wrong	8.7
Only Qwen3.5-4B wrong	7.2

Appendix

Table 41: Error analysis on the 8,879 test items with one gold option (three seeds of every model). Confidence: the probability of the chosen option.

			SanSi
	Item (source)	Gold	loop 1	loop 8	Qwen3.5-4B	SmolLM2
Fixed by the loops	“No road is dustless. Some streets are roads.” Is “Some streets are dustless.” true, false, or unknown? (FOLIO)	unknown	true (.83)	unknown (.97)	unknown (.89)	true (.57)
Fixed; larger model wrong	If 30,000 is divided by 10 and then divided by 10 again, what will be the resulting number? 3 / 30 / 300 / 3,000 (MMLU)	300	3 (.60)	300 (.91)	3,000 (.87)	300 (.38)
Missing knowledge	When cold temperatures are produced in a chemical reaction, the reaction is known as … (ARC)	endothermic	exothermic (.96)	exothermic (.99)	endothermic (.97)	exothermic (.48)
Broken by the loops	“By 9000 BP, Europe was fully forested.” Does the sentence contain the answer to “When was Europe fully forested and recovered from the last Ice Age?” (QNLI)	yes	yes (.98)	no (.89)	no (.77)	yes (.96)
Shared error	“He can’t be here.” Hypothesis: “He is here.” (WANLI)	neutral	contradiction (.99)	contradiction (.96)	contradiction (.99)	contradiction (.71)

Appendix

Table 42: Five test items, all from sources not seen in training, with the answers of SanSi after its first and its last loop and of two single-pass models. Each cell gives the answer ( right , wrong ) and, in parentheses, its probability (seed 0).

Explore similar work

Sep 20, 2026cs.CL

this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent

Software delegates more of its branches to models every year: which queue a ticket enters, whether a command is safe to run, whether a claim clears without a person. What the program needs back is not prose. It is one of n declared options and a number it can threshold. Today that costs a round trip to a frontier model -- hundreds of milliseconds, a per-token bill, and a parser -- for a question that is usually a conjunction of three clauses. this-that-model-1.0 is a 2B-parameter typed decision model. Its answer is read directly from the hidden state at a designated position and restricted to the option set the caller declared, so no text is generated, nothing can be malformed, and every question in a request is answered in the same forward pass. It decides in 30.9 ms on one laptop GPU and generates zero output tokens doing it, where a frontier API call costs 8758 ms and the hosted systems that answer these questions well spend between 21 and 212 generated tokens per question thinking first, billed for every one. It sustains 32 decisions per second on one consumer GPU and never lets the state leave the machine. On a third party's recorded cohort of 68 decision questions, on their inputs and their wording, it scores 0.941 with a Brier score of 0.042, against 0.765 and 0.133 for the hosted service Jev on the same items. One pass of our 42-family internal suite takes 32 seconds and 0.000217 USD of electricity; the most accurate hosted model we measured needs 155.2 minutes and 10.636 USD. We also report where it loses. On multi-step arithmetic, which a single forward pass cannot carry intermediate results through, it scores 0.560 against their 0.98 to 1.00, and a targeted second training round improved the five task families it was written for and transferred to none of the other 13. The model is open-sourced in https://huggingface.co/flock-io/this-that-model-1.0

Zehua Cheng, Wei Dai, Jiahao Sun

University of Oxford

Frontier Models Decisions Large Reasoning Models Application Programming Interfaces Delegation

Ask PDF

Sep 29, 2026cs.CL

StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate thirteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-51% of the time to outdated answers, a fast model 34% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use.

Jhen-Ke Lin, Chung Chun Wang

National Yang Ming Chiao Tung University

Large Language Model Decisions Large Language Model Judges Dialogue Benchmarks Decisions Language Model Computes Real-Time Systems+2

Ask PDF

Sep 29, 2026cs.LG

Benchmarking System One decision models against trained classifiers and language models for automated decision gates

Software that hands branching decisions to a model needs a declared option and a probability it can threshold. Typed decision models, also called System One models, return such probabilities without generating text, while supervised classifiers and generative language models are the established alternatives. One harness sends eight decision-model checkpoints from six families, including the hosted model Jev, and four open generative models from three developers the same semantic requests, and scores trained and zero-shot classifiers on the same workflow, intent, emotion and social-science items. With task labels, a fine-tuned DeBERTa-v3-large has the highest observed accuracy on every labeled benchmark but one. Without labels, no decision model is significantly more accurate than Jev on workflows or intents, but Gemma-4-31B matches it on workflows and exceeds it on CLINC-150 at higher cost and latency. Stated probabilities of generative models become unreadable when replies miss the key format, whereas key likelihoods avoid this but can saturate. A guaranteed 5 percent risk leaves Jev 0.528 of the intent decisions, and an in-scope threshold still accepts 0.310 of out-of-scope requests. On typed-decisions, swapping yes and no flips 50.5 answers per hundred for Jev and at least 16.8 for every generative model tested, against at most 6.5 for four fine-tuned decision checkpoints. Exposure to a benchmark's training data explains the largest lead of an open checkpoint, which vanishes on rater-labeled emotions. An intent-trained first stage escalating to Gemma-4-31B reaches that model's accuracy at about Jev's price. The results yield condition-dependent design rules for automated decision gates.

Amir Rafe, Subasish Das

Large Language Model Decisions

Ask PDF