System X: An LLM Decision Model That Replies from Its Own KV Cache

Syed Usama Khalid Bukhari; Syed Safwan Khalid; Muhammad Aamir

cs.AIOct 18, 2026

System X: An LLM Decision Model That Replies from Its Own KV Cache

Authors: Syed Usama Khalid Bukhari, Syed Safwan Khalid, Muhammad Aamir

Organizations: Turing Logic LTD · University of Oxford

Abstract

Decision models return probabilities over supplied options, but a deployed agent may also need to reply in words. The present study introduces System X, a large language model (LLM) decision model that adds a text-reply option to the LLM-as-Jev readout. When this option wins, or confidence is low, the same model continues from the decision's key-value (KV) cache, avoiding a repeated prompt prefill. Qwen3-0.6B has been fine-tuned with LoRA on Reason-9k and Bitext reply rows. JevBench accuracy shows no statistically significant change (55.4% to 54.5%, p=0.87), while expected calibration error falls from 0.292 to 0.169. Held-out Bitext routing accuracy reaches 98.3%. Cache reuse saves 15-37% of latency on an Apple M4 Pro, with token-identical replies in the FP32 demonstration cases. Scoring options on copies of the prompt cache matches full-sequence scoring to within 1.4e-6 in probability. Adding dialogue, industry and complaint data (System X+) improves support-task results but lowers IFEval accuracy from 55.5% to 45.0%. System X v4 uses Qwen3-1.7B, rebalanced data and teacher-rewritten reply targets. Against its base LLM, JevBench accuracy rises from 55.0% to 61.0% (p=0.04), with no statistically significant IFEval change (p=0.71). A matched latency test on unadapted Qwen3-1.7B saves 17-39%. One LLM judge prefers v4 replies to an unadapted 1.7B second call (31 to 11 pairs, p=0.003); the other's preference is not significant. These reply comparisons also differ in prompts and generation limits. State conflicts and unsupported reply details remain failure modes. Further studies with several seeds, human raters and real support data are recommended.

Figures & tables

Panel of Figure 1 — Figure 1: System X decides and replies from one prefill. The prompt is read once; every option’s suffix is scored on copies of its KV cache, which stays untouched. A confident action returns without text. If REPLY_WITH_TEXT (index $r$ ) wins or the top probability falls below $\tau$ , decoding continues from the same cache, so only r] Reply: and the reply tokens are processed. A two-call pipeline instead re-reads $x$ before replying (orange, dashed); that second prefill is the latency the method saves.

	Ours		Paper
	Base LLM	System X	Base LLM	Reason-9k	Best JevBench run
JevBench accuracy (%)	55.4	54.5	56.7	55.0	61.0
JevBench ECE	0.292	0.169	0.278	0.118	0.121
JevBench Brier	0.674	0.575	–	–	–
Banking77 accuracy (%)	18.5	53.5	22.2	60.4	59.0
Banking77 ECE	0.460	0.079	–	–	–

Table 1: Decisions on public benchmarks with Qwen3-0.6B. “Paper” columns are from LLM-as-Jev [ 24 ] . Their Reason-9k row uses full fine-tuning. Their best JevBench row uses LoRA on Reason+Long-12k with

\lambda=0.01

. System X uses LoRA on Reason-9k plus reply-task rows; these are different training conditions.

Set	Metric	Base LLM	System X
Held-out Bitext	accuracy (%)	58.7	98.3
	action / info / not offered	17.3 / 100 / 100	98.7 / 98.1 / 97.8
	ECE	0.389	0.020
	Brier	0.778	0.025
	reply ROUGE-L F1	0.099	0.231
	share answered with a reply (%)	91.0	49.3

Table 2: The reply task. Held-out: 300 Bitext messages never seen in training. Hand-written: 97 out-of-distribution cases. Accuracy by kind is measured before fallback. ROUGE-L uses only gold-reply rows on which a reply was generated (150 base-LLM rows and 147 System X rows), so the evaluated subsets differ.

Prompt tokens	Decision only	System X	Two calls	Saved
367	134 ms	414 ms	488 ms	74 ms (15%)
679	215 ms	508 ms	639 ms	131 ms (21%)
1,159	350 ms	671 ms	901 ms	230 ms (26%)
2,159	654 ms	1,019 ms	1,494 ms	476 ms (32%)
4,161	1,386 ms	1,824 ms	2,891 ms	1,067 ms (37%)

Table 3: End-to-end latency of decision plus a 32-token reply, Apple M4 Pro, bf16, median of 5 runs. Both paths use the same generate() loop; “two calls” re-reads the prompt for the reply.

Figure 2: Latency of cache reuse (Apple M4 Pro, bf16, median of 5 runs). (a) End-to-end time for a decision plus a 32-token reply with Qwen3-0.6B (Table 3 ); the shaded gap is the second prefill that System X avoids. (b) Relative saving for Qwen3-0.6B and for the untrained Qwen3-1.7B backbone (§ 9 ), at the prompt lengths measured for each.

	Accuracy (%)			ECE
Set	Base LLM	System X	System X+	Base LLM	System X	System X+
JevBench (231)	55.4	54.5	52.4	0.292	0.169	0.201
Banking77 (3,080)	18.5	53.5	57.2	0.460	0.079	0.081
Held-out Bitext (300)	58.7	98.3	99.0	0.389	0.020	0.019
Hand-written (97)	47.4	78.4	76.3	0.491	0.124	0.139
ABCD next step (300)	52.3	51.7	88.7	0.420	0.270	0.037

Table 4: Decisions of the base LLM, System X and System X+ (argmax, before fallback). JevBench in FP32; all other sets in bf16. ABCD, Bitext verticals and CFPB are held out from System X+’s training sources and absent from the other two fine-tuning mixtures; base-model pretraining overlap was not assessed.

Set	Check	Base LLM	System X	System X+
Bitext verticals	fact-copy check	2.4	28.8	46.4
	unsupported identifier	0.0	2.5	0.7
	instruction echo	66.4	17.4	2.6
ABCD	instruction echo	62.8	26.2	0.6
Held-out Bitext	instruction echo	65.6	20.3	16.2
	template slot	2.6	12.8	16.9

Table 5: Automatic reply checks (% of replies written). Fact copying is measured only on generated replies whose references contain a qualifying State fact (84, 80 and 84 Bitext-vertical rows for the three models). The evaluated subsets differ.

Model	R@1	R@5	R@10	MRR
Base LLM	7.2	18.2	28.8	0.148
System X	8.0	23.0	32.8	0.166
System X+	15.0	36.2	49.4	0.263
ABCD BERT-base	21.7	46.6	58.7	–
ABCD RoBERTa-Large	22.0	47.8	59.1	–

Table 6: ABCD utterance ranking among 100 candidates (500 test turns, PMI scoring, RTX 5090, bf16). The ABCD rows are from Chen et al. [10] , Table 4: retrieval models trained for this task and evaluated on the full test set, averaged over three seeds.

Source	Helpful	Grounded	Clean	Send (%)	System X+ W / T / L	Sign test
Base LLM	1.44	2.93	2.53	9.3	101 / 35 / 13	$p<0.001$
System X	2.27	3.83	3.89	22.7	60 / 41 / 49	$p=0.34$
System X+	2.41	4.41	4.34	20.7	–
Two-call Qwen3-0.6B	1.73	4.32	2.99	9.3	98 / 29 / 23	$p<0.001$
Two-call Qwen3-1.7B	3.43	4.26	4.83	43.3	32 / 30 / 88	$p<0.001$
Reference	3.51	4.33	4.45	38.7	35 / 21 / 94	$p<0.001$

Table 7: LLM-judge results on 150 reply rows (Gemma 4 31B). Left: mean scores (1–5) and share the judge would send. Right: System X+’s pairwise wins, ties (including order flips) and losses against each source, with exact two-sided sign tests. The base-LLM pairwise comparison has 149 usable rows: both judgments failed on one row. All other pairwise comparisons have 150.

Model	GSM8K	IFEval	TriviaQA	LAMBADA	Avg.	WikiText ppl
Paper: base LLM	44.4	55.0	4.0	39.0	35.6	27.52
Base LLM	42.4	55.5	12.0	39.6	37.4	26.13
System X	46.4	52.0	12.7	40.4	37.9	24.57
System X+	40.0	45.0	14.3	39.8	34.8	24.12

Table 8: General capabilities (accuracy in %, WikiText-2 word perplexity). “Paper” is the untrained Qwen3-0.6B row of LLM-as-Jev (its Table 9). GSM8K is strict match; IFEval is prompt-level strict accuracy. Avg. is the mean of the four accuracies.

	Qwen3-0.6B			Qwen3-1.7B
Set	Base LLM	System X+	System X v4-0.6B	Base LLM	System X v4
JevBench (231)	55.4 (.292)	52.4 (.201)	51.5 (.175)	55.0 (.404)	61.0 ( .123 )
Banking77 (3,080)	18.5 (.460)	57.2 (.081)	55.5 (.056)	28.3 (.617)	65.2 (.062)
Hand-written (97)	47.4 (.491)	76.3 (.139)	84.5 (.115)	72.2 (.251)	86.6 ( .070 )
Held-out Bitext (300)	58.7 (.389)	99.0 (.019)	96.3 (.022)	73.3 (.235)	97.3 (.025)
ABCD next step (300)	52.3 (.420)	88.7 (.037)	88.3 (.035)	48.3 (.483)	86.0 (.048)

Table 9: Decisions (argmax accuracy in %, ECE in parentheses). System X+ and the 0.6B base LLM were measured on the M4 Pro, and the other columns on the RTX 5090. JevBench in FP32; all other sets in bf16.

Figure 3: System X v4 against its untrained Qwen3-1.7B base LLM on every decision set (Table 9 ; argmax before fallback). Accuracy rises and calibration error falls on all seven sets. JevBench in FP32; all other sets in bf16.

Source	Helpful	Grounded	Clean	Send (%)	v4 W / T / L	Sign test
System X+	2.40	4.40	4.33	20.0	99 / 26 / 25	$p<0.001$
System X v4-0.6B	2.93	4.35	4.65	38.0	78 / 39 / 33	$p<0.001$
1.7B base (cache)	2.96	4.33	4.27	40.0	84 / 33 / 33	$p<0.001$
Two-call Qwen3-1.7B	3.45	4.26	4.83	44.0	66 / 36 / 48	$p=0.11$
Reference	3.49	4.31	4.43	36.7	79 / 27 / 44	$p=0.002$
System X v4	3.92	4.33	4.86	56.7	–

Table 10: Gemma judge on 150 reply rows with System X v4 as the focus system: mean scores (1–5), share judged sendable, and System X v4’s pairwise wins / ties (including order flips) / losses with exact two-sided sign tests. The 1.7B two-call baseline is the same unadapted Qwen3-1.7B reply call as in Table 7 .

Figure 4: Pairwise reply comparisons with System X v4 as the focus system: shares of pairs it wins, ties (including order flips) and loses, with counts inside the bars and exact two-sided sign tests on the right. Top: Gemma on 150 rows per opponent (Table 10 ). Bottom: Claude on its blind subset of 25 or 50 pairs per opponent. The vertical line marks 50%.

Figure 5: Support-task gains against instruction following. Each model is placed by its held-out ABCD next-step accuracy (Tables 4 and 9 ) and IFEval prompt-level strict accuracy (Table 8 and § 9 ); arrows run from each untrained base LLM to the models fine-tuned from it. At 0.6B, the support gains of System X+ and System X v4-0.6B come with lower IFEval scores. At 1.7B, System X v4 reaches similar ABCD accuracy with an IFEval change that is not statistically significant.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

System1-Arena: Which Decisions Can Be Left to System 1?
Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-Making LLM Evaluation
The University of Texas at Austin
Releasing the Inner Jev: A Simple Recipe for Open Decision Models
Oct 12, 2026Jonathan Geuter, Andrew Cheng, David Alvarez-MelisLLM Decision-Making LLM Fine-Tuning
Harvard University · Kempner Institute
Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
Oct 1, 2026Jiawei LiLLM Agent Evaluation LLM Decision-Making

Oct 15, 2026J/K move · Enter open · S save

System1-Arena: Which Decisions Can Be Left to System 1?

Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang+1

The University of Texas at Austin

LLM Decision-Making LLM Evaluation Efficient Language Model Inference

Open paper AskPDF arXiv

Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.

Abstract

LLM Decision-Making

Explore similar work

System1-Arena: Which Decisions Can Be Left to System 1?

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses