Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

Jiawei Li

cs.AIOct 1, 2026

Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

Authors: Jiawei Li

Abstract

Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection. System-1 decision models answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LLM calls. We present a paired evaluation of an open-weight (Laya) and a hosted (Jev) System-1 model on 11 agent decision points built from 18 public sources: 7,283 base cases plus 6,640 robustness variants, with byte-identical inputs, paired tests, and cross-hardware and cross-day reproducibility checks. Jev is significantly more accurate on 9 of 11 decision points (+10.8 to +46.0 pp). Neither model beats chance on zero-shot model routing, and they tie on RAG relevance gating. Laya changes 30% of its answers when the option order is reversed and degrades sharply with many or similar candidates (31% at 50 nearest-neighbour tools, vs. 98% for Jev on items with a unique correct tool). We also audit our own pipeline. Three analysis errors and one design confound distorted headline deployment claims: an omitted pre-screen cost (reported 23.9% saving, actual 4.3%), gate accuracy reported as end-to-end quality (58% vs. 98%), in-sample thresholds (5% target, up to 17% held-out misses), and a "channel effect" on injection false positives that vanishes with channel-native content. Two other suspected confounds did not change the conclusions. All cases, raw outputs and analysis code are available at https://github.com/David-DL-Space/sys1-eval.

Figures & tables

Decision point	Source(s)	$n$	Type
A Model routing	RouterBench	400	ch(2)
B Tool selection	BFCL	400	ch( $K$ )
C Tool-call guard	BFCL + irrelevance	300	y/n
D RAG relevance	BEIR $\times$ 5 + BM25	1,920	y/n
E Injection guard	deepset, jackhhao	400	y/n
F Large label set	BANKING77, MASSIVE	200	ch(60+)

Table 1: Decision points. B2, G2 and J2 are the control experiments of § 7 . S adds a policy-labelled gold standard for § 6 .

K\in\{2,5,10,20,50\}

with 80 cases per value. ch = choice, y/n = yes/no.

Figure 1: Accuracy per decision point. Right: paired difference Jev

-

Laya (pp);

p<0.001

unless stated.

Figure 2: Tool selection vs. number of candidates. Solid: random distractors (track B). Dashed: nearest-neighbour distractors (B2).

Figure 3: (a) Share of decisions that change when only the option order is reversed (

n=1{,}000

). (b) Accuracy of each track under 4 instruction wordings

\times

2 case samples (600 cases each); dots are the 8 runs.

Figure 4: (a) RAG gate: share of LLM calls kept vs. answerable queries dropped (in-sample). (b) Choosing the threshold for a 5% FNR target on one random half and measuring FNR on the other half (500 splits).

Figure 5: Two-tier safety review ( Jev pre-screen

\rightarrow

LLM reviewer): cost vs. end-to-end miss rate over thresholds

\tau

. Dashed: the original accounting, which omitted the pre-screen’s own cost.

Original claim	After audit	Root cause
Errors that changed a recommendation
Pre-screen at $\tau{=}0.1$ saves 23.9% of safety-review cost	Saves 4.3%; $\tau{=}0.05$ costs 4.7% more	Pre-screen token cost omitted from the two-tier total
RAG gate at 5% FNR: “end-to-end correctness 45–58%”	97.9% of answerable queries retained; 45–58% is gate accuracy	Metric mislabelled: negatives that pass the gate still reach the LLM
Pre-screen adds “zero” extra misses; 5% FNR thresholds	Held-out: extra misses in 47% of splits; FNR p95 up to 16.7%	Thresholds chosen and evaluated on the same data
Tool-output channel doubles/triples benign false positives; calibrate per channel	Holds only for transplanted user-style text; 0% $\to$ 0% ( Jev ) on native documents	Benign user queries pasted into tool outputs are anomalous in that channel
Errors in secondary numbers

Table 2: What a re-derivation from raw outputs and three control experiments changed.

Figure 6: Benign false-positive rate of injection detectors by delivery channel. (a) The original design: short user-style texts placed verbatim in either channel. (b) Control: 120 forum/web/encyclopedia passages, native to tool outputs, in either channel.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Laya			Jev
Track	AUC	FNR	FPR	AUC	FNR	FPR
C guardrail	.56	6.7	74.7	.86	2.7	40.0
D RAG (400)	.58	57.8	29.9	.70	42.2	37.2
E injection	.88	22.0	21.5	.98	7.0	7.0
G user turn	.90	29.2	8.3	.98	11.7	4.2
G tool output	.85	17.5	25.8	.97	5.8	10.0

Appendix

Table 3: Yes/no tracks at threshold 0.5 (FNR/FPR in %). For H the positive class is “supported”.

	B77	en	zh	ja	de	fr	es	ko	pt	ar	hi
Laya	40	30	30	60	30	60	40	20	30	0	0
Jev	84	80	50	100	90	70	90	90	80	80	50

Appendix

Table 4: Track F accuracy (%). BANKING77

n{=}100

; MASSIVE locales

n{=}10

each (trends only).

	Short (orig. G)		Documents (G2)
Model / channel	TPR	FPR	TPR	FPR
Laya user	70.8	8.3	47.5	6.7
Laya tool	82.5	25.8	50.8	10.0
Jev user	88.3	4.2	93.3	0.0
Jev tool	94.2	12.5	90.8	0.0
LLM ref. user	85.0	3.3	97.5	0.0

Appendix

Table 5: Channel control (%, day-2 runs). Benign FPR change user

\to

tool, paired McNemar: short texts Laya

p<10^{-4}

, Jev

p{=}0.002

, LLM

p<10^{-6}

; documents Laya

p{=}0.34

, Jev and LLM identical.

Negatives	Placeholder		Natural
	AUC	FPR	AUC	FPR
Laya	.41	39	.40	36
Jev	.81	66	.82	63
LLM ref.	.85	25	.88	12

Appendix

Table 6: PII control (J vs. J2). TPR on the shared positives: Laya 23%, Jev 85%, LLM 84%.

	Random (B)		Nearest (B2)
	all	unique	all	unique
$n$	400	385	400	281
Laya	84.0	84.9	54.0	61.6
Jev	99.5	100.0	84.5	98.9
LLM ref.	98.5	–	83.0	–

Appendix

Table 7: Tool selection (%). “Unique”: items whose LLM audit lists exactly the gold tool as valid.

Figure 7: Reliability diagrams (10 bins,

\geq

10 items per point).

	Call	FNR	FPR	Cost
LLM only	100	11.6	7.0	100.0
Jev $\tau{=}0.05$	75.2	11.6	7.0	104.7
Jev $\tau{=}0.1$	65.6	11.6	7.0	95.7
Jev $\tau{=}0.2$	58.9	12.0	5.1	87.8
Jev $\tau{=}0.3$	53.3	15.1	3.5	80.9
Laya $\tau{=}0.1$	99.3	11.6	7.0	99.4

Appendix

Table 8: Two-tier review, in-sample (%; cost relative to LLM-only, pre-screen cost included).

n{=}540

, 225 unsafe.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,139 papers

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent Harnesses LLM Agents
University of Illinois Urbana-Champaign · Princeton University · Google
System1-Arena: Which Decisions Can Be Left to System 1?
Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-Making LLM Evaluation
The University of Texas at Austin
Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents
Sep 24, 2026Joas Antonio dos Santos BarbosaProbability Calibration Cybersecurity
Independent Researcher — AI and Offensive Security S˜ao Paulo, SP, Brazil

Oct 15, 2026J/K move · Enter open · S save

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Kunlun Zhu, Tao Feng, Xu Han, Heng Wang+4

University of Illinois Urbana-Champaign · Princeton University · Google

LLM Agent Harnesses LLM Agents Tool Selection for Language Model Agents LLM Decision-Making AI Agent Benchmarks LLM Agent Training+1

Open paper AskPDF arXiv

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

Abstract

LLM Agent Evaluation

Explore similar work

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

System1-Arena: Which Decisions Can Be Left to System 1?

Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents