SecJev: Bringing Security Expertise to System One Decision Models

Zheng Chen; Fei Yu; Haohao Huang; Yang Li; Anlong Chen; Lei Chen

cs.CROct 2, 2026

SecJev: Bringing Security Expertise to System One Decision Models

Authors: Zheng Chen, Fei Yu, Haohao Huang, Yang Li, Anlong Chen, Lei Chen

Organizations: University of Electronic Science and Technology of China · National Key Laboratory of Security Communication

Abstract

Security workflows need models that turn complex observations and explicit policies into decisions. System One models introduced by Jev return typed predictions and probabilities; security specialization supplies the domain expertise behind those predictions. We introduce SecJev, to our knowledge the first family of Jev-style decision models specialized for security, spanning 0.8B to 9B parameters. Built on Kev's single-pass candidate scorer, SecJev learns Boolean, choice, and ordered decisions from text, telemetry, and observation histories. We develop SecJev-Corpus to unify source-label prediction and explicit-policy evaluation across 14 tasks and eight sources. It covers tool outputs, traffic, federated updates, consensus, authentication, and vehicle messages. Scene-weighted training adapts the models across these domains while preserving a shared typed decision interface. Security specialization improves every model in the family; SecJev-0.8B outperforms general Kev-9B by 20.51 percentage points in task-macro accuracy. At 0.8B and 4B, SecJev achieves accuracy and latency close to answer-only generative fine-tuning with lower peak allocated GPU memory. SecJev-4B improves on Kev-4B and security-trained generation in prompt-injection and traffic transfer tests. We release adapters, decision heads, SecJev-Corpus, and training and inference code.

Figures & tables

Figure 1: SecJev’s shared decision path and security adaptation. Source annotations and explicit policies provide requests and targets. One backbone with layerwise LoRA adapters produces decision-marker and candidate-end states; the pointer head scores candidates to obtain typed outputs. Solid edges carry forward computation during both training and inference. Gray dashed edges supply gold labels and logits to scene-weighted cross-entropy; coral dashed edges update only LoRA and the head, leaving the backbone frozen. General-decision initialization uses official Kev for 0.8B/4B/9B and our pretrained 2B model.

Source	Decision task	Type	Questions
CICIoT2023	Attack-capture label	Boolean	48,000
ToN-IoT	Connection attack label	Boolean	40,000
Twins / Streamlet	Observed candidate-voter conflict	Boolean	7,397
	Candidate quorum	Boolean	7,397
	Certificate policy action	Choice	7,397
	Evidence level	Ordered	7,397

Table 1: Corpus coverage across all splits. Several tasks can share a scene; the 17,000 client-update questions instantiate one task.

Split	Scenes	Questions
Training	81,410	118,818
Calibration	9,333	14,821
Development	9,911	14,912
Test	14,670	21,634
Total	115,324	170,185

Table 2: Corpus partition. Calibration fits temperature; checkpoint selection is specified in Appendix A.1 .

			0.8B		2B		4B		9B
Task	$n$	Maj.	Init.	SecJev	Init.	SecJev	Init.	SecJev	Init.	SecJev
AgentDojo: boundary	426	77.46	51.88	80.28	40.85	92.96	54.23	99.53	61.97	98.83
AgentDojo: injection	426	77.46	32.86	79.81	29.81	92.72	73.00	98.36	64.55	98.83
ByzFL: client update	800	80.00	79.62	92.62	80.00	97.50	79.88	95.88	80.25	93.88
CICIoT2023: attack	6,000	75.00	25.00	91.80	25.00	92.13	26.28	91.72	37.93	90.27
InjecAgent: operation	124	56.45	44.35	100.00	51.61	98.39	62.90	100.00	57.26	100.00

Table 3: Full security-test accuracy (%). Maj. predicts the training-majority semantic class. Init. denotes official Kev for 0.8B/4B/9B and our general-decision model for 2B. Checkpoint choices are given in Appendix A.1 . Macro weights tasks equally; balanced macro also balances semantic classes.

Size	Method	Macro acc.	Short ms	Long ms	P95 ms	Q/s	GiB
0.8B	SecJev	90.46	79.2	164.1	256.0	7.32	2.56
0.8B	SecGen	90.01	80.4	167.4	260.1	7.22	3.52
0.8B	Direct-Qwen pointer	88.12	80.4	164.6	256.7	7.27	2.56
4B	SecJev	94.48	109.1	578.8	890.5	2.54	10.18
4B	SecGen	94.34	116.9	599.9	910.6	2.47	12.58
9B	Kev	69.79	169.1	861.0	1278.5	1.73	17.11

Table 4: Matched accuracy and inference cost. Accuracy (%) is re-evaluated at common precision. Short/long latencies are medians; P95 is the pooled 95th-percentile latency. Q/s is mean serial throughput at batch size one, and GiB is peak allocated GPU memory. Each pass measures 100 short and 100 long requests.

Dataset	Model	$n$	Accuracy	Precision	Recall	FPR
BIPIA	Kev-4B	1000	71.30	64.14	96.60	54.00
BIPIA	SecJev-4B	1000	83.60	75.38	99.80	32.60
BIPIA	SecGen-4B	1000	79.60	71.39	98.80	39.60
IoT-23	Kev-4B	1000	73.00	65.54	97.00	51.00
IoT-23	SecJev-4B	1000	90.20	86.95	94.60	14.20
IoT-23	SecGen-4B	1000	84.30	76.34	99.40	30.80

Table 5: Transfer performance of 4B models on BIPIA and IoT-23 (%), with 1,000 test examples per dataset. SecGen-4B is the answer-only generative baseline. Injected content and malicious traffic are the positive classes for precision, recall, and false-positive rate (FPR).

Change	SecJev size	Pairs	Original acc.	Changed acc.	Consistency	Both correct
Paraphrase	0.8B	1400	88.64	87.79	95.36	85.93
	2B	1400	90.79	89.86	96.57	88.79
	4B	1400	91.64	91.14	98.36	90.57
	9B	1400	90.71	90.64	96.43	88.93
Choice permutation	0.8B	500	83.00	82.60	98.80	82.20
	2B	500	84.80	84.00	97.40	83.40

Table 6: Paired semantic consistency (%). Original/changed accuracy uses the same eligible questions: 1,400 for paraphrasing and 500 for permutation. Consistency is semantic answer agreement; both-correct requires both predictions to match the label. Tasks have equal weight within each condition.

0.8B initialization	Dev. bal.	Test macro	Test bal.
Kev	90.36	90.34	88.00
Qwen3.5 directly	84.85	88.17	83.75

Table 7: Initialization comparison (%). Dev./test bal. are task-macro balanced accuracy. Both initializations select epoch 3 on development data. Test results cover all 21,634 questions.

	General decision		General transfer
Size	Init.	SecJev	Init.	SecJev
0.8B	83.50	83.08	68.45	66.01
2B	84.00	84.50	75.91	75.91
4B	87.08	86.58	83.69	83.84
9B	87.50	87.42	85.06	84.60

Table 8: General-task accuracy (%) before and after specialization: 1,200 general decision and 656 general transfer questions. Init. denotes the model before security training. Suite definitions appear in Section 6.3 .

Size	Probabilities	NLL	Brier	ECE (%)
0.8B	Raw	0.435	0.211	2.18
0.8B	Scaled	0.412	0.213	3.34
2B	Raw	0.399	0.191	1.80
2B	Scaled	0.385	0.194	3.28
4B	Raw	0.358	0.173	1.84
4B	Scaled	0.344	0.177	3.56

Table 9: Security-test confidence metrics, averaged over questions. Lower is better for all three. ECE uses 15 equal-width bins. Scaling uses only the held-out calibration split.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Scope	Result
Schema and label validity	170,185 questions	No violations found
Scene duplication and split isolation	115,324 scenes and source groups	No violations found under the recorded grouping rules
Independent policy recomputation	37,490 LANL/Twins questions	0 inconsistent answers
Semantic-class distribution	Every task and split	Maximum class share 80%. Maximum cross-split gap 8.58 pp
Source-record recheck	50 original records; 20 retained ByzFL records	No mismatches in the checked fields or label mappings
Independent human review	280 questions; 20 per task	261 answer agreements (93.21%); 180/180 tool-text and policy answers match labels

Appendix

Table 10: Corpus construction checks, source-record rechecks, and independent human review.

			Matches corpus label		Evidence sufficient
Task	Questions	A–C agree	A	C	A	C
AgentDojo: boundary action	20	20	20	20	20	19
AgentDojo: injection	20	20	20	20	20	20
CICIoT: attack	20	20	14	14	6	3
ByzFL: update attack	20	17	17	18	4	0
InjecAgent: operation	20	20	20	20	20	20

Appendix

Table 11: Independent human review of 280 questions. Entries are counts. A–C agreement compares the reviewers’ answers; label matches compare each answer with the corpus target. Evidence sufficiency is a separate judgment.

Task	0.8B	2B	4B	9B
AgentDojo: boundary action	78 / 81 / 79	91 / 90 / 88	99 / 99 / 99	97 / 97 / 98
AgentDojo: injection	85 / 86 / –	88 / 91 / –	95 / 95 / –	98 / 98 / –
CICIoT: attack	93 / 94 / –	94 / 93 / –	93 / 94 / –	88 / 95 / –
ByzFL: update attack	91 / 85 / –	95 / 94 / –	88 / 84 / –	83 / 85 / –
InjecAgent: operation	100 / 98 / 100	98 / 94 / 98	100 / 100 / 100	100 / 99 / 100
LANL: logon action	100 / 100 / 100	100 / 100 / 100	100 / 100 / 100	99 / 100 / 99

Appendix

Table 12: Accuracy by task (%): original / paraphrase / permutation, with 100 questions per applicable condition. A dash means permutation is not evaluated for that answer type.

Task	Questions	Kev initialization	Direct-Qwen initialization
AgentDojo: boundary action	426	80.28	82.86
AgentDojo: injection	426	79.81	71.60
ByzFL: update attack	800	92.62	95.38
CICIoT: attack	6000	91.80	91.47
InjecAgent: operation	124	100.00	96.77
LANL: logon action	540	98.89	97.78

Appendix

Table 13: Per-task test accuracy for the development-selected 0.8B checkpoints (%).

Size	Security task-macro	General decision micro	General transfer micro
0.8B	$+39.40$ $[+30.16,+43.48]$	$-0.42$ $[-1.42,+0.58]$	$-2.44$ $[-4.73,-0.15]$
2B	$+40.83$ $[+28.62,+45.32]$	$+0.50$ $[-0.50,+1.42]$	$+0.00$ $[-1.98,+1.98]$
4B	$+26.63$ $[+18.59,+30.75]$	$-0.50$ $[-1.17,+0.17]$	$+0.15$ $[-1.07,+1.37]$
9B	$+24.26$ $[+15.38,+28.40]$	$-0.08$ $[-0.58,+0.42]$	$-0.46$ $[-2.29,+1.37]$

Appendix

Table 14: SecJev minus initialization, in percentage points, with paired 95% group-bootstrap intervals. Positive differences favor SecJev. These are intervals for accuracy differences, not for absolute scores.

				Raw ( $T=1$ )			Stored temperature
Suite	Size	Model	$T$	NLL	Brier	ECE	NLL	Brier	ECE
General decision	0.8B	Init.	2.406	0.652	0.268	9.94	0.449	0.240	2.69
General decision	0.8B	SecJev	1.353	0.946	0.293	12.86	0.734	0.281	11.26
General decision	2B	Init.	1.000	0.574	0.241	9.22	0.574	0.241	9.22
General decision	2B	SecJev	1.292	0.749	0.249	9.88	0.616	0.241	8.48
General decision	4B	Init.	2.144	0.468	0.207	7.32	0.360	0.189	2.60

Appendix

Table 15: General-suite confidence before and after security specialization. Lower is better; ECE is in percent. No temperature is fitted on these general test subsets. Init. is Kev at 0.8B/4B/9B and the general-decision starting point at 2B.

Model	Epoch	Micro	Macro	Bal. macro
SecGen-0.8B	3	89.40	91.90	89.46
Cls-0.8B	3	89.09	90.76	88.39
SecGen-4B	3	91.15	94.11	92.35

Appendix

Table 16: Development accuracy (%) for selected SecGen and candidate-sequence classification (Cls) checkpoints. Balanced macro groups recall by semantic answer label.

Size	Method	Test micro	Test macro	Test balanced macro
0.8B	SecJev (Kev initialization)	85.03	90.34	88.00
0.8B	Direct-Qwen pointer	81.96	88.17	83.75
0.8B	SecGen	84.75	90.02	87.79
0.8B	Candidate-sequence classification	84.39	89.85	86.84
4B	SecJev (Kev initialization)	88.01	94.51	92.42
4B	SecGen	87.95	94.35	92.30

Appendix

Table 17: FP32 test accuracy for development-selected models (%). All select epoch 3. These scores use FP32; Table 4 uses a BF16 backbone.

		SecGen-0.8B		Cls-0.8B		SecGen-4B
Task	Questions	Acc.	Bal.	Acc.	Bal.	Acc.	Bal.
AgentDojo: boundary action	426	76.29	80.63	84.27	80.62	96.71	94.92
AgentDojo: injection	426	79.11	80.24	75.35	74.49	97.89	96.42
ByzFL: update attack	800	95.62	93.05	94.00	88.28	97.25	96.17
CICIoT: attack	6000	91.30	86.44	92.07	89.33	91.70	87.29
InjecAgent: operation	124	99.19	99.07	100.00	100.00	100.00	100.00

Appendix

Table 18: Full-test results for selected SecGen and classification (Cls) checkpoints (%). Balanced accuracy averages recall across semantic answer labels, independent of candidate order.

Size	Method	Micro	Macro	Semantic balanced macro
0.8B	SecJev	85.14	90.46	88.04
0.8B	SecGen	84.70	90.01	87.76
0.8B	Direct-Qwen pointer	81.85	88.12	83.72
4B	SecJev	87.94	94.48	92.38
4B	SecGen	87.95	94.34	92.26
9B	Kev	50.87	69.79	70.32

Appendix

Table 19: Full-test accuracy (%) recomputed with the serial BF16-backbone timing implementation. Each row contains 21,634 unique questions across 14 tasks.

Size	Method	Short median range (ms)	Long median range (ms)	Batch-one Q/s
0.8B	SecJev	79.0–79.7	163.5–164.4	$7.319\pm 0.016$
0.8B	SecGen	80.2–80.7	166.8–167.6	$7.220\pm 0.007$
0.8B	Direct-Qwen pointer	80.3–80.7	164.5–164.7	$7.272\pm 0.006$
4B	SecJev	107.4–110.3	577.6–579.2	$2.537\pm 0.001$
4B	SecGen	112.9–135.2	597.5–600.3	$2.470\pm 0.026$
9B	Kev	168.8–169.4	859.3–862.0	$1.726\pm 0.003$

Appendix

Table 20: Variation over five timing passes. Ranges describe per-pass medians, not confidence intervals. Serial batch-one throughput is mean

\pm

sample standard deviation over the same 200-request workload.

LLMs for Cybersecurity

Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.

See all 254 papers

Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation
Sep 27, 2026Yixuan LiuProbability Calibration Selective Prediction
TraceStone and Nanyang Technological University, Singapore
Evaluating and Benchmarking the System One Model Jev
Sep 29, 2026Tobias Deußer, Lorenz Sparrenberg, Rafet SifaMultilingual Language Model Evaluation LLM Evaluation
University of Bonn, Bonn, Germany · Lamarr-Institute for Machine Learning and Artificial Intelligence, Bonn, Germany · Fraunhofer IAIS, Sankt Augustin, Germany
Chinese-Jev: Bringing System One Model to Chinese-Language Tasks
Sep 29, 2026Zexiao Wang, Zihao Zhang, Xudong Wang +5LLM Decision-Making Efficient Language Model Inference
Fudan University · National University of Singapore · University of Chinese Academy of Sciences

Sep 27, 2026J/K move · Enter open · S save

Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

Yixuan Liu

TraceStone and Nanyang Technological University, Singapore

Probability Calibration Selective Prediction AI Agent Evaluation AI Agent Security Prompt Injection Detection LLM Agent Reliability

Open paper AskPDF arXiv

Agentic software connects language models to tools that modify files and interact with external services. Developers use model-based security checks to screen external content and user requests before agents act. System One models answer developer-defined questions with probabilities over predefined answers such as safe or unsafe, but classification accuracy alone does not establish whether these probabilities support reliable automation. We evaluate Jev, Laya, Decider, and Bespoke Nimble against specialized classifiers and language-model judges across prompt-injection detection, interaction-risk judgment, and harmful-request screening, examining decision accuracy, calibration, and selective automation. (1) High overall accuracy and low average calibration error can conceal attacks classified as safe with high confidence within particular groups. Developers should test candidate models on the intended security task and examine errors within relevant attack groups. (2) Policies selected under strict miss limits allow few test inputs automatically, and choosing allow and block thresholds separately increases automation mainly through additional blocks. Developers should verify limits on missed unsafe inputs and blocked benign inputs on independent data, and report the allowed and blocked fractions separately. (3) Comparing predictions on the same inputs shows that a language-model judge can detect unsafe inputs missed by a System One model, but can also repeat the System One model's confident mistakes and falsely flag benign inputs. Developers should test which missed unsafe inputs their review rule forwards, then measure the review model's misses and benign false alarms on those inputs.

Abstract

LLMs for Cybersecurity

Explore similar work

Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

Evaluating and Benchmarking the System One Model Jev

Chinese-Jev: Bringing System One Model to Chinese-Language Tasks