VX-One: Typed, Calibrated Decision Heads for Real-Time Perimeter Triage

Juan S. Santillana

cs.CROct 21, 2026

VX-One: Typed, Calibrated Decision Heads for Real-Time Perimeter Triage

Authors: Juan S. Santillana

Organizations: Independent research

Abstract

Perimeter triage faces a latency/semantics trade-off: signature engines are fast and format-safe but blind to unseen attacks, while generative models are semantic but slow and unparseable. We present VX-One, a small multi-head network (100K--10M parameters) with calibrated typed outputs: a binary gate, a closed-set choice with an explicit unknown option where the benchmark supports it (and abstention elsewhere), and an ordinal score. Heads are temperature-calibrated, with 0.7--1.7 ms single-event CPU p50 latency. We evaluate UNSW-NB15, CIC-IDS2017, and EMBER2024 against calibrated XGBoost, Random Forest, MLP, and Kitsune baselines, and measure a Suricata-to-VX-One cascade over five days of CIC-IDS2017. Results are deliberately double-sided: on UNSW the gate ties the best baseline (0.9837+/-0.0002 versus MLP 0.9835+/-0.0013 across three seeds), while VX-One wins temporal CIC detection (0.734+/-0.022 versus 0.639+/-0.027) but loses multi-class attribution to XGBoost. Under leave-one-family-out, the gate detects all nine UNSW families but drops below chance on CIC recon; CIC trees transfer better. Suricata resolves only about 0.1% of attacks on flow reconstructions and 0.2--2.2% on payload-bearing PCAPs; the strictly by-day Phase 2 recovers only the in-scope DoS/DDoS share on the wave day (recall 0.34, precision 0.97). Calibration collapses under temporal drift (ECE 0.04->0.40) for every method while ranking partly survives. We contribute a measured account of where typed calibrated decisions help perimeter triage, and where they fail.

Figures & tables

Dataset	Model	AUROC	FPR@95	macro-F1
UNSW-NB15	VX Small	0.9834	0.1211	0.4571
	XGBoost	0.9818	0.1369	0.4968
	RF	0.9792	0.1399	0.4378
	MLP	0.9821	0.1279	0.4445
	Kitsune	0.6086	0.5617	—
CIC stratified b	VX Small	0.9999	0.0003	0.8826

Table 1. Test-set comparison (calibrated). VX: VX-One at three sizes. FPR@95 is FPR@95%TPR. All rows now carry three seeds (42/43/44); the table shows seed 42 and the text reports seed means. Seed

\sigma

of the gate AUROC is

\leq

0.0013 on UNSW and EMBER, 0.017–0.027 on by-day CIC, and 0 on saturated CIC. With three seeds VX is 0.9837

\pm

0.0002 on UNSW vs MLP 0.9835

\pm

0.0013, XGBoost 0.9821

\pm

0.0003, RF 0.9793

\pm

0.0001: the in-distribution gate is a tie with MLP and a small but consistent edge over the trees. Row-paired bootstrap tests (

B{=}300

, same resampled rows for both models) are reported in the text. † = under temporal drift (its by-day macro-F1 column is seed-unstable: VX spread 0.24–0.45 across three seeds, and XGBoost’s 0.8336 is a single-seed value on the same unstable metric — Section 5.1 ); b = detection plateau of saturation (all models

\approx

1.0). Full per-dataset table, including ECE columns and Kitsune/RF/MLP rows, in the released results.

Excluded family	gate AUROC	FPR@95	abst. (%)	$n$ test
analysis	0.9992	0.0002	92.32	677
backdoor	0.9989	0.0028	63.46	583
dos	0.9941	0.0202	2.76	4,089
exploitation	0.9550	0.4177	8.19	11,132
fuzzing	0.8519	0.3899	28.41	6,062
generic	0.9957	0.0128	49.96	18,871

Table 2. LOAO—unseen-family detection (gate) and attribution honesty (choice abstention). The control—macro-F1 on known families—stays at 0.38–0.56 across folds, i.e., the family that is excluded does not degrade attribution of the others. Baseline LOAO runs (calibrated XGBoost/RF/MLP plus logistic and Mahalanobis OOD detectors) are summarized in the text and released in full. UNSW Small.

Figure 1. Abstention sweep over

\tau_{\mathrm{abs}}\in\{0.05,\ldots,0.99\}

, mean over LOAO folds, on excluded-family test rows. UNSW: an operating frontier exists—abstention and false attack attribution cross near

\tau{=}0.5

, and large

\tau

recovers honesty (97% abstain,

<

1% false attribution). CIC: no operating point—unseen-family events are absorbed into normal (

\sim

94%) at saturated confidence (median max-prob

1.000

; median margin

1.000

), so no post-hoc threshold or margin policy can recover them. Two line panels of per-fold mean abstention rate and false-attack-attribution rate as the abstention threshold sweeps from 0.05 to 0.99. On UNSW the two curves cross near 0.5 and high thresholds yield 97 percent abstention at under 1 percent false attribution. On CIC the abstention curve stays near zero across the sweep and about 94 percent of unseen-family rows are attributed to the normal class at saturated confidence.

Excluded family	gate AUROC	FPR@95	abst. (%)	$n$ test
botnet	0.7499	0.4038	0.00	295
bruteforce	0.9674	0.0778	0.05	2,076
dos_ddos	0.8524	0.3394	0.00	57,103
exploitation	0.9703	0.0297	0.00	1 †
infiltration	0.5329	0.8262	0.00	5 †
recon	0.4259	0.8685	0.00	23,840

Table 3. LOAO on stratified CIC (Small). Softmax confidence saturates: the fixed threshold is practically non-selective, and unseen families are predominantly attributed to normal with high confidence. † = degenerate folds (

n{=}1

and

n{=}5

test rows: the AUROC is a coin toss and is quoted for completeness only).

Scale	p50 ( $\mu$ s)	p95 ( $\mu$ s)	p99 ( $\mu$ s)	max ( $\mu$ s)
Nano	675	715	773	853
Small	955	999	1,022	1,059
Base	1,652	1,795	2,082	2,335

Table 4. E4 latency (single-event, batch=1, 1 thread, CPU). Well within a

<

10 ms design budget with margin

\geq 5\times

Day	flows	attack	PH1 acc	PH2 acc	comb acc	comb P	comb R
Mon 03	529,450	0	0.9989	0.9996	0.9985	—	—
Tue 04	445,641	13,832	0.9681	0.9994	0.9985	0.958	0.996
Wed 05	691,390	251,722	0.6359	0.9986	0.9979	0.997	0.997
Thu 06	458,483	2,216	0.9942	0.9971	0.9961	0.552	0.9779
Fri 07	702,713	288,785	0.5887	0.9979	0.9974	0.998	0.996

Table 5. E5 cascade over five days of CIC-IDS2017. “PH1 acc” is Phase 1 (Suricata) accuracy; “PH2 acc” is Phase 2 (VX-One) accuracy on that day’s test rows; “comb” is the estimated combined cascade; P/R = precision/recall of the combined cascade. PH2/comb use the stratified-split (in-distribution) Phase 2; the by-day Phase 2 re-estimation on the wave day is in the text (P 0.971 / R 0.339) and Mon–Thu are not computable under it. Stats are estimates with a documented independence assumption (Section 6 ).

Run	Effect tested	AUROC	m-F1	ECE-ch
UNSW ref	—	0.9834	0.4571	0.0496
UNSW a1	no text branch	0.9716	0.4079	0.0509
UNSW a2	single-heads	0.9784	0.4598	0.0552
UNSW a3	no calibration	0.9834	0.4571	0.0980
UNSW a4	no class tilt	0.9838	0.4545	0.0488
UNSW a6	no dropout	0.9839	0.4507	0.0451

Table 6. Ablations (ref = corresponding Small reference run).

Primitive / metric	Jev	MS-Dec.-1	Ours
Gate AUROC	0.568	0.544	0.981
Gate FPR@TPR95	0.932	0.872	0.157
Gate ECE15	0.265	0.118	0.097
Choice macro-F1	0.069	0.066	0.534
Choice accuracy	—	0.323	0.711
Severity MAE	0.809	0.969	0.333

Table 7. E7: commercial System One baselines, zero-shot, same frozen state and rows as our heads (2,021 stratified UNSW test rows, seed 42). Two independent vendors of the primitive triad.

Network Intrusion Detection

Network intrusion detection identifies malicious or abnormal network activity to help protect computers, connected devices, and critical infrastructure.

See all 168 papers

Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents
Sep 24, 2026Joas Antonio dos Santos BarbosaProbability Calibration Cybersecurity
Independent Researcher — AI and Offensive Security S˜ao Paulo, SP, Brazil
Benchmarking System One decision models against trained classifiers and language models for automated decision gates
Sep 29, 2026Amir Rafe, Subasish DasLLM Decision-Making LLM Evaluation
Texas State University, 601 University Drive, San Marcos, 78666, Texas, USA
SecJev: Bringing Security Expertise to System One Decision Models
Oct 2, 2026Zheng Chen, Fei Yu, Haohao Huang +3LLMs for Cybersecurity Cybersecurity
University of Electronic Science and Technology of China · National Key Laboratory of Security Communication

Sep 24, 2026J/K move · Enter open · S save

Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents

Joas Antonio dos Santos Barbosa

Independent Researcher — AI and Offensive Security S˜ao Paulo, SP, Brazil

Probability Calibration Cybersecurity LLM Agent Security Automated Penetration Testing

Open paper AskPDF arXiv

Autonomous penetration-testing harnesses use large language models (LLMs) for reconnaissance, exploitation, and reporting, but often rely on those same models to confirm findings, grade severity, and select agents. This can lead to false positives, inflated severity, and wasted compute. We examine how System One decision models, lightweight non-generative classifiers that return typed, calibrated verdicts, can support these decisions. We make five contributions. First, we define four decision points: finding adjudication, severity recalibration, agent pruning, and confirmation loops. Second, we present an exploratory NeuroSploit case study comparing one run with TypeSafe System One (Jev) and one without it against a web target containing 13 vulnerabilities. Differences in severity distribution, runtime, and grading by exposed data type motivate the architecture but do not establish statistical significance. Third, we review published specifications for Jev, Jev-Ultrafast, and the open-source Laya without assuming that results from other benchmarks transfer to penetration testing. Fourth, we discuss RLHF, RLAIF, RLCD, and RLHV as training approaches and their implications for trust in security decisions. Finally, we propose Rave, a domain-adapted System One model, and outline its training data, evaluation protocol, and potential effect on harness assurance.

Abstract

Network Intrusion Detection

Explore similar work

Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents

Benchmarking System One decision models against trained classifiers and language models for automated decision gates

SecJev: Bringing Security Expertise to System One Decision Models