COGNIT-Guard: Calibrated Standalone Direct-Decision Guardrails with Heterogeneous CPU-NPU Confidence Cascading under Explicit Latency and False-Positive Constraints

Hao Chen

cs.CRSep 27, 2026

COGNIT-Guard: Calibrated Standalone Direct-Decision Guardrails with Heterogeneous CPU-NPU Confidence Cascading under Explicit Latency and False-Positive Constraints

Authors: Hao Chen

Organizations: School of Cyber Security, Guangdong Polytechnic Normal University, Guangzhou, China

Abstract

When must a foundation-model safety gateway generate tokens, and when should it directly output a calibrated decision? We study calibrated standalone direct-decision foundation models for real-time pre-ingestion safety guardrails, jointly addressing probability calibration, dual-use false-positive control, and heterogeneous CPU-NPU routing under explicit latency SLOs. Pre-ingestion guardrails must screen prompts prior to target-LLM prefill with low false alarms on benign compliance inquiries; however, shallow classifiers are brittle to phrasing shifts, hidden-state probes require coupling to a target LLM, and generative guards incur high decoding latency and dual-use false positives. We present COGNIT-Guard, coupling a validation-calibrated CPU fast gatekeeper with confidence-gated escalation to an NPU-resident 322M bidirectional direct-decision model (Laya-322M) under an asymmetric false-positive penalty. On the clean unseen DUCS-Bench test split ( $N=607$ ), COGNIT-Guard achieves 98.85% accuracy (McNemar $p = 1.19 \times 10^{-4}$ vs. ML), reduces benign FPR to 0.42% ( $1/238$ ; Fisher's exact $p = 8.23 \times 10^{-4}$ vs. ML), and attains 1.12% ECE and 0.0104 Brier score. On Huawei Ascend 910C NPUs, pure NPU inference runs in 21.77 ms mean latency (45.90 QPS), while the live serial CPU-NPU cascade ( $θ^*_{\mathrm{deploy}}=0.70$ ) achieves 41.63 ms mean latency (P50: 39.47 ms, 99.23% accuracy, 0.00% FPR). Evaluation on SafetyBench-ZH ( $N=2,100$ ) and comparison against a bi-encoder direct-decision baseline (CLM-8B) disentangle in-domain gains, OOD alignment tax (60.33% $\to$ 56.81% on Laya; 55.10% on domain CLM-8B), and experience replay recovery, restoring OOD accuracy to 64.10%-65.05% and reaching 99.67%-99.84% in-domain accuracy with 0.00%-0.42% FPR.

Figures & tables

System / Guardrail Family	Moderation Paradigm	Standalone Pre-Ingestion	Target-LLM Decoupled	Mean Latency	Dual-Use FP Control / Calibration	Heterogeneous Routing / Hardware
Llama-Guard 3 [ 1 ] / WildGuard [ 2 ] / ShieldGemma [ 3 ]	Autoregressive Guard (7B–70B)	Yes	Yes	$\sim 200$ – $600$ ms	Uncalibrated Token Logits ( $7.2\%$ – $12.4\%$ XSTest FPR)	No (Single GPU Tier)
ShieldHead [ 4 ] / SingProbe [ 5 ] / Khatri et al. [ 6 ]	Intrinsic Hidden-State Probe	No (Requires LLM Prefill)	No (Coupled to Base LLM)	$\sim 45$ – $90$ ms (Prefill)	Uncalibrated on Dual-Use Compliance	No (Co-located on Target GPU)
GLiNER Guard [ 7 ] / SingGuard [ 8 ] / FlexGuard [ 9 ]	Single-Pass Encoder / Risk Scorer	Yes	Yes	$\sim 35$ – $50$ ms	Continuous Score ( $4.8\%$ XSTest FPR)	No (Single Accelerator Tier)
CLM-8B (Bi-Encoder Direct Baseline) [ 10 ]	Disaggregated Bi-Encoder Head	Yes	Yes	$45.78$ ms ( $1$ -Str.) / $6.64$ ms (Batch) / $0.60$ ms (Arena) §	$90.34\%$ (Zero-Shot) $\to$ 0.00% (Rehearsal, $0.86\%$ ECE)	HBM VectorArena (Ascend 910C)
Traditional ML Baseline	TF-IDF + XGBoost ( $<5$ M)	Yes	Yes	$85.36$ ms (Diag.) / $38.91$ ms (Fast CPU) †	Poorly Calibrated ( $13.11\%$ ECE, $5.88\%$ DUCS FPR)	CPU Only (Xeon CPU)
COGNIT-Guard (Ours)	Calibrated Direct Cascade (322M)	Yes	Yes	14.27 ms (Offline) / 21.77 ms (NPU) / 41.63 ms (Live E2E) ‡	0.00%–0.42% DUCS FPR (1.12% ECE)	Calibrated CPU–NPU Cascade (Ascend 910C)

TABLE I: Taxonomy and Architectural Comparison of Contemporary LLM Safety Guardrail Paradigms across Serving Dimensions.

Fig. 2: Evaluation Dynamics across Guardrail Paradigms: (a) Receiver Operating Characteristic (ROC) curves, (b) Precision-Recall (PR) curves, and (c) Latency-Accuracy Pareto Frontier comparing the offline model-compute sweep ( $\theta^{*}_{\text{offline}}=0.80$ at $14.27$ ms), pure NPU path ( $21.77$ ms), live serial cascade E2E ( $\theta^{*}_{\text{deploy}}=0.70$ at $41.63$ ms, $99.23\%$ accuracy), and standalone diagnostic CPU Mode A ( $85.36$ ms).

Strict Clean Unseen Partition ( $N=607$ : $238$ Benign, $369$ Violations)
Evaluation Metric	Traditional ML	Zero-Shot DFM	COGNIT-Guard (Ours)
Accuracy	95.39% (579/607)	76.61% (465/607)	98.85% (600/607)
[95% Wilson CI]	[93.38%, 96.81%]	[73.08%, 79.82%]	[97.64%, 99.44%]
Paired McNemar (Acc.)	Baseline	$p<0.001$	$p=1.19\times 10^{-4}$ (***)
False Positive Rate (FPR)	5.88% (14/238)	35.71% (85/238)	0.42% (1/238)
[95% Wilson CI]	[3.52%, 9.61%]	[29.88%, 41.98%]	[0.07%, 2.34%]

TABLE II: Empirical Comparison across Guardrail Paradigms under Clean Unseen (

N=607

) and Full (

N=635

) Tracks ( † Diagnostic vs. Live Gatekeeper CPU modes detailed in Table III ).

Part A: Offline Model-Compute Sweep ( $\mathbb{E}[T_{\text{model}}]$ , $N=635$ , Pre-Extracted Cached Features)
Configuration / Execution Stage	NPU Ratio	System Acc.	FPR	Mean Latency	P50 / P95 Lat.
Pure NPU Path ( $\theta=1.00$ )	100.0%	98.90%	0.42%	21.76 ms	21.53 / 23.58 ms
Cascade ( $\theta=0.90$ )	61.3%	98.58%	0.00%	17.40 ms	17.10 / 22.71 ms
Cascade ( $\theta^{*}_{\text{offline}}=0.80$ )	33.7%	98.58%	0.00%	14.27 ms	10.50 / 21.83 ms
Cascade ( $\theta=0.70$ )	8.3%	97.32%	2.52%	11.47 ms	10.50 / 21.49 ms
Mode C: Cached-Feature Tree ( $\theta=0.50$ )	0.0%	97.17%	2.52%	10.50 ms	10.50 / 10.50 ms

TABLE III: Offline Threshold Sweep (

N=635

) and Stage-by-Stage Latency Deconstruction of CPU & Live Cascade Paths (

N=130

Execution Strategy (Ascend 910C)	Mean Latency	P95 Latency	Cold-Start / Alloc Overhead	Speedup / Throughput
Laya-322M: Cold-Start Unpinned Execution	48.20 ms	62.50 ms	14.80 ms	1.00 $\times$ (20.7 QPS)
Laya-322M: Pre-Allocated Memory Buffer	32.40 ms	41.10 ms	6.20 ms	1.49 $\times$ (30.9 QPS)
Laya-322M: Warm On-Chip HBM Residency (Ours)	21.77 ms	23.44 ms	0.00 ms	2.21 $\times$ (45.9 QPS)
CLM-8B Baseline: Uncached Single-Stream ( $B=1$ )	45.78 ms	48.51 ms	0.00 ms	21.8 QPS ( $B=1$ )
CLM-8B Baseline: Batched ( $B=64$ ) / HBM VectorArena	6.64 / 0.60 ms	7.12 / 0.89 ms	0.00 ms	150.6 / $>1000$ QPS

TABLE IV: Ablation of Hardware Memory Residency and Disaggregated VectorArena Caching on Huawei Ascend 910C.

System / Evaluated Model	SafetyBench Acc.	SafetyBench FPR	DUCS FPR (ZH)	XSTest FPR (EN)	Mean Latency
GPT-4 (Zero-Shot) [ 21 ]	89.10%	$\sim 10.5\%$	–	18.40% (46/250)	$>400$ ms (API)
Llama-2-70B-Chat [ 21 ]	68.70%	$\sim 21.3\%$	–	41.60% (104/250)	$>500$ ms (GPU)
WildGuard (7B Decoder) [ 2 ]	–	–	–	7.20% (18/250)	$>200$ ms (GPU)
SingGuard (Lightweight) [ 8 ]	–	–	–	4.80% (12/250)	$\sim 50$ ms (GPU)
Traditional ML (Fast CPU Mode)	53.29% (1119/2100)	37.98% (534/1406)	5.88% (14/238)	–	22.79 ms (CPU)
Zero-Shot Cross-Enc. DFM (Laya-322M)	60.33% (1267/2100)	24.68% (347/1406)	35.71% (85/238)	–	44.85 ms (NPU)

TABLE V: Cross-Benchmark Evaluation Disentangling In-Domain Specialization, OOD Alignment Tax, and Rehearsal Recovery on SafetyBench-ZH (

N=2,100

), DUCS-Bench (

N_{\text{safe}}=238

), and XSTest (EN).

Explore similar work

Sep 29, 2026cs.LG

Guard Models Are Overconfident Where Base Models Are Uncertain

Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for prompt classification and find that although several are nearly calibrated on clean inputs, adversarial attacks degrade their calibration by an order of magnitude, turning false negatives into high-confidence errors indistinguishable from correct detections. Comparing each guard with its corresponding base LM, we find that uncertainty signals often remain available, with the base model typically expressing uncertainty on the same inputs where the guard fails. Layer-wise analyses localize this guard-base divergence to later layers, where guard models exhibit sharper safe/unsafe separation and lower-rank representations, while adversarial harmful inputs lie closer to the clean-safe region. These findings highlight a mismatch between guard confidence and base model uncertainty under attack.

Jonghyun Hong, MinJae Jung, Minwoo Kim

DATUMO INC.

Large Language Model Safety Confidence Calibration Overconfidence Divergence

Ask PDF

May 27, 2026cs.AI

Robust and Efficient Guardrails with Latent Reasoning

Maintaining the safety of large language models (LLMs) is crucial as they are increasingly deployed in real-world applications. Existing safety guardrails typically rely on single-pass classification or, more recently, distilled reasoning. Reasoning-based guardrails significantly outperform classification-only baselines, but they incur substantial query latency and token overhead that make them impractical for highthroughput deployment. To address this challenge, we propose COLAGUARD, a guardrail model that transfers multi-step safety reasoning into a continuous latent space through a stage-wise training curriculum, enabling direct hidden-state propagation at inference. Evaluated on ten prompt- and response-moderation settings spanning eight safety benchmarks, COLAGUARD improves macro-F1 by 8.24 points over Llama Guard 3 and matches our explicit reasoning baseline, GuardReasoner, in macroF1 while delivering a 12.9X speedup and 22.4X reduction in token usage. Our results suggest that latent reasoning offers a practical alternative to explicit rationale generation for deployable guardrails, jointly improving safety robustness and inference efficiency rather than treating them as competing objectives.

Siddharth Sai, Xiaofei Wen, Muhao Chen

University of California, Davis

Large Language Model Safety Guardrail LLM Reasoning Strategies Inference-Time Defense

Ask PDF

Jul 2, 2026cs.LG

kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail

Large language models (LLMs) are increasingly deployed in domains requiring guardrails to detect unsafe, off-topic, or adversarial prompts. Existing guardrails predominantly rely on fine-tuning to build classifiers, which often suffer from low generalization and high inference latency. We present kNNGuard, a training-free guardrail that utilizes the activation space of an off-the-shelf LLM. Given a small bank of 50 safe and unsafe prompts, kNNGuard extracts hidden activations and performs multi-layer kNN fusing activation-space and embedding-space scores for classification. Across six domains spanning topical and security prompts, kNNGuard achieves competitive or superior F1 compared to fine-tuned state-of-the-art guardrails while running 2.7x faster than the best comparable guardrail, and 10x faster than a fine-tuned safety classifier without gradient updates or fine-tuning. Domain adaptation requires only updating the labeled bank, which can be constructed in under 10 seconds and several orders of magnitude faster than established guardrails. We also analyze the impact of system prompts, layer selection, and integration into production LLM pipelines as a configurable, low-latency guardrail.

Mahmoud Abdelfattah, Hamid Nasiri, Peter Garraghan

Lancaster University · Mindgard

Large Language Model Safety Adversarial Prompts Model Activations

Ask PDF