ADTGuard: Adaptive Depth-Time Safeguard for Streaming Inference in Large Language Models

Chuming Wei; Churui Yang; Zihan Zhang; Ruien Li; Wusheng Zhang; Yan Zhang; Guangwen Yang

cs.AIOct 27, 2026

ADTGuard: Adaptive Depth-Time Safeguard for Streaming Inference in Large Language Models

Authors: Chuming Wei, Churui Yang, Zihan Zhang, Ruien Li, Wusheng Zhang, Yan Zhang, Guangwen Yang

Organizations: Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University · Faculty of Science and Engineering, University of Nottingham Ningbo China · Department of Computer Sciences, University of Wisconsin–Madison · Department of Computer Science and Technology, Tsinghua University

Abstract

Large Language Models (LLMs) can generate fluent responses to diverse user requests, yet streaming may expose users to harmful content before generation is complete. Existing guardrails face two limitations in this setting. First, post-hoc moderation at response or chunk boundaries delays intervention, risking harmful exposure when content is released before verification. Second, even with token-level detection, coarse response-level supervision does not specify when harmful content begins, while fixed-layer readouts may miss relevant safety signals at other depths. To address these limitations, we introduce ADTGuard, which checks each token before release, progressively refines token-level training targets, and adapts layer selection to the current prefix. ADTGuard models risk across depth and time through Depth-Time Safety Dynamics (DTSD), which integrates safety evidence across layers at each token and uses it to guide risk accumulation over successive tokens. To adapt the choice of observation layers to the current prefix, we propose the Causal Depth Router (CDR). With only response-level labels available, we further develop Risk-Onset Target Refinement (RTR) to progressively construct token-level training targets for both components. Through extensive experiments across multiple safety benchmarks, we demonstrate that ADTGuard consistently achieves strong streaming detection performance, matching or surpassing state-of-the-art safeguards while requiring only 16M trainable parameters and maintaining competitive per-token latency.

Figures & tables

Figure 1: Overview of ADTGuard. At each generation step, CDR selects a prefix-dependent Top- $K$ set of layers and normalizes their routing weights. The selected features are individually reweighted and processed in depth order by DSD, while their weighted fusion is fed to TSD. The DSD output further conditions the TSD update, coupling depth-wise safety evidence with temporal risk evolution to produce a causal safety score for each response prefix over a frozen LLM.

Qwen3-8B Base
Method	SGB-S-Eval	SGB-WildGuard	AEGIS2.0	BeaverTails	Public WildGuard
SIREN (in-domain)	92.22 / 79.84	78.90 / 55.93	84.40 / 83.31	87.02 / 82.99	71.91 / 60.08
SIREN (official)	55.36 / 56.91	70.32 / 52.31	82.84 / 78.75	85.67 / 82.16	65.71 / 53.54
Kelp	88.05 / 88.33	53.59 / 60.12	48.74 / 49.82	83.27 / 83.20	55.04 / 57.52
ADTGuard	93.67 / 93.57	83.94 / 85.87	85.79 / 86.84	87.33 / 87.39	74.76 / 73.86
Llama-3.1-8B Base

Table 1: Comparison on the same backbone. F1 comparison among representation-based safeguards, including SIREN, Kelp, and ADTGuard, using Qwen3-8B Base, Llama-3.1-8B Base, and Gemma-7B Base as shared backbones. Response / Streaming unsafe-class F1 (%) are reported. Bold and underline denote the best and second-best results, respectively, for each metric and dataset within the same backbone.

Model	SGB-S-Eval	SGB-WildGuard	AEGIS2.0	BeaverTails	Public WildGuard
Qwen3Guard-Gen-8B	0.9341 / 0.8806	0.8592 / 0.7903	0.8645 / —	0.8723 / —	0.7892 / —
Qwen3Guard-Stream-8B	0.9338 / 0.9278	0.8456 / 0.8082	0.8752 / 0.8625	0.8754 / 0.8673	0.7559 / 0.7828
Llama-Guard-4-12B	0.9242 / 0.8546	0.8419 / 0.7429	0.8340 / —	0.8623 / —	0.7403 / —
SentGuard-4B	0.9249 / 0.8294	0.8546 / 0.7486	0.8311 / 0.8500	0.8592 / 0.8620	0.7109 / 0.7559
ADTGuard	0.9367 / 0.9357	0.8394 / 0.8587	0.8579 / 0.8684	0.8733 / 0.8739	0.7476 / 0.7386

Table 2: Comparison with dedicated moderation models. F1 comparison with standalone moderation models, including Qwen3Guard-Gen-8B, Qwen3Guard-Stream-8B, Llama-Guard-4-12B, and SentGuard-4B. ADTGuard uses Qwen3-8B Base as the protected backbone. Response / Streaming unsafe-class F1 are reported. Bold and underline denote the best and second-best results, respectively, for each metric and dataset.

Configuration	Response	Streaming
Baseline	87.31	86.05
+ CDR	88.06	86.09
+ DTSD	92.83	87.51
+ RTR	93.58	93.58
+ Mono (ours)	93.67	93.57

Table 3: Component ablation. Starting from an approximately parameter-matched token-wise MLP baseline, we progressively introduce CDR, replace the MLP with DTSD, and then add RTR and Mono on SGB-S-Eval with Qwen3-8B Base. Final-epoch unsafe-class F1 (%) is reported. Bold and underline denote the best and second-best results, respectively.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Safe	Unsafe	Safe	Unsafe
	Training		Test
SGB-S-Eval	5,501	3,499	467	533
SGB-WildGuard	30,163	7,771	1,441	284
AEGIS2.0	10,890	3,541	458	394
BeaverTails	11,603	15,582	1,288	1,733
Public WildGuard	29,566	8,368	1,425	284

Appendix

Table 5: Dataset statistics after dataset-level label filtering. Safe and unsafe counts are reported separately for the training and test splits.

Parameter	Value
Projection / feature dimension	1,024 / 1,024
Router hidden dimension	32
Selective SSM state dimension	256
Time-step projection rank	64
SSM feed-forward dimension	512
Time-conditioning bottleneck	32

Appendix

Table 6: Additional ADTGuard implementation parameters.

Parameter	Qwen3Guard- Gen-8B	Qwen3Guard- Stream-8B	Llama-Guard- 4-12B	SentGuard-4B	Kelp	SIREN (in-domain)	SIREN (official-data)
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW	Adam	Adam
Learning rate	$5\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-5}$	$0.01$ / tuned	$0.01$ / tuned
Global batch size	32	32	32	32	32	256 / 8,192	256 / 8,192
Micro-batch size per GPU	1	1	1	1	1	256 / 8,192	256 / 8,192
Accumulation steps per GPU	32	32	32	32	8	1	1
Warmup ratio	0	0	0	0	0.05	0	0

Appendix

Table 7: Training configurations of the baseline methods. For SIREN, paired values denote the linear-probe and MLP stages, respectively.

Figure 2: Recorded layer routing and aligned risk. (a) Response-equal selection frequencies over the $256$ logged training examples at epoch $5$ . Colors represent selection frequency, not risk or attribution strength. (b) Binary top- $4$ selections for unsafe training sample #175, with $376$ valid response positions and $18$ distinct selected sets. (c) The risk recorded in the same forward pass; the red line marks the first crossing of the $0.5$ threshold at token $174$ . Layer and token positions are one-based. The example is the lower-median-length member of the $16$ logged unsafe responses with $128$ – $512$ tokens and epoch- $5$ RTR targets. No ground-truth harmful-onset boundary is available.

Figure 3: RTR target evolution and recorded training predictions. (a) Actual targets for training sample #175 over all five epochs, using $\beta=(1,0.5,0.25,0,0)$ . (b) Its epoch- $4$ cached probabilities (purple, dashed), epoch- $5$ targets (orange-red, dash-dotted), and epoch- $5$ training-forward predictions (blue, solid). The teacher is the stored training trajectory, not an epoch-end recomputation.

$P$	$N$	Total latency (s)		Added latency (ms/token)		Overhead
		Base	+ADTGuard	Mean	95% CI	(%)
1,000	512	9.6473	10.8488	2.3467	[2.3385, 2.3556]	12.45
1,000	1,024	19.3229	21.7108	2.3320	[2.3235, 2.3410]	12.36
128	1,024	18.9478	21.3519	2.3477	[2.3390, 2.3571]	12.69
512	1,024	19.0917	21.4575	2.3103	[2.3017, 2.3191]	12.39
2,048	1,024	19.8408	22.2309	2.3340	[2.3253, 2.3434]	12.05

Appendix

Table 8: Model-side latency of Qwen3-8B Base with and without ADTGuard.

P

and

N

denote input and output token counts. Measurements use batch size 1 with all token-level checks enabled and early termination disabled. Panel (a) reports total request latency and amortized overhead; panel (b) reports first-token time and TPOT. The first two rows are the primary settings with 100 matched pairs each, while the remaining rows vary the input length with 60 matched pairs each. Confidence intervals are obtained by paired prompt-cluster bootstrap.

Case	Reference tokens	Maximum risk	First alarm	Released tokens
Benign discussion (#42)	424	0.0036	None	424
Euphemistic ridicule (#869)	236	0.9651	105	104

Appendix

Table 9: Observed outcomes for the selected cases. Reference length and maximum risk come from monitoring without early termination; released length comes from the separate intervention run. Termination tokens are excluded.

Figure 4: Token-level monitoring preserves benign discussion. Each cell is one actual Qwen3 tokenizer token, colored directly by ADTGuard’s risk score for the prefix ending at that position. Tokens $1$ – $159$ are shown; the complete $424$ -token response is released, with maximum risk $0.0036$ . Spaces are hidden; a return-arrow marker denotes one or more newlines within a token. Row labels indicate token positions.

Figure 5: Token-level intervention on euphemistic ridicule. All $236$ reference tokens are shown, each with its own recorded prefix-risk score. Actual intervention releases tokens $1$ – $104$ and withholds token $105$ at the first alarm. The dashed panel is an unreleased monitoring reference. The enlarged boundary shows individual scores around the decision; token $105$ is a Markdown delimiter, not an annotated harmful onset. The first four suggestions were already released.

LLM Safety Monitoring

See all 54 papers

SentGuard: Sentence-Level Streaming Guardrails for Large Language Models
Jun 1, 2026Jiaqi Yu, Xin Wang, Yixu Wang +4Language Model Safety Evaluation LLM Guardrails
1Fudan University · 2Shanghai AI Laboratory
Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs
Jun 9, 2026Huizhen Shu, Xuying Li, Piao XueLLM Guardrails Safety Filtering
ModelOneAI · yunshanai
Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming
Apr 5, 2026Pride Kavumba, Koki Wataoka, Huy H. Nguyen +2LLM Guardrails LLM Safety Monitoring
SB Intuitions Corp.

Jun 1, 2026J/K move · Enter open · S save

SentGuard: Sentence-Level Streaming Guardrails for Large Language Models

Jiaqi Yu, Xin Wang, Yixu Wang, Jie Li+3

1Fudan University · 2Shanghai AI Laboratory

Language Model Safety Evaluation LLM Guardrails Content Moderation Safety Filtering LLM Safety LLM Safety Monitoring

Open paper AskPDF arXiv

Large language models increasingly stream long, reasoning-intensive responses in real time, making when to moderate as critical as whether to moderate. Existing guardrails fall into two unsatisfactory extremes: response-level methods delay intervention until the full output is generated, whereas token-level methods act on incomplete semantics, often producing unstable decisions and excessive guard invocations. To address this challenge, we propose SentGuard, a sentence-level streaming guardrail that operates in parallel with generation. A lightweight waiting buffer groups streamed tokens into sentence chunks and releases only verified chunks to the user, introducing a small offset that enables SentGuard to assess the current prefix while the target LLM decodes subsequent content. To support this, we construct StreamSafe, a benchmark with structured per-sentence annotations across 8 harm categories, capturing the evolution of safety risks across both reasoning and response segments. We further train SentGuard with a coarse-to-fine objective to detect unsafe intent as soon as it emerges at sentence boundaries. Experiments on 5 safety benchmarks show that SentGuard outperforms existing baselines, detecting 90.5% of unsafe cases within two sentences while maintaining a low streaming false-positive rate of 7.41%.

Abstract

LLM Safety Monitoring

Explore similar work

SentGuard: Sentence-Level Streaming Guardrails for Large Language Models

Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs

Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming