As streaming generation becomes the dominant serving paradigm for both text-only and multimodal large language models, ensuring safety throughout the generation process has become a critical challenge. We introduce PGuard, a native, streaming, and plug-in guardrail that transforms safety monitoring from an external serving dependency into an integrated capability of the backbone model. PGuard directly leverages the backbone's intermediate representations for token-level safety detection, without altering its original inference pathway or generation behavior. Specifically, dynamic layer gating selectively aggregates residual-stream representations across layers to extract safety-relevant signals during both prefill and autoregressive decoding. To balance detection accuracy and computational efficiency, PGuard supports multiple representation-modeling architectures, including a lightweight stateless variant and sequence-aware variants built on multi-scale causal EMA-attention and GRU-based recurrent modules. Its design also accounts for the execution characteristics of LLM serving frameworks, such as SGLang, optimizing computation and data access to minimize additional runtime and synchronization overhead. As a result, PGuard incurs only 0.82-2.26% end-to-end overhead on the hundred-billion-parameter backbone Qwen3.5-122B-A10B served by SGLang. Extensive evaluations across more than 30 benchmarks spanning text and multimodal safety, jailbreaks, cyber and CBRN risks, multilingual scenarios, and real-world traffic demonstrate state-of-the-art performance among native guardrails. On text queries with Qwen3.5-9B, PGuard achieves an average F1 score of 0.836, approaching dedicated external guardrails, while supporting deployment-time configuration of category-specific thresholds and enforcement actions.
Figures & tables
Figure 1: Overview of PGuard . (a) PGuard extracts intermediate representations from selected layers of a frozen backbone during prefill and decode. (b) Dynamic layer gating combines layer features for stateless, EMA-attention, or GRU modeling, and task-specific classifiers produce risk scores. Serving co-design integrates selective computation, parallelism, non-blocking execution, and fusion (Section 3 ). (c) Per-token risk scores are returned through engine metadata, and the application applies its policy when a score crosses the threshold.
Figure 2: PGuard integrated into the SGLang serving engine. PGuard runs as a side path of the backbone forward pass. Hidden states from the selected layers are sent to the PGuard head, which reads and updates a per-request state for the EMA-attention and GRU variants. The resulting risk scores are streamed to downstream applications together with the generated tokens.
Figure 3: Selective guard scoring during decode. The captured decode graph is split into segments. The backbone and the state update are replayed at every step. The PGuard scoring segment is replayed only at the sampled steps (steps 2 and 5 in this example) and is skipped at all other steps.
Figure 4: Parallel prefill execution. (a) Chunked EMA scan. Within each chunk, the EMA states are computed by one matrix multiplication with a lower-triangular decay matrix D . The state at the end of each chunk is carried into the next chunk. States are computed only at the sampled tokens and at chunk ends. (b) Rank-sharded scoring. Each tensor-parallel rank scores a disjoint subset of the sampled tokens and computes a local maximum. One all-reduce with maximum reduction then gives every rank the per-request maximum risk score.
Model
ToxicChat
XSTest
OAI-Mod
Aegis2
WGMix
S-Eval
Avg
External guardrails
Qwen3Guard-Gen-8B
0.636
0.908
0.682
0.865
0.887
0.906
0.814
Qwen3Guard-Stream-8B
0.715
0.921
0.740
0.869
0.871
0.890
0.834
WildGuard-7B
0.651
0.932
0.724
0.831
0.869
0.854
0.810
PolyGuard-Qwen-7B
0.677
0.923
0.732
0.866
0.877
0.898
0.829
YuFeng-XGuard-8B
0.705
0.944
0.720
0.859
0.866
0.906
0.833
Table 1: Query safety evaluation on text benchmarks. OAI-Mod denotes OpenAI Moderation; WGMix denotes WildGuardMix. The highest F1 in each column is shown in bold .
Model
BeaverTails
Q3G-Test
FineHarm
SafeRLHF
Aegis2
WGMix
XSTest
S-Eval
Avg
External guardrails
Qwen3Guard-Gen-8B
0.865
0.924
0.972
0.909
0.863
0.781
0.827
0.785
0.866
Qwen3Guard-Stream-8B
0.851
0.803
0.960
0.930
0.827
0.746
0.890
0.633
0.830
WildGuard-7B
0.841
0.868
0.991
0.925
0.838
0.721
0.912
0.429
0.816
PolyGuard-Qwen-7B
0.802
0.911
0.983
0.894
0.824
0.764
0.349
0.622
0.769
YuFeng-XGuard-8B
0.837
0.907
0.971
0.950
0.802
0.782
0.885
0.910
0.881
Table 2: Response safety evaluation on text benchmarks. Q3G-Test denotes the Qwen3Guard test set; WGMix denotes WildGuardMix. The highest F1 in each column is shown in bold .
Query side F1
Response side F1
Model
SPA-VL
UnsafeB
MM-SB
VLGuard
HoliSafe
VLSB
FigStep
BeaverT-V
Avg
SPA-VL
BeaverT-V
Avg
External guardrails
Llama Guard 4 (12B)
0.535
0.326
0.608
0.650
0.337
0.129
0.779
0.426
0.474
0.495
0.515
0.505
LlavaGuard-7B
0.256
0.582
0.269
0.557
0.424
0.641
0.224
0.535
0.436
–
–
–
Native guardrails
HiddenDetect (Llava-1.6-7B)
0.658
0.234
0.779
0.798
0.649
0.215
0.927
0.845
0.638
–
–
–
Table 3: Multimodal safety evaluation. The highest F1 in each column is shown in bold .
Model
StrongREJECT
S-Eval-JB
Avg
External guardrails
Qwen3Guard-Gen-8B
0.998
0.858
0.928
Qwen3Guard-Stream-8B
0.997
0.804
0.901
WildGuard-7B
0.995
0.785
0.890
PolyGuard-Qwen-7B
0.995
0.857
0.926
YuFeng-XGuard-8B
1.000
0.944
0.972
Table 4: Safety evaluation results on static jailbreak benchmarks.
Table 6: Multilingual safety evaluation for queries (Q) and responses (R). PolyGuardPrompts and RTP-LX contain text; Lingua-SafetyBench contains multimodal inputs. Stage indicates the supported decision stages: Q+R, query and response; R, response only; p+d, prefill and decode; pre, prefill only; dec, decode only.
Query
Response
Model
Stage
XSTest
OR-Bench
Avg
XSTest
OR-Bench
Avg
External guardrails
Qwen3Guard-Gen-8B
Q+R
12.88
7.71
10.29
1.81
1.44
1.63
Qwen3Guard-Stream-8B
Q+R
10.46
4.50
7.48
0.76
0.13
0.45
WildGuard-7B
Q+R
1.07
8.64
4.86
0.38
0.24
0.31
PolyGuard-Qwen-7B
Q+R
3.82
8.72
6.27
21.75
1.01
11.38
Table 7: Evaluation of over-refusal rates on XSTest, OR-Bench (%). Best performance in bold . R denotes response-only classification.
Query (Q)
Response (R)
Model
Stage
Alpaca
WildChat
Avg
Alpaca
WildChat
Avg
External guardrails
Qwen3Guard-Gen-8B
Q+R
0.43
13.02
6.73
0.46
3.94
2.20
Qwen3Guard-Stream-8B
Q+R
0.39
9.95
5.17
0.12
1.94
1.03
WildGuard-7B
Q+R
0.62
5.47
3.05
0.35
1.43
0.89
PolyGuard-Qwen-7B
Q+R
0.55
14.98
7.77
1.15
10.42
5.79
Table 8: Interception rates on real-world benign text benchmarks (%), reported separately for the query (Q) and response (R) sides. Best performance noted in bold . R denotes response-only classification.
Query (Q)
Response (R)
Model
Stage
ShareGPT4V
LLaVA-Instr.
MMStar
Avg
ShareGPT4V
LLaVA-Instr.
MMStar
Avg
External guardrails
Llama Guard 4 (12B)
Q+R
0.74
0.01
0.67
0.47
0.67
0.04
1.33
0.68
LlavaGuard-7B
Q
1.19
1.11
0.67
0.99
–
–
–
–
Native guardrails
HiddenDetect (LV-1.6-7B)
pre
1.20
0.21
0.00
0.47
–
–
–
–
Table 9: Interception rates on real-world benign multimodal benchmarks (%), for the query (Q) and response (R) sides. Best performance noted in bold .
Figure 5: Early interception and prediction stability of PGuard .
Figure 6: Learned layer-gate profiles across backbones. Each panel shows gate weights against relative transformer depth (horizontal) and training progress (vertical). Brighter bands indicate greater weight assigned to a layer.
Backbone
General safety
Jailbreak
Cyber & CBRN
Multilingual
Text Q
Text R
Img Q
Img R
Text Q
Text R
Img Q
lasttok
Qwen3-8B
0.824
0.815
–
–
0.906
0.708
0.811
0.792
–
Qwen3.5-9B
0.822
0.795
0.635
0.644
0.917
0.833
0.819
0.773
0.627
Qwen3.5-35B-A3B
0.822
0.812
0.734
0.658
0.941
0.816
0.754
0.791
0.756
Qwen3.5-122B-A10B
0.838
0.819
0.798
0.730
0.950
0.765
0.839
0.803
0.863
Table 10: Safety performance (macro-average F1 ) across backbones and representation-modeling variants. The highest value in each column within each variant group is shown in bold .
Backbone
Input
Stateless
EMA-attention
GRU
TTFT (%)
TPOT (%)
Thpt. (%)
TTFT (%)
TPOT (%)
Thpt. (%)
TTFT (%)
TPOT (%)
Thpt. (%)
Qwen3.5-9B (TP=1)
2048
+1.43
+1.78
+1.73
+3.56
+1.69
+1.76
+0.73
+1.64
+1.55
4096
+1.54
+1.47
+1.45
+3.08
+1.72
+1.81
+0.88
+1.59
+1.48
8192
+0.04
+1.29
+1.07
+2.48
+1.80
+1.87
+0.69
+1.43
+1.28
Qwen3.5-35B-A3B (TP=2)
2048
-0.34
+0.95
+0.82
+2.08
+1.19
+1.19
+1.38
+3.07
+2.89
4096
+0.25
-0.93
-0.88
+2.15
+1.42
+1.43
+0.97
+2.18
+2.02
Table 11: Relative changes in TTFT, TPOT, and output throughput (Thpt.) of PGuard compared with the bare backbone, with an output length of 512 tokens. Each backbone uses its best combination of optimizations. For TTFT and TPOT, positive values indicate increased latency. For throughput, positive values indicate reduced output throughput. Lower is better for all metrics.
Configuration
Qwen3.5-9B
Qwen3.5-35B-A3B
Qwen3.5-122B-A10B
Unoptimized
2.42
3.67
2.42
+ Merge fusion
2.39
3.69
2.49
+ Adapter fusion
2.48
3.54
2.29
+ Classifier fusion
1.50
2.10
1.17
+ Async risk copy
1.62
2.59
1.50
+ Selective decode scoring
1.21
0.16
0.76
Table 12: Cumulative ablation of the stateless variant. Each cell reports the output throughput loss (%) relative to the bare backbone.
Configuration
Qwen3.5-9B
Qwen3.5-35B-A3B
Qwen3.5-122B-A10B
Unoptimized
4.62
5.28
4.55
+ Scan chunk tuning
4.58
5.73
3.93
+ Rank-sharded scoring
–
4.66
3.65
+ QKV fusion
4.79
4.33
4.25
+ Classifier fusion
4.07
0.79
3.70
+ Decode TP fusion
–
1.68
2.52
Table 13: Cumulative ablation of the EMA-attention variant. Each cell reports the output throughput loss (%) relative to the bare backbone.
Configuration
Qwen3.5-9B
Qwen3.5-35B-A3B
Qwen3.5-122B-A10B
GRU baseline (pool prefill)
3.30
4.87
3.48
+ Vectorized prefill
3.28
4.41
3.24
+ Layer aggregation fusion
3.20
4.02
3.26
+ GRU update fusion
2.91
4.17
3.21
+ Adapter fusion
2.94
4.06
3.21
+ Async risk copy
1.16
1.75
1.29
Table 14: End-to-end output throughput (TPS) loss (%) relative to the bare backbone (lower is better), with input/output lengths 4096/512, concurrency 8, 256 requests, and four paired rounds.
Backbone
Variant
Guard weights
Per-request state
Peak workspace
(GiB/GPU)
(MiB/GPU)
(GiB/GPU)
Qwen3.5-9B
Stateless
0.06
–
0.19
EMA-attention
0.41
0.094
2.69
GRU
0.03
0.0016
0.69
Qwen3.5-35B-A3B
Stateless
0.04
–
0.09
EMA-attention
0.10
0.047
1.35
Table 15: Memory overhead of PGuard .
Figure 7: The cost–accuracy plane of the evaluated guardrails. Cost is the serving overhead measured with SGLang on the same GPU driven to saturation and normalized by the bare backbone; accuracy is the macro F1 of the corresponding task suites.
Method
Query
Response
Avg
Block Rate(%)
PGuard (Qwen3.5-Flash)
97.56 (95.24/100.00)
90.48 (95.00/86.36)
95.16 (95.16/95.16)
0.215
YuFeng-XGuard
95.24 (90.91/100.00)
88.37 (90.48/86.36)
92.91 (90.77/95.16)
0.144
PGuard +YuFeng-XGuard
98.77 (97.56/100.00)
92.68 (100.00/86.36)
96.72 (98.33/95.16)
0.056
Table 16: Two-tier guard comparison. Query, Response, and Avg are evaluated on the S-Eval-Jailbreak stress workload, with responses generated by Qwen3.5-Flash. Each cell reports the F1 with its precision/recall breakdown in parentheses, i.e. score(P/R) ; higher is better. Avg is computed on the pooled query and response samples. Block Rate is measured separately as the fraction of Alpaca responses blocked.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Risk dimension
Risk category
Description
A. Crimes & Illegal Activities
a1. Pornographic Contraband
Obscene or sexually explicit material, including sexual depictions and pornographic resources, plus non-consensual or illegal sexual conduct and date-rape drugs.
a2. Drug Crimes
Cultivation, synthesis, trafficking, and glorified personal use of illegal drugs and controlled substances.
a3. Dangerous Weapons
Illicit blades, firearms, and explosives, notably CBRN (chemical, biological, radiological, nuclear) agents and devices such as anthrax, mustard/chlorine, Cs-137/Co-60 dirty bombs, and TATP/RDX car bombs.
a4. Property Infringement
Unlawfully taking or damaging others’ property via burglary, theft, fraud, extortion, or robbery.
a5. Economic Crimes
Financial fraud, market manipulation, tax evasion, bribery, money laundering, and IP infringement that disrupt economic order.
Appendix
Table 17: PGuard safety taxonomy adapted from YuFeng-XGuard Lin et al. [2026] : 9 risk dimensions and 29 fine-grained categories, each with its definition.
Modality
Source
#Samples
Description
Text
In-house annotated data
157,497
Safety-annotated data mapped to the shared taxonomy.
Public safety corpora
71,095
Train splits of PolyGuardMix Kumar and others [2025] , SafeRLHF Dai et al. [2024] , Aegis 2.0 Ghosh and others [2025] , WildGuardMix Han et al. [2024] , BeaverTails Ji et al. [2023] , and SORRY-Bench Xie and others [2025] .
Text total
228,592
MultiModality
SPA-VL (train split)
78,247
Safety preference alignment data for vision-language models Zhang and others [2025b] .
Honey-Data
20,000
General multimodal SFT corpus Zhang and others [2025a] for benign data.
Image-text total
98,247
Appendix
Table 18: Training-data composition by modality.
Configuration
Stage ( τ , %)
A Crimes
B Hate
C Health
D Ethics
E Privacy
F Cyber
G Extremism
H Inappr.
I Minors
lasttok (Qwen3-8B)
Q-T
65/36/54/42/53
66/59/10/18
71/47
71/69
75/74
87/62/25/62/10
31/48/62
68/54/34
87/50/24
R-T
55/82/55/19/55
33/43/24/20
47/65
61/14
45/70
41/65/30/43/29
59/44/68
77/67/56
38/72/12
gru (Qwen3-8B)
Q-T
75/83/61/28/68
73/24/10/10
61/47
62/56
46/74
80/33/40/21/19
29/70/56
70/62/16
83/57/18
R-T
72/97/79/21/79
43/23/13/15
77/99
90/35
91/89
72/45/29/42/68
86/70/66
87/69/10
74/85/19
ema (Qwen3-8B)
Q-T
45/25/65/19/66
42/10/10/10
26/35
30/11
45/35
30/39/11/13/12
27/66/19
49/43/14
37/30/10
R-T
45/38/34/37/54
34/34/16/12
37/41
58/19
56/69
35/59/14/29/10
50/31/65
44/46/10
18/49/10
Appendix
Table 19: Decision thresholds τc,y in percent, calibrated separately for each guard. Rows are split by stage and input: Q and R denote the query and response stages, and T and VL denote text and image-text inputs. Columns are the nine risk dimensions (A–I). Each cell lists the per-category thresholds of that dimension in a1 → an order.
Large language models increasingly stream long, reasoning-intensive responses in real time, making when to moderate as critical as whether to moderate. Existing guardrails fall into two unsatisfactory extremes: response-level methods delay intervention until the full output is generated, whereas token-level methods act on incomplete semantics, often producing unstable decisions and excessive guard invocations. To address this challenge, we propose SentGuard, a sentence-level streaming guardrail that operates in parallel with generation. A lightweight waiting buffer groups streamed tokens into sentence chunks and releases only verified chunks to the user, introducing a small offset that enables SentGuard to assess the current prefix while the target LLM decodes subsequent content. To support this, we construct StreamSafe, a benchmark with structured per-sentence annotations across 8 harm categories, capturing the evolution of safety risks across both reasoning and response segments. We further train SentGuard with a coarse-to-fine objective to detect unsafe intent as soon as it emerges at sentence boundaries. Experiments on 5 safety benchmarks show that SentGuard outperforms existing baselines, detecting 90.5% of unsafe cases within two sentences while maintaining a low streaming false-positive rate of 7.41%.
Large Language Models (LLMs) can generate fluent responses to diverse user requests, yet streaming may expose users to harmful content before generation is complete. Existing guardrails face two limitations in this setting. First, post-hoc moderation at response or chunk boundaries delays intervention, risking harmful exposure when content is released before verification. Second, even with token-level detection, coarse response-level supervision does not specify when harmful content begins, while fixed-layer readouts may miss relevant safety signals at other depths. To address these limitations, we introduce ADTGuard, which checks each token before release, progressively refines token-level training targets, and adapts layer selection to the current prefix. ADTGuard models risk across depth and time through Depth-Time Safety Dynamics (DTSD), which integrates safety evidence across layers at each token and uses it to guide risk accumulation over successive tokens. To adapt the choice of observation layers to the current prefix, we propose the Causal Depth Router (CDR). With only response-level labels available, we further develop Risk-Onset Target Refinement (RTR) to progressively construct token-level training targets for both components. Through extensive experiments across multiple safety benchmarks, we demonstrate that ADTGuard consistently achieves strong streaming detection performance, matching or surpassing state-of-the-art safeguards while requiring only 16M trainable parameters and maintaining competitive per-token latency.
Chuming Wei, Churui Yang, Zihan Zhang +4
Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University · Faculty of Science and Engineering, University of Nottingham Ningbo China · Department of Computer Sciences, University of Wisconsin–Madison +1
In many practical LLM deployments, a single guardrail is used for both prompt and response moderation. Prompt moderation operates on fully observed text, whereas streaming response moderation requires safety decisions to be made over partial generations. Existing text-based streaming guardrails commonly frame this output-side problem as boundary detection, training models to identify the earliest prefix at which a response has already become unsafe. In this work, we introduce StreamGuard, a unified model-agnostic streaming guardrail that instead formulates moderation as a forecasting problem: given a partial prefix, the model predicts the expected harmfulness of likely future continuations. We supervise this prediction using Monte Carlo rollouts, which enables early intervention without requiring exact token-level boundary annotations. Across standard safety benchmarks, StreamGuard performs strongly both for input moderation and for streaming output moderation. At the 8B scale, StreamGuard improves aggregated input-moderation F1 from 86.7 to 88.2 and aggregated streaming output-moderation F1 from 80.4 to 81.9 relative to Qwen3Guard-Stream-8B-strict. On the QWENGUARDTEST response_loc streaming benchmark, StreamGuard reaches 97.5 F1, 95.1 recall, and 92.6% on-time intervention, compared to 95.9 F1, 92.1 recall, and 89.9% for Qwen3Guard-Stream-8B-stric, while reducing the miss rate from 7.9% to 4.9%. We further show that forecasting-based supervision transfers effectively across tokenizers and model families: with transferred targets, Gemma3-StreamGuard-1B reaches 81.3 response-moderation F1, 98.2 streaming F1, and a 3.5% miss rate. These results show that strong end-to-end streaming moderation can be obtained without exact boundary labels, and that forecasting future risk is an effective supervision strategy for low-latency safety intervention.