PGuard: A Native, Streaming, and Plug-in Guardrail for Multimodal Large Language Models

Junyu Lin; Mengjie Wu; Yonghao Dong; Yuchuan Fu; Jinfeng Li; Xiaodan Li; Xiaohan Yuan; Yicheng Liu

cs.CLNov 2, 2026

PGuard: A Native, Streaming, and Plug-in Guardrail for Multimodal Large Language Models

Authors: Junyu Lin, Mengjie Wu, Yonghao Dong, Yuchuan Fu, Jinfeng Li, Xiaodan Li, Xiaohan Yuan, Yicheng Liu,

Organizations: Alibaba AAIG

Abstract

As streaming generation becomes the dominant serving paradigm for both text-only and multimodal large language models, ensuring safety throughout the generation process has become a critical challenge. We introduce PGuard, a native, streaming, and plug-in guardrail that transforms safety monitoring from an external serving dependency into an integrated capability of the backbone model. PGuard directly leverages the backbone's intermediate representations for token-level safety detection, without altering its original inference pathway or generation behavior. Specifically, dynamic layer gating selectively aggregates residual-stream representations across layers to extract safety-relevant signals during both prefill and autoregressive decoding. To balance detection accuracy and computational efficiency, PGuard supports multiple representation-modeling architectures, including a lightweight stateless variant and sequence-aware variants built on multi-scale causal EMA-attention and GRU-based recurrent modules. Its design also accounts for the execution characteristics of LLM serving frameworks, such as SGLang, optimizing computation and data access to minimize additional runtime and synchronization overhead. As a result, PGuard incurs only 0.82-2.26% end-to-end overhead on the hundred-billion-parameter backbone Qwen3.5-122B-A10B served by SGLang. Extensive evaluations across more than 30 benchmarks spanning text and multimodal safety, jailbreaks, cyber and CBRN risks, multilingual scenarios, and real-world traffic demonstrate state-of-the-art performance among native guardrails. On text queries with Qwen3.5-9B, PGuard achieves an average F1 score of 0.836, approaching dedicated external guardrails, while supporting deployment-time configuration of category-specific thresholds and enforcement actions.

Figures & tables

Figure 1: Overview of PGuard . (a) PGuard extracts intermediate representations from selected layers of a frozen backbone during prefill and decode. (b) Dynamic layer gating combines layer features for stateless, EMA-attention, or GRU modeling, and task-specific classifiers produce risk scores. Serving co-design integrates selective computation, parallelism, non-blocking execution, and fusion (Section 3 ). (c) Per-token risk scores are returned through engine metadata, and the application applies its policy when a score crosses the threshold.

Figure 2: PGuard integrated into the SGLang serving engine. PGuard runs as a side path of the backbone forward pass. Hidden states from the selected layers are sent to the PGuard head, which reads and updates a per-request state for the EMA-attention and GRU variants. The resulting risk scores are streamed to downstream applications together with the generated tokens.

Figure 3: Selective guard scoring during decode. The captured decode graph is split into segments. The backbone and the state update are replayed at every step. The PGuard scoring segment is replayed only at the sampled steps (steps 2 and 5 in this example) and is skipped at all other steps.

Figure 4: Parallel prefill execution. (a) Chunked EMA scan. Within each chunk, the EMA states are computed by one matrix multiplication with a lower-triangular decay matrix

{\bm{D}}

. The state at the end of each chunk is carried into the next chunk. States are computed only at the sampled tokens and at chunk ends. (b) Rank-sharded scoring. Each tensor-parallel rank scores a disjoint subset of the sampled tokens and computes a local maximum. One all-reduce with maximum reduction then gives every rank the per-request maximum risk score.

Model	ToxicChat	XSTest	OAI-Mod	Aegis2	WGMix	S-Eval	Avg
External guardrails
Qwen3Guard-Gen-8B	0.636	0.908	0.682	0.865	0.887	0.906	0.814
Qwen3Guard-Stream-8B	0.715	0.921	0.740	0.869	0.871	0.890	0.834
WildGuard-7B	0.651	0.932	0.724	0.831	0.869	0.854	0.810
PolyGuard-Qwen-7B	0.677	0.923	0.732	0.866	0.877	0.898	0.829
YuFeng-XGuard-8B	0.705	0.944	0.720	0.859	0.866	0.906	0.833

Table 1: Query safety evaluation on text benchmarks. OAI-Mod denotes OpenAI Moderation; WGMix denotes WildGuardMix. The highest

F_{1}

in each column is shown in bold .

Model	BeaverTails	Q3G-Test	FineHarm	SafeRLHF	Aegis2	WGMix	XSTest	S-Eval	Avg
External guardrails
Qwen3Guard-Gen-8B	0.865	0.924	0.972	0.909	0.863	0.781	0.827	0.785	0.866
Qwen3Guard-Stream-8B	0.851	0.803	0.960	0.930	0.827	0.746	0.890	0.633	0.830
WildGuard-7B	0.841	0.868	0.991	0.925	0.838	0.721	0.912	0.429	0.816
PolyGuard-Qwen-7B	0.802	0.911	0.983	0.894	0.824	0.764	0.349	0.622	0.769
YuFeng-XGuard-8B	0.837	0.907	0.971	0.950	0.802	0.782	0.885	0.910	0.881

Table 2: Response safety evaluation on text benchmarks. Q3G-Test denotes the Qwen3Guard test set; WGMix denotes WildGuardMix. The highest

F_{1}

in each column is shown in bold .

	Query side $F_{1}$									Response side $F_{1}$
Model	SPA-VL	UnsafeB	MM-SB	VLGuard	HoliSafe	VLSB	FigStep	BeaverT-V	Avg	SPA-VL	BeaverT-V	Avg
External guardrails
Llama Guard 4 (12B)	0.535	0.326	0.608	0.650	0.337	0.129	0.779	0.426	0.474	0.495	0.515	0.505
LlavaGuard-7B	0.256	0.582	0.269	0.557	0.424	0.641	0.224	0.535	0.436	–	–	–
Native guardrails
HiddenDetect (Llava-1.6-7B)	0.658	0.234	0.779	0.798	0.649	0.215	0.927	0.845	0.638	–	–	–

Table 3: Multimodal safety evaluation. The highest

F_{1}

in each column is shown in bold .

Model	StrongREJECT	S-Eval-JB	Avg
External guardrails
Qwen3Guard-Gen-8B	0.998	0.858	0.928
Qwen3Guard-Stream-8B	0.997	0.804	0.901
WildGuard-7B	0.995	0.785	0.890
PolyGuard-Qwen-7B	0.995	0.857	0.926
YuFeng-XGuard-8B	1.000	0.944	0.972

Table 4: Safety evaluation results on static jailbreak benchmarks.

	WMDP			CyberSecEval3			SOSBench
Model	bio	chem	cyber	mitre	p.inj	interp.	bio	chem	med	pharm	phys	psy	Avg
External guardrails
Qwen3Guard-Gen-8B	0.085	0.463	0.156	0.876	0.615	0.962	0.951	0.959	0.935	0.951	0.900	1.000	0.738
Qwen3Guard-Stream-8B	0.036	0.241	0.070	0.773	0.421	0.956	0.884	0.859	0.777	0.807	0.755	0.989	0.631
WildGuard-7B	0.111	0.341	0.207	0.860	0.475	0.956	0.929	0.839	0.908	0.792	0.862	0.983	0.689
PolyGuard-Qwen-7B	0.124	0.623	0.240	0.888	0.615	0.967	0.951	0.953	0.897	0.807	0.958	0.989	0.751

Table 5: Query safety evaluation results on Cyber & CBRN benchmarks.

		PolyGuardPrompts		RTP-LX		Lingua
Model	Stage	Q	R	Q	R	Q
External guardrails
Qwen3Guard-Gen-8B	Q+R	0.865	0.781	0.877	0.793	–
Qwen3Guard-Stream-8B	Q+R	0.865	0.770	0.852	0.855	–
WildGuard-7B	Q+R	0.726	0.649	0.589	0.821	–
PolyGuard-Qwen-7B	Q+R	0.869	0.749	0.833	0.769	–

Table 6: Multilingual safety evaluation for queries (Q) and responses (R). PolyGuardPrompts and RTP-LX contain text; Lingua-SafetyBench contains multimodal inputs. Stage indicates the supported decision stages: Q+R, query and response; R, response only; p+d, prefill and decode; pre, prefill only; dec, decode only.

		Query			Response
Model	Stage	XSTest	OR-Bench	Avg	XSTest	OR-Bench	Avg
External guardrails
Qwen3Guard-Gen-8B	Q+R	12.88	7.71	10.29	1.81	1.44	1.63
Qwen3Guard-Stream-8B	Q+R	10.46	4.50	7.48	0.76	0.13	0.45
WildGuard-7B	Q+R	1.07	8.64	4.86	0.38	0.24	0.31
PolyGuard-Qwen-7B	Q+R	3.82	8.72	6.27	21.75	1.01	11.38

Table 7: Evaluation of over-refusal rates on XSTest, OR-Bench (%). Best performance in bold . R denotes response-only classification.

		Query (Q)			Response (R)
Model	Stage	Alpaca	WildChat	Avg	Alpaca	WildChat	Avg
External guardrails
Qwen3Guard-Gen-8B	Q+R	0.43	13.02	6.73	0.46	3.94	2.20
Qwen3Guard-Stream-8B	Q+R	0.39	9.95	5.17	0.12	1.94	1.03
WildGuard-7B	Q+R	0.62	5.47	3.05	0.35	1.43	0.89
PolyGuard-Qwen-7B	Q+R	0.55	14.98	7.77	1.15	10.42	5.79

Table 8: Interception rates on real-world benign text benchmarks (%), reported separately for the query (Q) and response (R) sides. Best performance noted in bold . R denotes response-only classification.

		Query (Q)				Response (R)
Model	Stage	ShareGPT4V	LLaVA-Instr.	MMStar	Avg	ShareGPT4V	LLaVA-Instr.	MMStar	Avg
External guardrails
Llama Guard 4 (12B)	Q+R	0.74	0.01	0.67	0.47	0.67	0.04	1.33	0.68
LlavaGuard-7B	Q	1.19	1.11	0.67	0.99	–	–	–	–
Native guardrails
HiddenDetect (LV-1.6-7B)	pre	1.20	0.21	0.00	0.47	–	–	–	–

Table 9: Interception rates on real-world benign multimodal benchmarks (%), for the query (Q) and response (R) sides. Best performance noted in bold .

Panel of Figure 5 — Figure 5: Early interception and prediction stability of PGuard .

Figure 6: Learned layer-gate profiles across backbones. Each panel shows gate weights against relative transformer depth (horizontal) and training progress (vertical). Brighter bands indicate greater weight assigned to a layer.

Backbone	General safety				Jailbreak	Cyber & CBRN	Multilingual
Backbone	Text Q	Text R	Img Q	Img R	Jailbreak	Cyber & CBRN	Text Q	Text R	Img Q
lasttok
Qwen3-8B	0.824	0.815	–	–	0.906	0.708	0.811	0.792	–
Qwen3.5-9B	0.822	0.795	0.635	0.644	0.917	0.833	0.819	0.773	0.627
Qwen3.5-35B-A3B	0.822	0.812	0.734	0.658	0.941	0.816	0.754	0.791	0.756
Qwen3.5-122B-A10B	0.838	0.819	0.798	0.730	0.950	0.765	0.839	0.803	0.863

Table 10: Safety performance (macro-average

F_{1}

) across backbones and representation-modeling variants. The highest value in each column within each variant group is shown in bold .

Backbone	Input	Stateless			EMA-attention			GRU
Backbone	Input	TTFT (%)	TPOT (%)	Thpt. (%)	TTFT (%)	TPOT (%)	Thpt. (%)	TTFT (%)	TPOT (%)	Thpt. (%)
Qwen3.5-9B (TP=1)	2048	+1.43	+1.78	+1.73	+3.56	+1.69	+1.76	+0.73	+1.64	+1.55
	4096	+1.54	+1.47	+1.45	+3.08	+1.72	+1.81	+0.88	+1.59	+1.48
	8192	+0.04	+1.29	+1.07	+2.48	+1.80	+1.87	+0.69	+1.43	+1.28
Qwen3.5-35B-A3B (TP=2)	2048	-0.34	+0.95	+0.82	+2.08	+1.19	+1.19	+1.38	+3.07	+2.89
	4096	+0.25	-0.93	-0.88	+2.15	+1.42	+1.43	+0.97	+2.18	+2.02

Table 11: Relative changes in TTFT, TPOT, and output throughput (Thpt.) of PGuard compared with the bare backbone, with an output length of 512 tokens. Each backbone uses its best combination of optimizations. For TTFT and TPOT, positive values indicate increased latency. For throughput, positive values indicate reduced output throughput. Lower is better for all metrics.

Configuration	Qwen3.5-9B	Qwen3.5-35B-A3B	Qwen3.5-122B-A10B
Unoptimized	2.42	3.67	2.42
+ Merge fusion	2.39	3.69	2.49
+ Adapter fusion	2.48	3.54	2.29
+ Classifier fusion	1.50	2.10	1.17
+ Async risk copy	1.62	2.59	1.50
+ Selective decode scoring	1.21	0.16	0.76

Table 12: Cumulative ablation of the stateless variant. Each cell reports the output throughput loss (%) relative to the bare backbone.

Configuration	Qwen3.5-9B	Qwen3.5-35B-A3B	Qwen3.5-122B-A10B
Unoptimized	4.62	5.28	4.55
+ Scan chunk tuning	4.58	5.73	3.93
+ Rank-sharded scoring	–	4.66	3.65
+ QKV fusion	4.79	4.33	4.25
+ Classifier fusion	4.07	0.79	3.70
+ Decode TP fusion	–	1.68	2.52

Table 13: Cumulative ablation of the EMA-attention variant. Each cell reports the output throughput loss (%) relative to the bare backbone.

Configuration	Qwen3.5-9B	Qwen3.5-35B-A3B	Qwen3.5-122B-A10B
GRU baseline (pool prefill)	3.30	4.87	3.48
$+$ Vectorized prefill	3.28	4.41	3.24
$+$ Layer aggregation fusion	3.20	4.02	3.26
$+$ GRU update fusion	2.91	4.17	3.21
$+$ Adapter fusion	2.94	4.06	3.21
$+$ Async risk copy	1.16	1.75	1.29

Table 14: End-to-end output throughput (TPS) loss (%) relative to the bare backbone (lower is better), with input/output lengths 4096/512, concurrency 8, 256 requests, and four paired rounds.

Backbone	Variant	Guard weights	Per-request state	Peak workspace
Backbone	Variant	(GiB/GPU)	(MiB/GPU)	(GiB/GPU)
Qwen3.5-9B	Stateless	0.06	–	0.19
	EMA-attention	0.41	0.094	2.69
	GRU	0.03	0.0016	0.69
Qwen3.5-35B-A3B	Stateless	0.04	–	0.09
	EMA-attention	0.10	0.047	1.35

Table 15: Memory overhead of PGuard .

Figure 7: The cost–accuracy plane of the evaluated guardrails. Cost is the serving overhead measured with SGLang on the same GPU driven to saturation and normalized by the bare backbone; accuracy is the macro

F_{1}

of the corresponding task suites.

Method	Query	Response	Avg	Block Rate(%)
PGuard (Qwen3.5-Flash)	97.56 (95.24/100.00)	90.48 (95.00/86.36)	95.16 (95.16/95.16)	0.215
YuFeng-XGuard	95.24 (90.91/100.00)	88.37 (90.48/86.36)	92.91 (90.77/95.16)	0.144
PGuard +YuFeng-XGuard	98.77 (97.56/100.00)	92.68 (100.00/86.36)	96.72 (98.33/95.16)	0.056

Table 16: Two-tier guard comparison. Query, Response, and Avg are evaluated on the S-Eval-Jailbreak stress workload, with responses generated by Qwen3.5-Flash. Each cell reports the

F_{1}

with its precision/recall breakdown in parentheses, i.e. score(P/R) ; higher is better. Avg is computed on the pooled query and response samples. Block Rate is measured separately as the fraction of Alpaca responses blocked.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix


Risk dimension	Risk category	Description
A. Crimes & Illegal Activities	a1. Pornographic Contraband	Obscene or sexually explicit material, including sexual depictions and pornographic resources, plus non-consensual or illegal sexual conduct and date-rape drugs.
	a2. Drug Crimes	Cultivation, synthesis, trafficking, and glorified personal use of illegal drugs and controlled substances.
	a3. Dangerous Weapons	Illicit blades, firearms, and explosives, notably CBRN (chemical, biological, radiological, nuclear) agents and devices such as anthrax, mustard/chlorine, Cs-137/Co-60 dirty bombs, and TATP/RDX car bombs.
	a4. Property Infringement	Unlawfully taking or damaging others’ property via burglary, theft, fraud, extortion, or robbery.
	a5. Economic Crimes	Financial fraud, market manipulation, tax evasion, bribery, money laundering, and IP infringement that disrupt economic order.

Appendix

Table 17: PGuard safety taxonomy adapted from YuFeng-XGuard Lin et al. [2026] : 9 risk dimensions and 29 fine-grained categories, each with its definition.

Modality	Source	#Samples	Description
Text	In-house annotated data	157,497	Safety-annotated data mapped to the shared taxonomy.
	Public safety corpora	71,095	Train splits of PolyGuardMix Kumar and others [2025] , SafeRLHF Dai et al. [2024] , Aegis 2.0 Ghosh and others [2025] , WildGuardMix Han et al. [2024] , BeaverTails Ji et al. [2023] , and SORRY-Bench Xie and others [2025] .
	Text total	228,592
MultiModality	SPA-VL (train split)	78,247	Safety preference alignment data for vision-language models Zhang and others [2025b] .
	Honey-Data	20,000	General multimodal SFT corpus Zhang and others [2025a] for benign data.
	Image-text total	98,247

Appendix

Table 18: Training-data composition by modality.

Configuration	Stage ( $\tau$ , %)	A Crimes	B Hate	C Health	D Ethics	E Privacy	F Cyber	G Extremism	H Inappr.	I Minors
lasttok (Qwen3-8B)	Q-T	65/36/54/42/53	66/59/10/18	71/47	71/69	75/74	87/62/25/62/10	31/48/62	68/54/34	87/50/24
lasttok (Qwen3-8B)	R-T	55/82/55/19/55	33/43/24/20	47/65	61/14	45/70	41/65/30/43/29	59/44/68	77/67/56	38/72/12
gru (Qwen3-8B)	Q-T	75/83/61/28/68	73/24/10/10	61/47	62/56	46/74	80/33/40/21/19	29/70/56	70/62/16	83/57/18
gru (Qwen3-8B)	R-T	72/97/79/21/79	43/23/13/15	77/99	90/35	91/89	72/45/29/42/68	86/70/66	87/69/10	74/85/19
ema (Qwen3-8B)	Q-T	45/25/65/19/66	42/10/10/10	26/35	30/11	45/35	30/39/11/13/12	27/66/19	49/43/14	37/30/10
ema (Qwen3-8B)	R-T	45/38/34/37/54	34/34/16/12	37/41	58/19	56/69	35/59/14/29/10	50/31/65	44/46/10	18/49/10

Appendix

Table 19: Decision thresholds

\tau_{c,y}

in percent, calibrated separately for each guard. Rows are split by stage and input: Q and R denote the query and response stages, and T and VL denote text and image-text inputs. Columns are the nine risk dimensions (A–I). Each cell lists the per-category thresholds of that dimension in a1

\to

an order.

Modeling Method	Backbone Model	Top-3 Layers
lasttok	Qwen3-8B	17,18,19
gru / ema	Qwen3-8B	19,20,35
lasttok / gru / ema	Qwen3.5-9B	20, 24, 28
lasttok / gru / ema	Qwen3.5-flash	20, 24, 28
ema	Qwen3.5-122B-A10B	32,36,40
lasttok / gru	Qwen3.5-122B-A10B	36,40,44

Appendix

Table 20: Top-3 Layer Selection Configurations

Backbone	TP	SGLang	Capture indices	Memory fraction
Qwen3.5-9B	1	0.5.11	20, 24, 28	0.9 / 0.8 / 0.8
Qwen3.5-35B-A3B	2	0.5.11	20, 24, 28	0.9 / 0.8 / 0.8
Qwen3.5-122B-A10B	4	0.5.11	EMA-attention: 32, 36, 40 Stateless/GRU: 36, 40, 44	0.9 / 0.8 / 0.8

Appendix

Table 21: Serving configuration of each backbone. Memory fractions are listed for the stateless, EMA-attention, and GRU variants, respectively.

LLM Guardrails

Large language model guardrails detect or constrain harmful outputs, prompt attacks, and unsafe agent actions before they cause harm.

See all 202 papers

SentGuard: Sentence-Level Streaming Guardrails for Large Language Models
Jun 1, 2026Jiaqi Yu, Xin Wang, Yixu Wang +4Language Model Safety Evaluation LLM Guardrails
1Fudan University · 2Shanghai AI Laboratory
ADTGuard: Adaptive Depth-Time Safeguard for Streaming Inference in Large Language Models
Oct 27, 2026Chuming Wei, Churui Yang, Zihan Zhang +4LLM Safety Monitoring Inference-Time Adaptation of Language Models
Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University · Faculty of Science and Engineering, University of Nottingham Ningbo China · Department of Computer Sciences, University of Wisconsin–Madison +1
Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming
Apr 5, 2026Pride Kavumba, Koki Wataoka, Huy H. Nguyen +2LLM Guardrails LLM Safety Monitoring
SB Intuitions Corp.

Jun 1, 2026J/K move · Enter open · S save

SentGuard: Sentence-Level Streaming Guardrails for Large Language Models

Jiaqi Yu, Xin Wang, Yixu Wang, Jie Li+3

1Fudan University · 2Shanghai AI Laboratory

Language Model Safety Evaluation LLM Guardrails Content Moderation Safety Filtering LLM Safety LLM Safety Monitoring

Open paper AskPDF arXiv

Large language models increasingly stream long, reasoning-intensive responses in real time, making when to moderate as critical as whether to moderate. Existing guardrails fall into two unsatisfactory extremes: response-level methods delay intervention until the full output is generated, whereas token-level methods act on incomplete semantics, often producing unstable decisions and excessive guard invocations. To address this challenge, we propose SentGuard, a sentence-level streaming guardrail that operates in parallel with generation. A lightweight waiting buffer groups streamed tokens into sentence chunks and releases only verified chunks to the user, introducing a small offset that enables SentGuard to assess the current prefix while the target LLM decodes subsequent content. To support this, we construct StreamSafe, a benchmark with structured per-sentence annotations across 8 harm categories, capturing the evolution of safety risks across both reasoning and response segments. We further train SentGuard with a coarse-to-fine objective to detect unsafe intent as soon as it emerges at sentence boundaries. Experiments on 5 safety benchmarks show that SentGuard outperforms existing baselines, detecting 90.5% of unsafe cases within two sentences while maintaining a low streaming false-positive rate of 7.41%.

Abstract

LLM Guardrails

Explore similar work

SentGuard: Sentence-Level Streaming Guardrails for Large Language Models

ADTGuard: Adaptive Depth-Time Safeguard for Streaming Inference in Large Language Models

Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming