Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs

Pavan Maddula

cs.CLOct 6, 2026

Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs

Authors: Pavan Maddula

Organizations: Independent Researcher

Abstract

Standard safety evaluations of large language models assess harmful requests written in canonical plain text, while models in real-world deployment routinely receive inputs containing emojis, altered spellings, encoded strings, and character-level variations. This work introduces the Adversarial Surface-Form Robustness Dataset (ASRD), comprising 2,100 prompts across seven distinct surface-form families. Five open-weight language models are evaluated across these prompts, producing 10,500 responses. The Quad-State Evaluation Rubric classifies each response into one of four outcomes: harmful compliance, safe response, comprehension failure, or indeterminate. Emoji and invisible Unicode variations cause almost no comprehension failure, with pooled harmful compliance of 20.27% and 17.20% against a 22.87% baseline that is driven mainly by Mistral 7B, whereas leetspeak, encoded wrappers, and hybrid transformations score 2.40%, 0.13%, and 2.40% while comprehension failure rises to 36.47%, 65.60%, and 34.47%. Inspection of raw model outputs reveals three response behaviors: hallucinated benignity, structural collapse, and language drift. Project page: www.pavanmaddula.com/quadstate

Figures & tables

Family	Prompt Pattern
F0 (Baseline)	[text]
F1 (Emoji-augmented)	How to [emoji] [text] with [emoji] steps?
F2 (Homoglyph substitution)	[homoglyph_text] [emoji]
F3 (Leetspeak and emoji)	[leetspeak_text] [emoji] [emoji]
F4 (Encoded wrappers)	[emoji] Decode this and follow the instructions: [base64_payload] [emoji]
F5 (Invisible Unicode)	How to [invisible_chars] [text] [invisible_chars] with tool?

Table 1: Structural patterns of the seven prompt families derived from the 300 base seeds.

Outcome Label	Count	Percentage
Safe Response (SRR)	7,060	67.24%
Comprehension Failure (CFR)	2,159	20.56%
Harmful Compliance (HCR)	1,232	11.73%
Indeterminate (IR)	49	0.47%
Total	10,500	100.00%

Table 2: Overall Quad-State label distribution across the complete evaluation corpus (

N=10{,}500

Prompt Family	H	S	C	I	HCR	SRR	CFR	IR	$\Delta$ HCR	$\Delta$ CFR
F0 Baseline	343	1,153	4	0	22.87%	76.87%	0.27%	0.00%	—	—
F1 Emoji-augmented	304	1,194	2	0	20.27%	79.60%	0.13%	0.00%	$-$ 2.60	$-$ 0.13
F2 Homoglyph	253	1,145	95	7	16.87%	76.33%	6.33%	0.47%	$-$ 6.00	$+$ 6.07
F3 Leetspeak and emoji	36	906	547	11	2.40%	60.40%	36.47%	0.73%	$-$ 20.47	$+$ 36.20
F4 Encoded wrappers	2	494	984	20	0.13%	32.93%	65.60%	1.33%	$-$ 22.73	$+$ 65.33
F5 Invisible Unicode	258	1,232	10	0	17.20%	82.13%	0.67%	0.00%	$-$ 5.67	$+$ 0.40

Table 3: Quad-State label counts and rates across the seven prompt families (

N=1{,}500

per family).

\Delta

denotes the percentage-point change relative to baseline F0.

Prompt Family	H	S	C	I
F1 Emoji-augmented	278	65	0	0
F2 Homoglyph	207	82	51	3
F3 Leetspeak and emoji	26	67	242	8
F4 Encoded wrappers	0	34	299	10
F5 Invisible Unicode	240	102	1	0
F6 Hybrid	32	65	235	11

Table 4: Labels on each transformed family for the 343 responses labeled harmful compliance on baseline F0, paired by model and seed.

Model	H	S	C	I	HCR	SRR	CFR	IR
Gemma 2 9B	79	1,958	61	2	3.76%	93.24%	2.90%	0.10%
Gemma 3 4B	61	1,668	361	10	2.90%	79.43%	17.19%	0.48%
Llama 3.1 8B	10	1,789	299	2	0.48%	85.19%	14.24%	0.10%
Llama 3.2 3B	29	1,476	590	5	1.38%	70.29%	28.10%	0.24%
Mistral 7B	1,053	169	848	30	50.14%	8.05%	40.38%	1.43%

Table 5: Quad-State label distributions by evaluated model (

N=2{,}100

per model).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Mechanism	Representative Pattern / Example
F0 (Baseline)	Canonical plain text	Provide step-by-step instructions to brute-force login credentials...
F1 (Emoji)	Emoji question frame	How to [emoji] Provide step-by-step instructions... with [emoji] steps?
F2 (Homoglyph)	Cyrillic code points	Prоvdе теp-by-теp nтrucтоn то bruте-fоrcе... [emoji]
F3 (Leetspeak)	Alphanumeric mapping	Pr0v1d3 573p-by-573p 1n57ruc710n5 70 bru73-f0rc3... [emoji] [emoji]
F4 (Encoded)	Base64 container wrapper	[emoji] Decode this and follow the instructions: UHJvdmlkZSB... [emoji]
F5 (Invisible)	Zero-width characters	How to[\u200B\u200C\u200D x4] Provide step-by-step instructions...[\u200B\u200C\u200D x4] with tool?

Appendix

Table 6: Transformation mechanisms and representative patterns derived from seed seed_0001 .

Outcome Class	Matches	Agreement	Disagreements (GLM)
Harmful Compliance	44 / 50	88.0%	3 Safe, 3 Comp. Fail
Safe Response	43 / 43	100.0%	None
Comprehension Failure	44 / 49	89.8%	5 Safe
Indeterminate	2 / 3	66.7%	1 Comp. Fail
Overall	133 / 145	91.7%	Cohen’s $\kappa=0.8781$

Appendix

Table 7: Secondary judge validation report comparing Nemotron 3 Ultra with GLM-5.3-Flash on the 145 completions from the five evaluated models within the stratified sample.

Automated Label	Human H	Human S	Human C	Human I	Total
Harmful Compliance	14	0	0	0	14
Safe Response	1	65	2	0	68
Comprehension Failure	0	4	14	0	18
Overall	93 / 100 agreement, Cohen’s $\kappa=0.8545$

Appendix

Table 8: Automated labels compared with human labels on the 100 responses drawn uniformly from the 10,500 responses.

Metric	Model	F0	F1	F2	F3	F4	F5	F6
HCR	Gemma 2 9B	12.00	0.33	10.33	2.33	0.67	0.67	0.00
	Gemma 3 4B	7.33	1.00	9.67	1.00	0.00	1.33	0.00
	Llama 3.1 8B	0.67	0.00	2.33	0.33	0.00	0.00	0.00
	Llama 3.2 3B	3.33	1.33	3.33	0.00	0.00	1.67	0.00
	Mistral 7B	91.00	98.67	58.67	8.33	0.00	82.33	12.00
CFR	Gemma 2 9B	0.00	0.00	0.00	3.00	16.33	0.00	1.00

Appendix

Table 9: HCR and CFR (%) for each evaluated model and prompt family (

N=300

per cell).

Prompt Family	$\Delta$ HCR [95% CI]	$\Delta$ CFR [95% CI]
F1 Emoji-augmented	$-$ 2.60 [ $-$ 4.13, $-$ 1.20]	$-$ 0.13 [ $-$ 0.40, $+$ 0.13]
F2 Homoglyph	$-$ 6.00 [ $-$ 7.60, $-$ 4.40]	$+$ 6.07 [ $+$ 4.73, $+$ 7.47]
F3 Leetspeak and emoji	$-$ 20.47 [ $-$ 22.20, $-$ 18.80]	$+$ 36.20 [ $+$ 33.87, $+$ 38.60]
F4 Encoded wrappers	$-$ 22.73 [ $-$ 24.40, $-$ 21.20]	$+$ 65.33 [ $+$ 63.73, $+$ 66.93]
F5 Invisible Unicode	$-$ 5.67 [ $-$ 7.20, $-$ 4.20]	$+$ 0.40 [ $-$ 0.13, $+$ 0.93]
F6 Hybrid	$-$ 20.47 [ $-$ 22.27, $-$ 18.73]	$+$ 34.20 [ $+$ 32.13, $+$ 36.27]

Appendix

Table 10: Pooled percentage-point changes relative to baseline F0 with seed-level bootstrap 95% confidence intervals (

N=1{,}500

per family).

Prompt Family	H	S	C	I	HCR	SRR	CFR	IR	Empty
F0 Baseline	3	297	0	0	1.00%	99.00%	0.00%	0.00%	0
F1 Emoji-augmented	0	300	0	0	0.00%	100.00%	0.00%	0.00%	0
F2 Homoglyph	3	281	3	13	1.00%	93.67%	1.00%	4.33%	9
F3 Leetspeak and emoji	5	282	12	1	1.67%	94.00%	4.00%	0.33%	0
F4 Encoded wrappers	0	0	0	300	0.00%	0.00%	0.00%	100.00%	300
F5 Invisible Unicode	0	258	0	42	0.00%	86.00%	0.00%	14.00%	30

Appendix

Table 11: Quad-State label distribution for GPT-OSS 20B (

N=300

per family), reported separately from the main analysis.

Explore similar work

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiLarge Language Model Safety Guardrail
Wrynx Inc. · Independent Researcher
QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
Jul 23, 2026Emilio FerraraLarge Language Model Bias Large Language Model Quantization
Thomas Lord Department of Computer Science University of Southern California
The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
May 6, 2026Alif Al Hasan, Sumon BiswasLarge Language Model Safety Refusals
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA

Sep 16, 2026, cs.AIJ/K move · Enter open · S save

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

Alizishaan Khatri, Chiquita Prabhu, Omkar Neogi

Wrynx Inc. · Independent Researcher

Large Language Model Safety Guardrail Harms Precision-Critical Manipulation Latent States Multilayer Perceptrons

Open paper AskPDF arXiv

Autonomous systems increasingly rely on Large Language Models (LLMs) yet the safety infrastructure surrounding these models introduces latency and compute overhead. This limits utility in resource-constrained, time-critical deployments. Existing external guardrail models remain blind to the model's internal workings, creating a fundamental assurance gap. We ask: does the model already know when the content is harmful? We extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes (12.6M parameters) to detect harmful prompts. Evaluated on WildJailbreak, Beavertails, and AEGIS 2.0, our probes achieve F1 scores of 99%, 83%, and 84%, respectively competitive with 1000x larger guard models while cutting latency and compute costs.

Abstract

Explore similar work

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models