KV cache compression is widely used for long context LLM inference under memory constraints, while deployed systems typically score refusals after generation with keyword filters or learned classifiers. Such monitors are intended to indicate whether a model declined a harmful request under the serving regime actually used. However, it remains unclear whether matched compression that preserves task accuracy also preserves agreement between lightweight lexical monitors and stronger refusal classifiers. We study this with a paired protocol on n=200 harmful prompts with a long filler context: each prompt is answered once under full retention and once under matched eviction after a shared prefill, and the same replies are scored by keyword heuristics, the HarmBench Llama-2-13B classifier, an auxiliary LLM judge, and humans on disagreements. On Qwen2.5-3B, keyword refusal falls from 98.0% to 80.5% (McNemar p~1e-8) while classifier refusal stays near ceiling (99.0%-99.5%) and MMLU accuracy is unchanged (50.0%); human labels predominantly follow the classifier, consistent with soft refusals. The gap is not universal and weakens under short fillers and paired SnapKV, so safety auditing under compression should rely on several judges matched to the serving context rather than on keyword rates alone.
Figures & tables
Figure 1: Matched evaluation protocol under KV cache compression. Prompts concatenate system instructions, a 2048-token filler, and a user query (harmful prompt for safety evaluation, or an MMLU item for task accuracy). After prefill, KV cache is kept intact (ret =1.0 ) or compressed (ret =0.3 ; primary setting uses tail eviction with sink =4 ). Each of n=200 harmful prompts is evaluated under both retentions (paired). The same generated response is scored by keyword heuristics, the HarmBench Llama-2-13B classifier (primary), an auxiliary HarmBench-style LLM judge, and human annotators on keyword–classifier disagreements ( n=56 ). Other models and compressors in Table 2 follow the same protocol.
Figure 2: Qwen2.5-3B under matched KV eviction ( n=200 ). Keyword refusal drops 17.5 pp at ret =0.3 (McNemar p≈10−8 ); HarmBench classifier is unchanged. Error bars: 95% Wilson CIs.
Metric (Qwen2.5-3B eviction)
ret =1.0
ret =0.3
MMLU math accuracy ( n=100 )
50.0
50.0
Keyword refusal ( n=200 )
98.0
80.5
HarmBench cls refusal ( n=200 )
99.0
99.5
Llama-3.2 LLM judge ( n=200 )
95.0
95.5
Table 1: Matched task vs. monitor outcomes under the primary eviction setting (filler =2048 ). MMLU covers five mathematics subjects (20 items each). The Llama-3.2 row is an auxiliary HarmBench-style LLM judge on the same Qwen generations. Rates in %.
Figure 3: Boundaries of the monitoring gap. (a) Keyword vs. classifier refusal change under compression (including a matched 4-bit sanity on Qwen2.5-3B and paired SnapKV). (b) At ret =0.3 , keyword refusal rises with prefix sink size and saturates by sink =64 ; SnapKV preserves near-ceiling keyword refusal.
Figure 4: Scale and family boundaries. Keyword refusal changes little for Qwen2.5-7B but drops substantially for Llama-3.2-3B under the reported 4-bit serving configurations.
Setting
Kw. 1.0
Kw. 0.3
Cls 1.0
Cls 0.3
Qwen2.5-3B eviction (bf16)
98.0
80.5
99.0
99.5
Qwen2.5-3B eviction (4-bit)
99.0
87.5
99.5
97.0
Qwen2.5-7B eviction (4-bit)
98.0
96.5
99.5
99.0
Qwen2.5-3B SnapKV (bf16)
98.0
98.0
99.0
99.5
Qwen2.5-3B sink =128 (bf16)
—
87.5
—
99.5
Llama-3.2-3B (4-bit)
96.0
39.0
100.0
100.0
Table 2: Refusal rates (%) on n=200 harmful prompts under matched filler contexts (primary setting: filler =2048 ). Kw.: keyword heuristic; Cls: HarmBench Llama-2-13B classifier. Rows mark precision: the 4-bit Qwen2.5-3B row is a matched sanity for the primary bf16 eviction run; Qwen2.5-7B and Llama rows are 4-bit serving boundaries, not precision-matched to bf16. SnapKV is paired (ret ∈{1.0,0.3} ); sink =128 is reported at ret =0.3 only (see Table 3 for the full sink sweep). Official HB uses the official HarmBench prompt subset under the same Qwen2.5-3B eviction protocol.
Sink
Kw. refusal
Cls refusal
4 (primary)
80.5
99.5
16
82.0
99.0
64
87.5
99.5
128
87.5
99.5
Table 3: Prefix-sink dose response on Qwen2.5-3B under eviction at ret =0.3 (filler =2048 , n=200 ). Rates in %.
Filler
Kw. Δ
p (Kw)
Cls Δ
p (Cls)
0
0.0
1.00
− 0.5
1.00
512
3.0
0.21
1.0
0.50
2048
17.5
≈10−8
− 0.5
1.00
Table 4: Filler-length ablation on Qwen2.5-3B under matched eviction ( n=200 ). Δ is the refusal-rate drop in percentage points from ret =1.0 to ret =0.3 ; p is the paired McNemar exact test.
System
Kw. Δ
p (Kw)
Cls Δ
p (Cls)
full (primary)
17.5
≈10−8
− 0.5
1.00
none
20.0
≈10−11
2.0
0.13
Table 5: System-prompt ablation on Qwen2.5-3B under matched eviction (filler =2048 , n=200 ). Δ is the refusal-rate drop in percentage points from ret =1.0 to ret =0.3 .
Figure 5: Human validation of soft refusals. On keyword–classifier disagreements ( n=56 ), human labels agree with the classifier in 94.6% of cases and with the keyword heuristic in 5.4%.
Condition
Kw. Δ
Cls Δ
Primary (Qwen2.5-3B bf16, fill. 2048)
17.5
− 0.5
Qwen2.5-3B 4-bit (matched)
11.5
2.5
SnapKV (paired, Qwen2.5-3B)
0.0
− 0.5
Filler =0 (Qwen2.5-3B)
0.0
− 0.5
Filler =512 (Qwen2.5-3B)
3.0
1.0
System = none (Qwen2.5-3B)
20.0
2.0
Table 6: Summary of paired refusal-rate drops (pp) from ret =1.0 to ret =0.3 ( n=200 unless noted). Positive Δ means refusal decreases under compression. Qwen2.5-7B and Llama-3.2-3B rows are 4-bit serving boundaries. Sink dose at ret =0.3 on Qwen2.5-3B: Kw. 80.5/82.0/87.5/87.5 for sink =4/16/64/128 (Table 3 ), reported as a ret =0.3 dose curve rather than a paired Δ .
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China