Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generation

Linze Wu; Xinrui Chen

cs.LGAug 28, 2026

Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generation

Authors: Linze Wu, Xinrui Chen

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China

Abstract

LLM agents use structured generation for JSON, SQL, and function calls. Constrained decoding ensures valid formats, but KV compression can discard history needed for correct function and argument selection. Structural responsibility can diverge from model saliency: importance scores can overlook generated tokens' structural roles. Parser transitions already computed for legality encode these roles, including required arguments and structural boundaries, providing a free retention signal. We introduce PASK (Parser-Aware Structural KV Persistence), which reuses this signal for KV persistence by layer group without online importance estimation. Task-error sensitivity sets protection floors, while attention-output distortion ranks upgrades under the remaining budget. Offline calibration produces a frozen table for structural lookup and budget accounting. PASK outperforms budget-controlled baselines on function calling and text-to-SQL, improving BFCL accuracy by up to 8.67 pp over the strongest comparator. Against Full KV, it cuts peak GPU memory by 47.5 percent and achieves up to 3.3 times TPOT speedup.

Figures & tables

Figure 1: Motivation: parser transitions expose structural roles complementing model-side saliency.

Figure 2: Overview of PASK. Parser transitions and layer groups index an offline-calibrated table that assigns high-precision retention, low-precision retention, or release to generated KV states.

Model	Method	BFCL non-live					BFCL live
Model	Method	Simple	Multiple	Parallel	Par.-multiple	Overall	Simple	Multiple	Parallel	Par.-multiple	Overall
Qwen3-4B	Full KV	95.00	95.00	90.00	84.17	91.83	78.57	74.57	60.00	78.57	75.22
	H2O \venuebox NeurIPS 2023	75.83	81.67	31.67	19.17	56.83	45.45	48.18	0.00	21.43	46.61
	SnapKV \venuebox NeurIPS 2024	85.00	87.50	42.50	35.00	67.00	55.19	59.08	0.00	14.29	56.84
	MiKV \venuebox arXiv 2024	87.50	90.83	$\underline{76.67}$	$\underline{76.67}$	$\underline{83.83}$	$\underline{64.94}$	64.45	30.00	$\mathbf{50.00}$	$\underline{63.87}$
	RocketKV \venuebox ICML 2025	75.42	65.83	33.33	25.00	55.00	48.05	47.24	0.00	$\underline{28.57}$	46.49

Table 1: BFCL accuracy (%) across models and the non-live and Live subsets. Red and orange shading indicate the best and second-best accuracy, respectively; Full KV is shown as a reference and excluded from ranking.

Figure 3: BFCL Overall accuracy versus measured relative total KV cost

B

across three models and both subsets.

Figure 4: Relative serving efficiency for Qwen3-4B at

c=8

and approximate total KV budget

B=0.33

. (a,c) TPOT speedup, defined as

\mathrm{TPOT}_{\mathrm{Full}}/\mathrm{TPOT}_{\mathrm{method}}

. (b,d) Throughput gain, defined as

\mathrm{TPS}_{\mathrm{method}}/\mathrm{TPS}_{\mathrm{Full}}

Method	KV budget $B$
Method	100%	60%	50%	40%	27.5%
H2O	56.28	41.10	35.51	32.55	24.08
SnapKV	56.28	49.83	46.77	44.24	36.56
MiKV	56.28	54.01	52.27	50.26	48.25
RocketKV	56.28	54.36	$\underline{53.66}$	50.09	41.10
CriticalKV	56.28	50.96	44.68	41.80	37.52

Table 2: Execution accuracy (EX, %) on BIRD with Qwen3-4B. Gray entries show the Full KV reference.

Method	NVIDIA L40S		NVIDIA A100
Method	Peak (MiB) $\downarrow$	Saving $\uparrow$	Peak (MiB) $\downarrow$	Saving $\uparrow$
Full KV	40986	—	41008	—
SnapKV	25090	38.78%	25086	38.83%
RocketKV	24976	39.06%	24976	39.09%
CriticalKV	24976	39.06%	24978	39.09%
PASK (Ours)	21528	47.47%	21536	47.48%

Table 3: Peak GPU memory for Qwen3-4B at a fixed logical capacity of 229,896 tokens.

Figure 5: PASK ablations. (a) Policy components: accuracy (bars) and Decode budget (line). (b) Structural components: accuracy at an aligned Decode budget.

Figure 6: Hyperparameter sensitivity on Qwen3-4B BFCL non-Live. (a) Accuracy across 28 threshold pairs; bubbles mirror the heatmap, and bold marks the selected pair. (b,c) Accuracy and Decode-budget changes (pp) for $\beta$ and $K$ , relative to the default $(0.8,8)$ .

$B$	Method	Qwen3-4B	Qwen3-14B	LLaMA-3.1-8B
$B$	Method	Live	non-Live	non-Live
0.33	Best baseline	63.87	80.17	82.83
	PASK (trans)	69.67	88.67	87.33
	PASK (recalib)	72.01	88.83	91.17
0.50	Best baseline	71.76	85.33	84.17
	PASK (trans)	73.98	90.83	89.50

Table 4: BFCL accuracy (%) under calibration transfer.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Function calling (BFCL)	Text-to-SQL (BIRD)
Class	pure_scaffold , schema_bearing , free_generation	pure_scaffold , schema_bearing , free_generation
Role	scaffold_separator , required , optional , dispatch_key	projection , relation , predicate , outside_schema
State	boundary , critical	stable , boundary , critical
Next	lookahead_critical , no_lookahead	lookahead_critical , no_lookahead

Appendix

Table 5: Task-specific vocabularies for the four structural-tag components.

Semantic label	Assignment rule
Protected content	Prompt text outside validated schema spans: instructions, requests, and chat context.
Schema scaffold	JSON/DDL structure and unmatched tokens within validated schema spans.
Function name	BFCL tool/function names; BIRD table identifiers.
Parameter key	BFCL property names and required entries; BIRD column identifiers.
Type constraint	BFCL type values; BIRD declared column types.
Enum constraint	BFCL enum values; recognized BIRD constraints, e.g., PRIMARY KEY , REFERENCES , and NOT NULL .

Appendix

Table 6: Semantic labels for prefill buckets in BFCL and BIRD.

Figure 7: PASK backend. Token–layer KV is retained in BF16/INT4 or released after recent protection. Disjoint index streams, on-demand dequantization, and shared FP32 online softmax support mixed-precision attention.

Benchmark	Subset	Cases	Test	Description
BFCL non-live (cases 1,000)	Simple	400	240	A single tool call matching one target function schema.
	Multiple	200	120	One tool call selected among several candidate functions.
	Parallel	200	120	Several independent tool calls emitted in a single response.
	Parallel-multiple	200	120	Several calls drawn from several candidate functions.
	All subsets	1,000	600	200 calibration and 200 development cases.
BFCL Live (cases 1,351)	Simple	258	154	Real-user queries with a single tool call.

Appendix

Table 7: Benchmark composition. Cases denotes the full benchmark and Test the held-out evaluation subset.

Baseline	Representative tuned knobs	Candidate values
H2O	Sink size; recent window	Sink $\in\{2,4,8\}$ ; recent window $\in\{16,32,64,128\}$ .
SnapKV	Observation window; pooling; pooling kernel	Observation window $\in\{16,32,64,128\}$ ; pooling $\in\{\mathrm{avg},\mathrm{max}\}$ ; kernel $\in\{3,5,7\}$ .
MiKV	High-precision ratio; low-bit width; group size	High-precision ratio $\in\{0.10,0.20,0.30,0.40\}$ ; low-bit width $\in\{2,4\}$ ; group size $\in\{32,64,128\}$ .
RocketKV	Observation window; pooling kernel; prefill ratio	Observation window $\in\{16,32,64,128\}$ ; kernel $\in\{31,63,127\}$ ; prefill ratio $\in\{0.20,0.25,0.30,0.35\}$ .
CriticalKV	First-stage ratio; stabilizer $\epsilon$ ; scorer window	First-stage ratio $\in\{0.25,0.40,0.50,0.70\}$ ; $\epsilon\in\{10^{-5},10^{-4},10^{-3}\}$ ; scorer window $\in\{16,32,64\}$ .
LazyEviction	Observation/recent window; attention threshold $\alpha$	Observation/recent window $\in\{8,16,32,64\}$ ; $\alpha\in\{10^{-5},10^{-4},3{\times}10^{-4},10^{-3}\}$ .

Appendix

Table 8: Representative baseline hyperparameter search spaces.

Method	B	BFCL non-live					BFCL live
Method	B	Simple	Multiple	Parallel	Par.-multiple	Overall	Simple	Multiple	Parallel	Par.-multiple	Overall
Full KV	1.00	95.00	95.00	90.00	84.17	91.83	78.57	74.57	60.00	78.57	75.22
H2O	0.33	75.83	81.67	31.67	19.17	56.83	45.45	48.18	0.00	21.43	46.61
SnapKV	0.33	85.00	87.50	42.50	35.00	67.00	55.19	59.08	0.00	14.29	56.84
MiKV	0.33	87.50	90.83	76.67	76.67	83.83	64.94	64.45	30.00	50.00	63.87
RocketKV	0.33	75.42	65.83	33.33	25.00	55.00	48.05	47.24	0.00	28.57	46.49

Appendix

Table 9: Accuracy (%) at multiple KV budgets on Qwen3-4B.

Method	B	BFCL non-live					BFCL live
Method	B	Simple	Multiple	Parallel	Par.-multiple	Overall	Simple	Multiple	Parallel	Par.-multiple	Overall
Full KV	1.00	96.25	98.33	92.50	81.67	93.00	79.87	75.20	70.00	64.29	75.83
H2O	0.33	72.08	85.00	22.50	16.67	53.67	54.55	51.82	0.00	0.00	50.80
SnapKV	0.33	82.50	91.67	60.00	55.83	74.50	61.69	66.98	0.00	14.29	64.24
MiKV	0.33	83.75	88.33	79.17	65.83	80.17	66.23	63.03	40.00	42.86	63.01
RocketKV	0.33	78.33	90.00	30.00	41.67	63.67	55.19	55.13	30.00	42.86	54.62

Appendix

Table 10: Accuracy (%) at multiple KV budgets on Qwen3-14B.

Method	B	BFCL non-live					BFCL live
Method	B	Simple	Multiple	Parallel	Par.-multiple	Overall	Simple	Multiple	Parallel	Par.-multiple	Overall
Full KV	1.00	94.58	95.00	91.67	80.83	91.33	75.32	60.98	40.00	57.14	63.38
H2O	0.33	75.42	90.00	15.83	12.50	53.83	46.75	36.65	0.00	14.29	37.73
SnapKV	0.33	87.92	94.17	43.33	35.00	69.67	51.30	53.55	10.00	35.71	52.28
MiKV	0.33	91.67	95.00	66.67	51.67	79.33	64.94	48.50	20.00	28.57	50.92
RocketKV	0.33	82.92	90.83	25.00	21.67	60.67	51.30	47.39	0.00	21.43	47.10

Appendix

Table 11: Accuracy (%) at multiple KV budgets on LLaMA-3.1-8B.

Method	Budget	Simple EX	Moderate EX	Challenging EX	Overall EX
Full KV	1.000	62.77	46.67	44.35	56.28
H2O	0.275	32.81	10.61	9.57	24.08
SnapKV	0.275	44.79	25.15	19.13	36.56
MiKV	0.275	55.63	37.27	34.78	48.25
RocketKV	0.275	50.36	29.70	17.39	41.10
CriticalKV	0.275	45.22	28.18	17.39	37.52

Appendix

Table 12: Detailed BIRD database-disjoint held-out execution accuracy (EX, %) for Qwen3-4B.

Input / Output	Method	NVIDIA A100		NVIDIA L40S
Input / Output	Method	TPOT (ms) $\downarrow$	Throughput (T/S) $\uparrow$	TPOT (ms) $\downarrow$	Throughput (T/S) $\uparrow$
8k / 256	Full KV	$\mathrm{92.73}_{\mathrm{(0.0\%)}}$	$\mathrm{79.14}_{\mathrm{(0.0\%)}}$	$\mathrm{65.09}_{\mathrm{(0.0\%)}}$	$\mathrm{107.26}_{\mathrm{(0.0\%)}}$
	SnapKV	$\mathrm{54.66}_{\mathrm{(41.1\%)}}$	$\mathrm{120.00}_{\mathrm{(51.6\%)}}$	$\mathrm{56.80}_{\mathrm{(12.7\%)}}$	$\mathrm{116.04}_{\mathrm{(8.2\%)}}$
	RocketKV	$\mathrm{81.72}_{\mathrm{(11.9\%)}}$	$\mathrm{93.04}_{\mathrm{(17.6\%)}}$	$\mathrm{61.05}_{\mathrm{(6.2\%)}}$	$\mathrm{103.41}_{\mathrm{(-3.6\%)}}$
	CriticalKV	$\mathrm{54.34}_{\mathrm{(41.4\%)}}$	$\mathrm{99.68}_{\mathrm{(26.0\%)}}$	$\mathrm{49.09}_{\mathrm{(24.6\%)}}$	$\mathrm{115.07}_{\mathrm{(7.3\%)}}$
	PASK (Ours)	$\mathbf{45.89}_{\mathrm{(50.5\%)}}$	$\mathbf{140.16}_{\mathrm{(77.1\%)}}$	$\mathbf{43.62}_{\mathrm{(33.0\%)}}$	$\mathbf{146.15}_{\mathrm{(36.3\%)}}$

Appendix

Table 13: Serving efficiency for Qwen3-4B at concurrency

c=8

and approximate total KV budget

B=0.33

across input/output lengths. Parenthetical percentages are relative to Full KV on the same GPU: positive TPOT values indicate reductions, positive throughput values indicate increases, and negative values indicate regressions.

GPU	Input	$c$	Throughput (tokens/s) $\uparrow$			TPOT (ms) $\downarrow$
GPU	Input	$c$	Full KV	PASK	SnapKV	Full KV	PASK	SnapKV
A100	32K	8	$\mathrm{33.22}_{\mathrm{(0.0\%)}}$	$\mathbf{51.02}_{\mathrm{(53.6\%)}}$	$\mathrm{42.44}_{\mathrm{(27.8\%)}}$	$\mathrm{227.52}_{\mathrm{(0.0\%)}}$	$\mathbf{68.69}_{\mathrm{(69.8\%)}}$	$\mathrm{89.09}_{\mathrm{(60.8\%)}}$
A100	32K	16	$\mathrm{33.26}_{\mathrm{(0.0\%)}}$	$\mathbf{63.23}_{\mathrm{(90.1\%)}}$	$\mathrm{51.84}_{\mathrm{(55.9\%)}}$	$\mathrm{331.35}_{\mathrm{(0.0\%)}}$	$\mathbf{68.00}_{\mathrm{(79.5\%)}}$	$\mathrm{140.50}_{\mathrm{(57.6\%)}}$
A100	32K	32	$\mathrm{33.34}_{\mathrm{(0.0\%)}}$	$\mathbf{72.23}_{\mathrm{(116.6\%)}}$	$\mathrm{51.66}_{\mathrm{(54.9\%)}}$	$\mathrm{333.99}_{\mathrm{(0.0\%)}}$	$\mathbf{68.96}_{\mathrm{(79.4\%)}}$	$\mathrm{141.81}_{\mathrm{(57.5\%)}}$
L40S	32K	8	$\mathrm{31.92}_{\mathrm{(0.0\%)}}$	$\mathbf{50.84}_{\mathrm{(59.3\%)}}$	$\mathrm{48.15}_{\mathrm{(50.8\%)}}$	$\mathrm{83.94}_{\mathrm{(0.0\%)}}$	$\mathbf{60.58}_{\mathrm{(27.8\%)}}$	$\mathrm{74.27}_{\mathrm{(11.5\%)}}$
L40S	32K	16	$\mathrm{32.02}_{\mathrm{(0.0\%)}}$	$\mathbf{60.38}_{\mathrm{(88.6\%)}}$	$\mathrm{48.29}_{\mathrm{(50.8\%)}}$	$\mathrm{85.54}_{\mathrm{(0.0\%)}}$	$\mathbf{60.74}_{\mathrm{(29.0\%)}}$	$\mathrm{73.96}_{\mathrm{(13.5\%)}}$

Appendix

Table 14: High-concurrency system results for 32K inputs at the 1024-token checkpoint. Each cell is the median over three successful repetitions. Parenthetical percentages are relative to Full KV on the same GPU: positive throughput values indicate increases and positive TPOT values indicate reductions.

Figure 8: Development-set sensitivity to (a) Decode tolerance

\alpha_{\mathrm{D}}

and (b) Prefill tolerance

\alpha_{\mathrm{P}}

. Accuracy and stage-budget changes are measured in percentage points relative to the respective defaults

5\%

and

2\%

	Runtime-V	Runtime-D
All-high control	220.4 s	260.5 s
Candidate time, median	218.8 s	222.1 s
Candidate time, p90	221.1 s	226.5 s
GPU util, mean / peak	62% / 100%
Peak memory	75.8 GiB
Full 54-candidate scan	1.64 h	1.67 h

Appendix

Table 15: Measured calibration cost on one NVIDIA A100-80GB (Qwen3-4B, BFCL non-Live calibration split, 200 cases per candidate).

Action	Model-side score	Recall @20%	Mismatch 20,50
Release $R$	$D(q,R)$	$0.0\%$	$100.0\%$
Retain-low $L$	$D(q,L)$	$0.0\%$	$83.3\%$

Appendix

Table 16: Direct comparison between effective task responsibility and Runtime-D distortion on 27 Qwen3-4B BFCL non-live calibration buckets. The effective responsibility values use the monotone envelope in Eq. 5 .

Figure 9: Responsibility–distortion mismatch across 27 calibrated structure–layer buckets. Blue circles represent release ( $R$ ), and orange diamonds represent retain-low ( $L$ ). Axes show percentile ranks of distortion $D(q,a)$ and task sensitivity $\bar{V}(q,a)$ . Red outlines highlight top-20% sensitivity buckets ranked in the bottom 50% by distortion. Top and right curves show marginal kernel density estimates.

Variant	Prefill policy	Decode policy	$B_{\mathrm{P}}^{\mathrm{meas}}$ $\downarrow$	$B_{\mathrm{D}}^{\mathrm{meas}}$ $\downarrow$	Total $B$ $\downarrow$	BFCL non-live Acc. $\uparrow$
Full KV	Full	Full	1.000	1.000	1.000	93.00
Prefill-only	PASK	Full	0.250	1.000	0.489	91.17
Decode-only	Full	PASK	1.000	0.501	0.841	90.67
Combined PASK	PASK	PASK	0.250	0.501	0.330	88.83

Appendix

Table 17: Prefill–decode stage decomposition for Qwen3-14B on BFCL non-live under the strict JSON-schema protocol.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
Sep 23, 2026Linfeng DongKV-Cache Management KV-Cache Compression
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
Sep 7, 2026Michael Wang, Keith Li, Roozbeh BostandoostLLM Inference Efficiency LLM Inference
Lake Washington School District · XSchool · UMass Amherst
BellowsKV: One Latent for Many Layers, Expanded on Demand
Oct 26, 2026Jiamu Zhang, Hanjie ChenKV-Cache Compression LLM Inference
Rice University

Sep 23, 2026J/K move · Enter open · S save

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

Linfeng Dong

KV-Cache Management KV-Cache Compression

Open paper AskPDF arXiv

As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete. A cache supports two access modes: associative lookup by content and sequential traversal by position; current compressors implement only the first. The gap matters in practice: retrieval-augmented generation, code completion, and structured-data extraction all require the model to reproduce identifiers, field values, or code tokens verbatim from the context. Under compression, content-based eviction retains the head of such a sequence but discards its continuation, causing verbatim copying to break irreversibly midway, a failure we call sequential forgetting. This failure resists better scoring, larger budgets, summary compensation, and dynamic re-scoring; it is the dominant source of remaining quality loss under compression. We propose KVFetch, a training-free, drop-in framework that opens a temporal recall channel for any score-based compressor. It demotes evicted candidates to a quantized cold tier, detects active copying through a monotone read pointer, and prefetches positional successors into fixed-size hot-tier slots without increasing attention cost. On RULER-16K under an iso-budget control, KVFetch recovers verbatim copying from 0.8 to 78.4 and raises the 13-task average by +8.4, with gains concentrating on tasks that require sequential access. On LongBench, where no task requires sequential access, the channel remains dormant and imposes no cost.

Abstract

KV-Cache Compression

Explore similar work

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

BellowsKV: One Latent for Many Layers, Expanded on Demand