Large Language Model (LLM) agents are increasingly deployed under structured generation, where grammar constraints enforce output format at inference time. Whereas existing compression methods target unconstrained generation, we first identify a fundamental mismatch: under structured generation, grammar constraints reduce the model's output distribution to valid tokens, and the logits matter most at the small set of positions that decide the agent's action. Motivated by this finding, we propose SKIM (Selective Knowledge-Informed Masking), the first pruning and distillation framework for LLM agents under structured generation, with both stages guided by token role. SKIM partitions token positions into context, decision, and format categories, and this partition drives two complementary axes: (1) category-aware pruning saliency scores that preserve parameters most relevant for decision positions; and (2) an on-policy distillation objective that aligns student and teacher on valid tokens, paired with category-aware feature matching. We evaluate SKIM on tool-calling benchmarks and within multi-step agent loops, across multiple model scales and sparsity levels. SKIM consistently improves accuracy over baselines, with its superiority becoming most striking in high-sparsity regimes where conventional methods struggle, demonstrating the potential of category-aware compression for deployment-time grammar-constrained agents.
Figures & tables
Figure 1: Target behavior.
Figure 2: Token categories and error decomposition under structured generation. Left: Example function call with tokens highlighted by category ( Context , Decision , Format ) and three response types distinguishing format errors (eliminated by structured decoding) from non-format errors (which require the model itself to be correct). Right: Pruned-then-distilled model performance on Qwen3.5-2B [ 33 ] and BFCL-v3 [ 31 ] at 25% sparsity. SKIM fully exploits the grammar formatter: without it, SKIM looks broken because format errors dominate, but with it, SKIM easily beats Base (Wanda-sp). Format errors are essentially free under structured decoding, so reducing non-format errors at decision positions is what matters.
BFCL-v3
API-Bank L1
Method / Sparsity
10%
25%
50%
10%
25%
50%
0.8B
Dense
78.0 / 72.6
57.1 / 47.6
Wanda-sp
72.9 / 60.2
53.6 / 30.4
7.5 / 0.0
51.9 / 50.1
35.6 / 29.3
4.0 / 0.0
FLAP
74.9 / 47.0
63.9 / 57.0
42.6 / 34.0
51.9 / 46.8
45.8 / 43.5
33.5 / 21.0
SlimGPT
74.5 / 57.0
59.4 / 48.1
35.9 / 13.4
55.4 / 49.9
44.1 / 39.6
28.3 / 9.5
SKIM (ours)
76.4 / 0.0
71.1 / 0.0
53.4 / 1.5
52.1 / 0.0
46.9 / 0.3
39.3 / 0.3
Table 1: Post-KD full-match accuracy of Qwen3.5 series across varying sparsity ratios with thinking mode enabled. All reported numbers are measured under the same distillation budget. Each cell reports structured-generation / free-generation accuracy. SKIM achieves the best structured-generation accuracy in most settings, with the gap to baselines widening at higher sparsity.
Figure 3: BFCL-v3 math example on baseline methods and SKIM. SKIM-free’s format errors are intentional: the freed capacity lets SKIM-structured produce the correct call. Baselines fail with wrong arguments or wrong tools, errors grammar cannot fix. See Appendix D for more examples.
BFCL-v3
API-Bank L1
Method / Sparsity
10%
25%
50%
10%
25%
50%
3B
Dense
89.4 / 79.9
76.2 / 75.9
Wanda-sp
88.6 / 87.4
83.8 / 81.0
72.2 / 65.5
76.9 / 77.2
73.4 / 73.9
55.4 / 54.1
FLAP
89.9 / 88.6
88.0 / 86.1
73.0 / 68.6
75.9 / 74.9
73.9 / 70.7
47.3 / 30.1
SlimGPT
89.4 / 87.6
85.1 / 82.6
46.4 / 36.1
75.9 / 77.4
72.2 / 71.4
39.1 / 30.8
SKIM (ours)
88.8 / 83.0
88.1 / 80.4
82.8 / 69.9
76.2 / 70.9
73.4 / 66.9
67.7 / 14.0
Table 2: Post-KD full-match accuracy of Hammer2.1-3B at various sparsity ratios with thinking mode enabled. Each cell reports structured-generation / free-generation accuracy. On the tool-specialist model, which is already fine-tuned for function calling, SKIM still achieves the best structured-generation accuracy at 50% sparsity on both benchmarks.
Table 6
Figure 4: Format-token weight ablation on Qwen3.5-2B (25% sparsity) with API-Bank L1. Higher αf trades structured accuracy for free-generation accuracy, as intended.
Variant
Structured
Free
SKIM (full)
80.8
0.0
( − ) w/o on-policy.
78.1
0.0
( − ) w/o feature distill.
79.1
0.5
( + ) w/ vanilla KD.
77.9
46.4
Table 5: Loss ablation on Qwen3.5-2B (25% sparsity) with BFCL-v3. Each variant degrades structured accuracy; the full SKIM is the best.
Table 9
Model
Decoding
Prefill
Qwen3.5-2B
1.29 ×
1.23 ×
Qwen3.5-9B
1.41 ×
1.37 ×
Qwen3.5-27B
1.59 ×
1.51 ×
Table 8: Speedups of SKIM over the dense model at 50% nominal sparsity (decoding throughput at batch size 256 and prefill speed for 4,096 tokens).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Surrogate precision
BFCL-v3
API-Bank
Exponential ( Eq. 19 )
80.3
63.7
Additive ( Eq. 18 )
88.1
72.9
Appendix
Table 9: Surrogate precision design comparison on Hammer-3B at 20% sparsity. Numbers are tool-use accuracy (%) after pruning and distillation under matched compute.
Figure 5: Pruned-then-distilled model performance averaged over three baselines, two benchmarks, and Qwen3.5-0.8B/2B ( § 3.2 ). Pruned baseline error grows with sparsity, but non-format error dominates: capacity is misallocated to format behavior the constraint mechanism enforces for free.
Figure 6: Task accuracy under category-specialized MLP ablation on BFCL-v3 (left) and API-Bank L1 (right) on Hammer2.1-3B model. context , decision , and format ablate MLP neurons whose activation budget on the calibration set concentrates ( sic≥0.6 ) on the corresponding token category; random ablates the same number of MLP neurons uniformly at random; dense is the unpruned baseline. Format-specialized ablation barely affects accuracy, while context- or decision-specialized ablation causes sharp drops in accuracy relative to the dense baseline.
Figure 7: Aggregate fraction of MLP neurons specialized for each token category on Hammer2.1-3B and Qwen3.5-2B. context , decision , format correspond to context, decision, and format positions respectively; mixed denotes neurons that do not satisfy the θ=0.6 threshold for any single category.
Figure 8: Layer-wise fraction of MLP neurons assigned to each token category by their largest selectivity score on Hammer-2.1-3B (top) and Qwen3.5-2B (bottom); unlike Fig. 7 , every neuron is assigned to one category. At most layers, Hammer-2.1-3B has the larger decision fraction.
Params
Whole-model
GPU memory
Peak GPU
sparsity
at load
memory
Dense
1.88 B
–
7.010 GiB
7.358 GiB
SKIM-10%
1.75 B
6.77%
6.792 GiB
7.130 GiB
SKIM-25%
1.56 B
17.07%
6.420 GiB
6.754 GiB
SKIM-50%
1.24 B
34.14%
5.821 GiB
6.151 GiB
Appendix
Table 12: Memory footprint of Qwen3.5-2B under SKIM, measured on physically compacted models. At load is the GPU memory after weight loading; peak is the maximum GPU memory observed at any point during one representative inference run.
Model
Parameters
Decode, batch 1
Decode, batch 256
Prefill, 4,096 tokens
Qwen3.5-2B
1.88B → 1.24B
1.03 ×
1.29 ×
1.23 ×
Qwen3.5-9B
8.95B → 6.06B
1.13 ×
1.41 ×
1.37 ×
Qwen3.5-27B
26.9B → 16.7B
1.17 ×
1.59 ×
1.51 ×
Appendix
Table 13: Speedups over the dense model at 50% nominal sparsity.
Figure 9: Function-call format example and <IMPORTANT> reminder block elided from the per-case figures. The block is token-for-token identical across all tool-calling cases.
Figure 10: BFCL-v3 success case. SKIM under free decoding collapses into a malformed loop that mixes tool-call fields with argument values; XGrammar restores the exact parallel call set. Wanda-SP at equal density emits one call with a corrupted function name and a truncated protein_name .
Figure 11: BFCL-v3 failure case. Grammar enforcement repairs the structure (well-formed JSON with the right function name and argument keys), but the model writes 3x^2 where the gold uses 3x**2 , so the case fails exact match despite format validity. Free decoding loops; Wanda-SP corrupts the polynomial expression in the generated call.
Figure 12: API-Bank L1 success case. The multi-turn dialog requires the assistant to authenticate before modifying the password; SKIM with XGrammar emits the correct GetUserToken call. Free decoding loses the function-name slot; Wanda-SP places the username in the name field.
Figure 13: API-Bank L1 failure case. The grammar-constrained decoder produces a perfectly valid call to DeleteAccount with a placeholder token ; the user is still in the authentication step and the expected call is GetUserToken . The format is right, the tool is wrong.
Figure 14: HotpotQA success case. The retrieval trajectory is noisy (the model loops through Open[1] and Open[3] after already having both birth years), but SKIM converges to the correct span; EM = 1.0, F1 = 1.0. Wanda-SP returns the wrong person.
Figure 15: HotpotQA failure case. The evidence chain is correct and the model’s own reasoning quotes the gold span verbatim, but the final answer drops the leading preposition “between”; the answer scores EM = 0.0 and F1 = 0.9333 against the gold span.
Large language model (LLM) agents typically append raw tool outputs to their context. Over multi-step interactions, long and noisy observations incur substantial context overhead and dilute the agent's attention to task-relevant information. Existing token pruning and summarization methods remain constrained by discrete text representations, limiting the achievable compression ratios. In this work, we challenge the prevailing assumption that agents must consume tool observations as explicit text tokens. To this end, we propose Coca-Cola, a continuous latent context compression model for LLM agents. Coca-Cola compresses tool observations into latent tokens and embeds them into the input embedding space of the agent. We further introduce Action-Preserving Training, which aligns the agent's next-action distributions under full and compressed observations. This trains the compression model to learn action-sufficient latent tokens that retain decision-critical information. Experiments with web-search agents on four QA benchmarks and two model families show that Coca-Cola reduces the agent's token consumption by 70.01%-89.24% and end-to-end runtime by 24.45%-55.97%, while consistently improving task performance. These results suggest that Coca-Cola offers a promising approach for context compression in agents. Our code is available at https://github.com/THU-Agent/Coca-cola.
The token-level extractive compressors widely used for general LM context are structurally inappropriate for LLM agents: across 17 (env, backbone, method) cells spanning two independent token-level method families, every cell collapses to mean reward <= 0.05 despite 1.3-13.3x realized compression. We name and characterize this failure mode as action-grammar destruction -- the tokens carrying action semantics (identifiers, brackets, action verbs) are exactly those self-information ranks lowest, so a general-purpose compressor reliably removes them and the environment rejects the residual. The diagnosis points to step-granularity compression. We introduce AGORA, an inference-free step-level compressor combining a structural prompt parser, an always-keep floor for format- and recency-critical content, and a 125M-parameter relevance scorer trained on counterfactual next-action-change labels (~2ms/step, zero per-step LLM toll). Across the compared inference-free and LLM-based methods, AGORA is the only one retaining >= 75% uncompressed performance in 8 of 9 cells (with the lone exception at 73%); a four-way component ablation isolates the structural floor as the dominant quality lever and the learned scorer as the source of 1.0-11.5x adaptive end-to-end compression from a single fixed keep ratio.
Haoran Zhang, Zhaohua Sun
AI Agent Technologies (Hong Kong) Limited · Department of Mechanical Engineering, The University of Hong Kong
As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.