cs.CLNov 5, 2026

SKIM: Pruning Large Language Model Agents via Selective Knowledge Informed Masking

Authors: Moonseok Choi, Jongwon Jeong, Minki Kang, Giung Nam, Juho Lee

Organizations: KAIST · University of Wisconsin–Madison

Abstract

Large Language Model (LLM) agents are increasingly deployed under structured generation, where grammar constraints enforce output format at inference time. Whereas existing compression methods target unconstrained generation, we first identify a fundamental mismatch: under structured generation, grammar constraints reduce the model's output distribution to valid tokens, and the logits matter most at the small set of positions that decide the agent's action. Motivated by this finding, we propose SKIM (Selective Knowledge-Informed Masking), the first pruning and distillation framework for LLM agents under structured generation, with both stages guided by token role. SKIM partitions token positions into context, decision, and format categories, and this partition drives two complementary axes: (1) category-aware pruning saliency scores that preserve parameters most relevant for decision positions; and (2) an on-policy distillation objective that aligns student and teacher on valid tokens, paired with category-aware feature matching. We evaluate SKIM on tool-calling benchmarks and within multi-step agent loops, across multiple model scales and sparsity levels. SKIM consistently improves accuracy over baselines, with its superiority becoming most striking in high-sparsity regimes where conventional methods struggle, demonstrating the potential of category-aware compression for deployment-time grammar-constrained agents.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Tools Need Not Speak in Text: Latent Context Compression for Agents

    Nov 4, 2026Yuetong Liu, Zhehao Zhou, Yitong Zhang +2Context Compaction for LLM AgentsLLM Context Compression

  2. AGORA: Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents

    May 26, 2026Haoran Zhang, Zhaohua SunLLM CompressionLLM Agents

  3. Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

    Sep 26, 2026Ritul Satish, Prasoon Sinha, Akiho Kawada +1Context Compaction for LLM AgentsContext Compression