CompressionBench: How can your coding agents compress LLMs?

Xinrui Zhong; Zhifeng Li; Zhiming Wang; Yongqi Liu; Shanglin Yang; Yang Zhou; Zikun Li; Zhuoming Chen

cs.AINov 4, 2026

CompressionBench: How can your coding agents compress LLMs?

Authors: Xinrui Zhong, Zhifeng Li, Zhiming Wang, Yongqi Liu, Shanglin Yang, Yang Zhou, Zikun Li, Zhuoming Chen,

Organizations: Carnegie Mellon University

Abstract

Recursive self-improvement requires AI systems to improve the models and infrastructure used by subsequent systems. LLM compression offers a concrete efficiency objective: retain a pretrained model's behavior within a smaller model storage budget. We study whether coding agents can carry out this step autonomously. We formulate compression as a terminal task in which an agent receives a source checkpoint, storage limit, inference environment, and time budget, then constructs compressed weights and executable inference code. The agent controls its algorithm and experiments; perplexity feedback guides search, and independent verification checks the final model. Across six agents and six model--size settings, 29 of 36 submissions pass verification. Selecting the lowest-perplexity agent design per task yields an average 3.4% perplexity increase over BF16 and 77.33% downstream accuracy, compared with 79.56% for BF16. Recorded experiments reveal how agents achieve these outcomes and where improvement breaks down. Agents lower PPL in 21 of 24 revisions at nearly fixed storage; complete-model feedback reverses 17 of 79 prior local rankings; and generation and inference tests expose failures that likelihood alone misses. The results identify representation design, experimental judgment, and capability validation as central to agent-driven model compression, a measurable component of efficiency-oriented self-improvement.

Figures & tables

Figure 1: Agent-driven compression outcomes. (a) Mean GSM8K, HumanEval, and TriviaQA accuracy relative to BF16 versus package bits per original parameter; values below 80% lie on the axis. (b) API cost versus within-task PPL rank; diamonds average six tasks per agent. Pareto frontiers use passing models for the accuracy–size comparison in (a).

Figure 2: Compression through terminal interaction. A task fixes the source checkpoint, storage allowance

C

, GPU and runtime, and search budget

T

. The agent constructs and tests compressed models, using reward feedback to revise weights and inference code. At the deadline, a separate evaluator checks the frozen artifact’s likelihood, generated answers, storage, and execution. Decode speed (TPS, batch size one) is reported separately from reward.

(a) Tighter storage limits
	Nemotron-3.5			Qwen3.5			Qwen3.8
	120B-A12B / 32 GiB			122B-A10B / 32 GiB			27B / 8 GiB
Method / agent	PPL $\downarrow$	Acc. $\uparrow$	TPS $\uparrow$	PPL $\downarrow$	Acc. $\uparrow$	TPS $\uparrow$	PPL $\downarrow$	Acc. $\uparrow$	TPS $\uparrow$
BF16	1.688	80.00	147.5	2.755	84.67	158.3	3.221	74.00	61.4
Released model	—	—	—	2.900	83.33	122.9	3.764	65.33	75.5
GPTQ	4.680	5.33	20.9	7.185	6.00	26.4	7.480	4.67	12.6

Table 1: Verified models under tighter (a) and looser (b) storage limits. Perplexity (PPL) is measured on held-out BF16-generated continuations; accuracy (%) averages GSM8K, HumanEval, and TriviaQA. TPS is decode tokens per second at batch size one, computed as

127/L

from the 128-token first-to-last interval

L

. Note that TPS is excluded from the PPL-based reward, and weight unpacking and dequantization can offset the benefits of reduced memory traffic. Compressed models use one H200 NVL; BF16 uses one for the dense model and two for the MoE models. Released models use Unsloth Dynamic UD-IQ2_XXS (Nemotron), Unsloth Dynamic (Qwen3.5), and GSQ-RCO (Qwen3.8). Complete measurements, including rejected artifacts, are in Table A3 .

Figure 4: Revisions within recorded searches. Thirty-five pairs from 21 runs compare candidate PPL through reward feedback. Excess cross-entropy is

E=\ln(\operatorname{PPL}/\operatorname{PPL}_{\mathrm{BF16}})

; the vertical axis is

100(E_{\mathrm{after}}/E_{\mathrm{before}}-1)

. (a) Changes by design factor, with lower-PPL/total counts. (b) The same pairs by package-size change; the band denotes changes within 0.01 GiB, and crosses mark candidates subsequently reverted.

Figure 5: Local error and model likelihood. Twelve Qwen3.8 variants use approximately 3.14 encoded bits per quantized decoder weight, with BF16 embeddings and output head. PPL is measured using BF16 reconstructions on the source continuations used for reward feedback. (a) Weight reconstruction MSE. (b) Linear-output MSE on BF16 FineWeb calibration activations. Each error is normalized by its BF16 squared norm; the dotted line denotes BF16 PPL.

Figure 6: Feedback during candidate selection. (a) Best-so-far PPL for four searches; ticks indicate whether complete-model feedback confirms or reverses a local ranking. (b) Outcomes of 301 reward evaluations across 36 runs, grouped by search time. Rewards within 0.002 of the best are treated as the same PPL. (c) The next outcome after 46 worse candidates. Reward equals ten times the source-to-compressed PPL ratio; all plotted evaluations occur during search.

Figure 7: Capability changes across candidate models. (a) Verification PPL and answer accuracies for Opus on Qwen3.8 / 8 GiB; shading marks distillation and dashed lines mark BF16. (b) Answers gained and lost across twelve successive-version pairs from four runs, plotted against PPL change. (c) HumanEval failure types for selected rejected artifacts and their BF16 references. Each benchmark has fifty questions.

Figure 8: Testing and acting on failures. MiniMax’s dense-model run (a) and GLM’s Qwen3.5 run (b) retain pruning; Opus continues dense-model distillation (c); MiniMax repairs expert indexing on Nemotron (d). Curves show feedback PPL. Diamonds show the agents’ generation tests relative to an unpruned three-bit reference (a), an eight-bit reference (b), or BF16 (c). Final markers show held-out benchmark accuracy relative to BF16.

Figure 9: From numerical design to execution. Sonnet compresses dense Qwen3.8 to 12 GiB using rotation, trellis codes, and sensitivity-based allocation (a). Opus compresses MoE Qwen3.5 to 48 GiB using rotated GPTQ and expert-specific precision (b). Rows connect the offline pipeline, component precision, inference operations, and candidate versions. Curves show search-time reward; boxes list explored alternatives that were discarded.

Source	Reward feedback PPL	Verification PPL	$A(p)$ (%)
Nemotron-3.5	1.792648	1.688011	81.33
Qwen3.5	2.768058	2.755500	85.33
Qwen3.8	3.097426	3.220743	74.00

Table A1: BF16 references fixed before agentic design. PPL is perplexity on disjoint source-generated continuations for reward feedback and verification.

A(p)

is mean accuracy (%) on GSM8K, HumanEval, and TriviaQA, used to set the accuracy gate.

Dataset	Shots	Tokens	Judgment
GSM8K	4	320	Final number
HumanEval	0	384	Unit tests
TriviaQA	5	24	Normalized aliases

Table A2: Multi-benchmark accuracy protocol. Each benchmark uses fifty held-out questions and greedy decoding. Shots gives in-context examples; Tokens gives maximum answer length. The three benchmark accuracies are averaged equally.

Figure A1 : Computation and stopping during agentic design. (a–c) Final-verification perplexity (PPL) increase over the best submission on the same task versus GPU-busy share of the run, first completion declaration as a fraction of the design budget, and tool calls divided by the task median. Values above 30% lie at the upper boundary. (d) Budget remaining at the first completion declaration for each of thirty-six runs. Filled points mark subsequent weight changes; counts give changed/total runs, and bars show medians. Agents often revise weights after declaring completion.

Method	PPL $\downarrow$	Acc. $\uparrow$	Package	Resident	TPS $\uparrow$	Gate
Nemotron-3.5 120B-A12B, 32 GiB
GPTQ	4.680	5.33	31.959	31.980	20.9	Fail
Sonnet	1.801	76.67	31.665	31.649	57.2	Pass
Opus	1.811	85.33	31.943	31.912	48.6	Pass
GLM	1.871	74.67	31.873	31.841	16.6	Pass
Kimi	2.051	58.67	31.999	31.975	12.5	Fail

Table A3 : Complete final-verification measurements by source model and storage limit. PPL scores held-out BF16-generated continuations; accuracy (%) averages GSM8K, HumanEval, and TriviaQA. Package and resident model storage are in GiB. TPS is batch-one decode speed,

127/L

for a 128-token first-to-last interval

L

. The accuracy gate requires at least 80% of the fixed BF16 reference.

32 GiB
Submission	Package (GiB)	PPL above BF16	TPS $\uparrow$ (bsz=1)
GLM: 4-bit GPTQ/Marlin	17.34	+3.09%	106.1
GLM: 4-bit GPTQ (no example)	17.34	+3.22%	106.0
Sol: optimized GPTQ	16.43	+3.38%	111.9
GLM: 5-bit, group 64	20.52	+0.60%	94.8
16 GiB

Table A4 : Pilot Qwen3.8 27B compression. PPL increase uses held-out C4 and BF16 PPL 9.9227. TPS is batch-one decode speed (

127/L

for the 128-token first-to-last interval

L

). GLM, Sol, Qwen, and MiMo denote GLM-5.3-Flash, GPT-6.1 Sol, Qwen3.8-Flash, and MiMo-V2.6-Pro, respectively.

Figure A2 : Agent responses during compression design. (a) Actions following 79 comparisons of local rankings with reward feedback in 26 of 36 runs; hatched bars denote ranking reversals, unhatched bars confirmations. Actions may co-occur. (b) Circles indicate attempted repairs; diamonds indicate matched local generation gains (eight cases, upper three rows) or execution within both storage limits (ten episodes, lower three rows).

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 294 papers

AGORA: Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents
May 26, 2026Haoran Zhang, Zhaohua SunLLM Compression LLM Agents
AI Agent Technologies (Hong Kong) Limited · Department of Mechanical Engineering, The University of Hong Kong
Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
Sep 26, 2026Ritul Satish, Prasoon Sinha, Akiho Kawada +1Context Compaction for LLM Agents Context Compression
The University of Texas at Austin
Paritok-4B: Intent-Conditioned Context Compression for Coding Agents
Aug 25, 2026Jiayu Shi, Luzhuo ChenContext Compaction for LLM Agents Context Compression
Paritok

May 26, 2026J/K move · Enter open · S save

AGORA: Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents

Haoran Zhang, Zhaohua Sun

AI Agent Technologies (Hong Kong) Limited · Department of Mechanical Engineering, The University of Hong Kong

LLM Compression LLM Agents Prompt Compression Context Compaction for LLM Agents

Open paper AskPDF arXiv

The token-level extractive compressors widely used for general LM context are structurally inappropriate for LLM agents: across 17 (env, backbone, method) cells spanning two independent token-level method families, every cell collapses to mean reward <= 0.05 despite 1.3-13.3x realized compression. We name and characterize this failure mode as action-grammar destruction -- the tokens carrying action semantics (identifiers, brackets, action verbs) are exactly those self-information ranks lowest, so a general-purpose compressor reliably removes them and the environment rejects the residual. The diagnosis points to step-granularity compression. We introduce AGORA, an inference-free step-level compressor combining a structural prompt parser, an always-keep floor for format- and recency-critical content, and a 125M-parameter relevance scorer trained on counterfactual next-action-change labels (~2ms/step, zero per-step LLM toll). Across the compared inference-free and LLM-based methods, AGORA is the only one retaining >= 75% uncompressed performance in 8 of 9 cells (with the lone exception at 73%); a four-way component ablation isolates the structural floor as the dominant quality lever and the learned scorer as the source of 1.0-11.5x adaptive end-to-end compression from a single fixed keep ratio.

Abstract

LLM Compression

Explore similar work

AGORA: Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

Paritok-4B: Intent-Conditioned Context Compression for Coding Agents