Recursive self-improvement requires AI systems to improve the models and infrastructure used by subsequent systems. LLM compression offers a concrete efficiency objective: retain a pretrained model's behavior within a smaller model storage budget. We study whether coding agents can carry out this step autonomously. We formulate compression as a terminal task in which an agent receives a source checkpoint, storage limit, inference environment, and time budget, then constructs compressed weights and executable inference code. The agent controls its algorithm and experiments; perplexity feedback guides search, and independent verification checks the final model. Across six agents and six model--size settings, 29 of 36 submissions pass verification. Selecting the lowest-perplexity agent design per task yields an average 3.4% perplexity increase over BF16 and 77.33% downstream accuracy, compared with 79.56% for BF16. Recorded experiments reveal how agents achieve these outcomes and where improvement breaks down. Agents lower PPL in 21 of 24 revisions at nearly fixed storage; complete-model feedback reverses 17 of 79 prior local rankings; and generation and inference tests expose failures that likelihood alone misses. The results identify representation design, experimental judgment, and capability validation as central to agent-driven model compression, a measurable component of efficiency-oriented self-improvement.
Figures & tables
Figure 1: Agent-driven compression outcomes. (a) Mean GSM8K, HumanEval, and TriviaQA accuracy relative to BF16 versus package bits per original parameter; values below 80% lie on the axis. (b) API cost versus within-task PPL rank; diamonds average six tasks per agent. Pareto frontiers use passing models for the accuracy–size comparison in (a).
Figure 2: Compression through terminal interaction. A task fixes the source checkpoint, storage allowance C , GPU and runtime, and search budget T . The agent constructs and tests compressed models, using reward feedback to revise weights and inference code. At the deadline, a separate evaluator checks the frozen artifact’s likelihood, generated answers, storage, and execution. Decode speed (TPS, batch size one) is reported separately from reward.
(a) Tighter storage limits
Nemotron-3.5
Qwen3.5
Qwen3.8
120B-A12B / 32 GiB
122B-A10B / 32 GiB
27B / 8 GiB
Method / agent
PPL ↓
Acc. ↑
TPS ↑
PPL ↓
Acc. ↑
TPS ↑
PPL ↓
Acc. ↑
TPS ↑
BF16
1.688
80.00
147.5
2.755
84.67
158.3
3.221
74.00
61.4
Released model
—
—
—
2.900
83.33
122.9
3.764
65.33
75.5
GPTQ
4.680
5.33
20.9
7.185
6.00
26.4
7.480
4.67
12.6
Table 1: Verified models under tighter (a) and looser (b) storage limits. Perplexity (PPL) is measured on held-out BF16-generated continuations; accuracy (%) averages GSM8K, HumanEval, and TriviaQA. TPS is decode tokens per second at batch size one, computed as 127/L from the 128-token first-to-last interval L . Note that TPS is excluded from the PPL-based reward, and weight unpacking and dequantization can offset the benefits of reduced memory traffic. Compressed models use one H200 NVL; BF16 uses one for the dense model and two for the MoE models. Released models use Unsloth Dynamic UD-IQ2_XXS (Nemotron), Unsloth Dynamic (Qwen3.5), and GSQ-RCO (Qwen3.8). Complete measurements, including rejected artifacts, are in Table A3 .
Figure 4: Revisions within recorded searches. Thirty-five pairs from 21 runs compare candidate PPL through reward feedback. Excess cross-entropy is E=ln(PPL/PPLBF16) ; the vertical axis is 100(Eafter/Ebefore−1) . (a) Changes by design factor, with lower-PPL/total counts. (b) The same pairs by package-size change; the band denotes changes within 0.01 GiB, and crosses mark candidates subsequently reverted.
Figure 5: Local error and model likelihood. Twelve Qwen3.8 variants use approximately 3.14 encoded bits per quantized decoder weight, with BF16 embeddings and output head. PPL is measured using BF16 reconstructions on the source continuations used for reward feedback. (a) Weight reconstruction MSE. (b) Linear-output MSE on BF16 FineWeb calibration activations. Each error is normalized by its BF16 squared norm; the dotted line denotes BF16 PPL.
Figure 6: Feedback during candidate selection. (a) Best-so-far PPL for four searches; ticks indicate whether complete-model feedback confirms or reverses a local ranking. (b) Outcomes of 301 reward evaluations across 36 runs, grouped by search time. Rewards within 0.002 of the best are treated as the same PPL. (c) The next outcome after 46 worse candidates. Reward equals ten times the source-to-compressed PPL ratio; all plotted evaluations occur during search.
Figure 7: Capability changes across candidate models. (a) Verification PPL and answer accuracies for Opus on Qwen3.8 / 8 GiB; shading marks distillation and dashed lines mark BF16. (b) Answers gained and lost across twelve successive-version pairs from four runs, plotted against PPL change. (c) HumanEval failure types for selected rejected artifacts and their BF16 references. Each benchmark has fifty questions.
Figure 8: Testing and acting on failures. MiniMax’s dense-model run (a) and GLM’s Qwen3.5 run (b) retain pruning; Opus continues dense-model distillation (c); MiniMax repairs expert indexing on Nemotron (d). Curves show feedback PPL. Diamonds show the agents’ generation tests relative to an unpruned three-bit reference (a), an eight-bit reference (b), or BF16 (c). Final markers show held-out benchmark accuracy relative to BF16.
Figure 9: From numerical design to execution. Sonnet compresses dense Qwen3.8 to 12 GiB using rotation, trellis codes, and sensitivity-based allocation (a). Opus compresses MoE Qwen3.5 to 48 GiB using rotated GPTQ and expert-specific precision (b). Rows connect the offline pipeline, component precision, inference operations, and candidate versions. Curves show search-time reward; boxes list explored alternatives that were discarded.
Source
Reward feedback PPL
Verification PPL
A(p) (%)
Nemotron-3.5
1.792648
1.688011
81.33
Qwen3.5
2.768058
2.755500
85.33
Qwen3.8
3.097426
3.220743
74.00
Table A1: BF16 references fixed before agentic design. PPL is perplexity on disjoint source-generated continuations for reward feedback and verification. A(p) is mean accuracy (%) on GSM8K, HumanEval, and TriviaQA, used to set the accuracy gate.
Dataset
Shots
Tokens
Judgment
GSM8K
4
320
Final number
HumanEval
0
384
Unit tests
TriviaQA
5
24
Normalized aliases
Table A2: Multi-benchmark accuracy protocol. Each benchmark uses fifty held-out questions and greedy decoding. Shots gives in-context examples; Tokens gives maximum answer length. The three benchmark accuracies are averaged equally.
Figure A1 : Computation and stopping during agentic design. (a–c) Final-verification perplexity (PPL) increase over the best submission on the same task versus GPU-busy share of the run, first completion declaration as a fraction of the design budget, and tool calls divided by the task median. Values above 30% lie at the upper boundary. (d) Budget remaining at the first completion declaration for each of thirty-six runs. Filled points mark subsequent weight changes; counts give changed/total runs, and bars show medians. Agents often revise weights after declaring completion.
Method
PPL ↓
Acc. ↑
Package
Resident
TPS ↑
Gate
Nemotron-3.5 120B-A12B, 32 GiB
GPTQ
4.680
5.33
31.959
31.980
20.9
Fail
Sonnet
1.801
76.67
31.665
31.649
57.2
Pass
Opus
1.811
85.33
31.943
31.912
48.6
Pass
GLM
1.871
74.67
31.873
31.841
16.6
Pass
Kimi
2.051
58.67
31.999
31.975
12.5
Fail
Table A3 : Complete final-verification measurements by source model and storage limit. PPL scores held-out BF16-generated continuations; accuracy (%) averages GSM8K, HumanEval, and TriviaQA. Package and resident model storage are in GiB. TPS is batch-one decode speed, 127/L for a 128-token first-to-last interval L . The accuracy gate requires at least 80% of the fixed BF16 reference.
Submission
Package (GiB)
PPL above BF16
TPS ↑ (bsz=1)
32 GiB
GLM: 4-bit GPTQ/Marlin
17.34
+3.09%
106.1
GLM: 4-bit GPTQ (no example)
17.34
+3.22%
106.0
Sol: optimized GPTQ
16.43
+3.38%
111.9
GLM: 5-bit, group 64
20.52
+0.60%
94.8
16 GiB
Table A4 : Pilot Qwen3.8 27B compression. PPL increase uses held-out C4 and BF16 PPL 9.9227. TPS is batch-one decode speed ( 127/L for the 128-token first-to-last interval L ). GLM, Sol, Qwen, and MiMo denote GLM-5.3-Flash, GPT-6.1 Sol, Qwen3.8-Flash, and MiMo-V2.6-Pro, respectively.
Figure A2 : Agent responses during compression design. (a) Actions following 79 comparisons of local rankings with reward feedback in 26 of 36 runs; hatched bars denote ranking reversals, unhatched bars confirmations. Actions may co-occur. (b) Circles indicate attempted repairs; diamonds indicate matched local generation gains (eight cases, upper three rows) or execution within both storage limits (ten episodes, lower three rows).
The token-level extractive compressors widely used for general LM context are structurally inappropriate for LLM agents: across 17 (env, backbone, method) cells spanning two independent token-level method families, every cell collapses to mean reward <= 0.05 despite 1.3-13.3x realized compression. We name and characterize this failure mode as action-grammar destruction -- the tokens carrying action semantics (identifiers, brackets, action verbs) are exactly those self-information ranks lowest, so a general-purpose compressor reliably removes them and the environment rejects the residual. The diagnosis points to step-granularity compression. We introduce AGORA, an inference-free step-level compressor combining a structural prompt parser, an always-keep floor for format- and recency-critical content, and a 125M-parameter relevance scorer trained on counterfactual next-action-change labels (~2ms/step, zero per-step LLM toll). Across the compared inference-free and LLM-based methods, AGORA is the only one retaining >= 75% uncompressed performance in 8 of 9 cells (with the lone exception at 73%); a four-way component ablation isolates the structural floor as the dominant quality lever and the learned scorer as the source of 1.0-11.5x adaptive end-to-end compression from a single fixed keep ratio.
Haoran Zhang, Zhaohua Sun
AI Agent Technologies (Hong Kong) Limited · Department of Mechanical Engineering, The University of Hong Kong
As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.
Coding agents re-send large file reads and tool outputs to a frontier LLM every turn, and this context dominates their token bill. General-purpose prompt compressors are trained on prose and suit code poorly: they paraphrase identifiers and drop the exact spans an agent needs to edit. We present Paritok-4B, a 4B LoRA compressor for coding-agent trajectories built on two commitments. It is extractive: it selects spans rather than rewriting them, and 96.0% of the identifiers, paths, and numbers it emits already appear in its input, holding at 96.2% on held-out SWE-bench Lite output. It is intent-conditioned: told the agent's current task, it acts chiefly inside a retained segment, selecting which lines survive (retained lines are +0.067 more intent-relevant than removed ones, paired 95% CI [+0.056, +0.078]) rather than changing how much is retained. We distil a gpt-4.1-mini teacher over 67,074 real OpenHands trajectories into 40,606 validated examples and fine-tune Qwen3-4B. On all 300 SWE-bench Lite instances, Paritok-4B compresses agent context to 25.7% of its size, 2.0x harder than a gpt-4.1-mini compressor (50.2%) and 2.4x harder than gpt-5 (61.9%), while retaining 86.5% of uncompressed single-shot solve quality. Fed the cat -n line-numbered input real agents produce, it compresses slightly less (27.8%) and retains more (89.3%); there the paired test is informative, with 30 instances solved only uncompressed and 17 only compressed, an exact McNemar p=0.079, so at this sample size compressing context to roughly a quarter of its size does not significantly reduce the solve rate. The model is a 264 MB adapter that self-hosts on one 24 GB GPU with no per-token compressor fee, which at list prices decides the economics: gpt-5 as a compressor is net-negative, costing more than the downstream tokens it saves. Weights, data, and evaluation scripts are open (Apache 2.0).