Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial

Warren Johnson; Charles Lee

cs.CLMar 6, 2026

Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial

Authors: Warren Johnson, Charles Lee

Organizations: Plexor Labs, Issaquah, WA, USA · Project Autobots, Seattle, WA, USA

Abstract

The economics of prompt compression depend not only on reducing input tokens but on how compression changes output length, which is typically priced several times higher. We evaluate this in a pre-registered six-arm randomized controlled trial of prompt compression on production multi-agent task-orchestration, analyzing 358 successful Claude Sonnet 4.5 runs (59-61 per arm) drawn from a randomized corpus of 1,199 real orchestration instructions. We compare an uncompressed control with three uniform retention rates (r=0.8, 0.5, 0.2) and two structure-aware strategies (entropy-adaptive and recency-weighted), measuring total inference cost (input+output) and embedding-based response similarity. Moderate compression (r=0.5) reduced mean total cost by 27.9%, while aggressive compression (r=0.2) increased mean cost by 1.8% despite substantial input reduction, consistent with small mean output expansion (1.03x vs. control) and heavy-tailed uncertainty. Recency-weighted compression achieved 23.5% savings and, together with moderate compression, occupied the empirical cost-similarity Pareto frontier, whereas aggressive compression was dominated on both cost and similarity. These results show that "compress more" is not a reliable production heuristic and that output tokens must be treated as a first-class outcome when designing compression policies.

Figures & tables

Paper	Year	Output Measured	Total Cost	Gap
LLMLingua ( Jiang et al., 2023 )	2023	No	No	Yes
LLMLingua-2 ( Pan et al., 2024 )	2024	No	No	Yes
FrugalGPT ( Chen et al., 2023 )	2023	Formula only	No	Yes
Selective Context ( Li et al., 2023 )	2023	No	No	Yes
RECOMP ( Xu et al., 2023 )	2023	No	No	Yes
Gist Tokens ( Mu et al., 2023 )	2023	No	No	Yes

Table 1: Output token dynamics gap in the prompt compression literature. The overwhelming majority of compression research measures only input token reduction, ignoring output tokens that cost 3–5

\times

more per token. This systematic omission undermines cost-effectiveness claims.

Task Type	Count	%	Mean Length (chars)	Est. Tokens
Implementation	707	59.0	895	$\sim$ 224
Breakdown	159	13.3	133	$\sim$ 33
Validation	153	12.8	709	$\sim$ 177
Post-orchestration	78	6.5	430	$\sim$ 108
Review	68	5.7	669	$\sim$ 167
Execution	23	1.9	702	$\sim$ 176

Table 2: Corpus composition by task type and length tercile (

N=1{,}199

Arm	Mean In Tok	Mean Out Tok	Mean Cost ($)	Total Cost ($)	Latency (ms)
Control	59	609	0.0093	0.0466	13,122
Light	49	811	0.0123	0.0616	16,285
Moderate	41	613	0.0093	0.0466	13,671
Aggressive	30	161	0.0025	0.0125	4,124
Adaptive	46	420	0.0064	0.0322	9,370
Recency	41	504	0.0077	0.0384	11,583

Table 3: Pilot run summary statistics (

n=30

; 5 trials per arm).

Panel of Figure 1 — Figure 1: CONSORT flow diagram. Enrollment: 1,577 records $\rightarrow$ 1,337 after exclusions $\rightarrow$ 1,199 after deduplication and randomization. All 1,199 trials were submitted to the API; 841 failed after retries (primarily credit-balance errors), leaving a complete-case inferential set of 358 successful trials (59–61 per arm).

Metric	Full Randomized ( $N=1{,}199$ )	Complete-Case ( $N=358$ )
Mean input tokens (original)	179.6	76.0
Median input tokens (original)	126.0	78.5
Implementation tasks, $n$ (%)	707 (59.0%)	176 (49.2%)
Breakdown tasks, $n$ (%)	159 (13.3%)	159 (44.4%)
Execution tasks, $n$ (%)	23 (1.9%)	23 (6.4%)
All other task types, $n$ (%)	310 (25.9%)	0 (0.0%)

Table 4: Full randomized set vs complete-case successful-response set.

Arm	Assigned	Successful	Mean Cost ($)	Successes/$	Cost Reduction vs Control
Control	197	59	0.004682	64.0	—
Light	199	60	0.003992	75.5	14.7%
Moderate	201	60	0.003256	91.7	30.5%
Aggressive	202	61	0.004419	68.3	5.6%
Adaptive	199	59	0.003878	76.4	17.2%
Recency	201	59	0.003389	86.6	27.6%

Table 5: Assignment-level deployment sensitivity across all randomized submissions.

Arm	$n$	Mean In	Mean Out	Mean Cost ($)	Savings
Control ( $r=1.0$ )	59	107	916	0.0141	—
Light ( $r=0.8$ )	60	88	788	0.0121	$-14.1\%$
Moderate ( $r=0.5$ )	60	61	664	0.0101	$-27.9\%^{***}$
Aggressive ( $r=0.2$ )	61	40	946	0.0143	$+1.8\%$
Adaptive	59	74	786	0.0120	$-14.5\%$
Recency	59	65	704	0.0108	$-23.5\%^{*}$

Table 6: Arm-level summary statistics for primary outcomes (

N=358

trials).

Arm	n	Mean Similarity (SD)	% Preserved ( $\geq 0.85$ )	$d$
Light ( $r=0.8$ )	60	$0.764^{***}$ (0.082)	15.0%	1.23
Adaptive	59	$0.758^{***}$ (0.088)	15.3%	1.15
Recency	59	$0.728^{***}$ (0.100)	11.9%	0.85
Moderate ( $r=0.5$ )	60	$0.724^{***}$ (0.092)	13.3%	0.85
Aggressive ( $r=0.2$ )	61	0.623 (0.138)	3.3%	—
Overall	299	0.719 (0.114)	11.7%	—

Table 7: Embedding response-similarity proxy by compression arm (cosine similarity).

Arm	Mean Cost	Savings	Similarity	Preserved	Pareto?
Control	$0.0141	—	1.000	100%	Baseline
Light	$0.0121	$-14.1\%$	0.764	15.0%	No
Moderate	$0.0101	$-27.9\%$	0.724	13.3%	Yes
Aggressive	$0.0143	$+1.8\%$	0.623	3.3%	No
Adaptive	$0.0120	$-14.9\%$	0.758	15.3%	No
Recency	$0.0108	$-23.5\%$	0.727	11.9%	Yes

Table 8: Cost–similarity Pareto analysis by arm.

Metric	Articles 1–5	Article 6 (Production)	Alignment
Code cliff threshold	$r=0.55$	$r\in[0.2,0.5]$	Directional only (coarse grid)
CoT floor threshold	$r=0.70$	Not directly testable in current analysis set	—
Output explosion (aggressive)	38 $\times$ (DeepSeek)	1.03 $\times$ (Claude)	Model-dependent
Output expansion (light)	Present on some benchmarks	0.86 $\times$ output; $-14.1\%$ cost	Not replicated
Optimal compression ratio	$r=0.5$ – $0.6$	$r=0.5$	Consistent

Table 9: Cross-study comparison of compression thresholds and effects.

Prompt Compression

Prompt compression reduces the tokens sent to language models while trying to preserve the information needed for accurate, efficient responses.

See all 35 papers

Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Mar 6, 2026Warren JohnsonPrompt Compression LLM Inference Efficiency
Plexor Labs Principal Researcher
Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
Sep 26, 2026Ritul Satish, Prasoon Sinha, Akiho Kawada +1Context Compaction for LLM Agents Context Compression
The University of Texas at Austin
Token Reduction Is Not Cost Reduction
Jul 13, 2026Sarel Weinberger, Amir HozezAI Coding Agents Cost-Aware Inference
PointFive

Mar 6, 2026J/K move · Enter open · S save

Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

Warren Johnson

Plexor Labs Principal Researcher

Prompt Compression LLM Inference Efficiency Efficient Language Model Inference LLM Evaluation

Open paper AskPDF arXiv

Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substantially lower expansion on HumanEval (5x, Psi approx 0.72), while GPT-4o-mini is comparatively stable across benchmarks. This reconciles the apparent discrepancy between previously reported extreme explosion and lower replication effects by identifying prompt structure, not provider identity alone, as the primary moderator. We introduce the Compression Robustness Index (CRI) for cross-benchmark evaluation and show that single-benchmark assessments can produce misleading conclusions about compression safety and efficiency. To contextualize energy claims, we incorporate companion direct NVML measurements from rented RunPod GPUs and show that token savings can overstate joule savings. These findings motivate benchmark-diverse testing and structure-aware compression policies for reliable, energy-conscious LLM deployment.

Abstract

Prompt Compression

Explore similar work

Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents

Token Reduction Is Not Cost Reduction