Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents

Prabal Vashisht; Zheng Hu; Ying Yang

cs.AIOct 22, 2026

Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents

Authors: Prabal Vashisht, Zheng Hu, Ying Yang

Organizations: Mx ML, DoorDash Inc.

Abstract

Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model families, and for one, correctness measurably improves.

Figures & tables

Family	Tokens (seed $\to$ winner)	Compression	$D^{\text{test}}$ $\Delta$ (95% CI)
Anthropic	36,336 $\to$ 33,567	$-7.6\%$	$+0.118$ [ $+0.073$ , $+0.164$ ]
Gemini	36,336 $\to$ 34,380	$-5.4\%$	$+0.016$ [ $-0.001$ , $+0.035$ ]
OpenAI	36,336 $\to$ 35,077	$-3.5\%$	$+0.006$ [ $-0.007$ , $+0.021$ ]

Table 1: GEPA compression results per subject-model family.

\Delta

is the paired difference (winner

-

seed) on

D^{\text{test}}

(

n=196

), mean of 5 rollouts per datapoint, with a 10,000-sample bootstrap 95% confidence interval.

Prompt Compression

Prompt compression reduces the tokens sent to language models while trying to preserve the information needed for accurate, efficient responses.

See all 35 papers

Adapting Context Compression for Long-Horizon Agents with Counterfactual Continuations
Sep 29, 2026Guanghui Min, Liang Wu, Mingjia Shi +4Counterfactual Evaluation LLM Agent Evaluation
University of Virginia · Nokia
Control Under Compression: Reliability Frontiers for Tool-Using Agents
Aug 2, 2026Yinghan Hou, Zongyou YangAI Agent Reliability AI Agent Benchmarks
Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom · Dyson School of Design Engineering Imperial College London London, United Kingdom
Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial
Mar 6, 2026Warren Johnson, Charles LeePrompt Compression Token Compression
Plexor Labs, Issaquah, WA, USA · Project Autobots, Seattle, WA, USA

Sep 29, 2026J/K move · Enter open · S save

Adapting Context Compression for Long-Horizon Agents with Counterfactual Continuations

Guanghui Min, Liang Wu, Mingjia Shi, Yinhan He+3

University of Virginia · Nokia

Counterfactual Evaluation LLM Agent Evaluation Long-Horizon LLM Agents Context Compression

Open paper AskPDF arXiv

Long-horizon agents require context compression to manage growing interaction histories. Compression quality, however, is ultimately determined by downstream execution. Existing prompt-adaptation methods infer compression errors by comparing full-context and compressed trajectories. Such comparisons cannot isolate individual compressions and are confounded by agent stochasticity. We first find that compression degrades reliability before solvability. Using matched counterfactual continuations that compare execution from the same agent state with versus without compression, we further show that severe degradation concentrates at isolated compression events. Motivated by this finding, we propose PAIR (Prompt Adaptation using Interventional Rollouts) for adapting structured compression prompts. PAIR identifies individual compressions that degrade subsequent execution, diagnoses their effects, and revises the relevant sections of a fixed compression template. PAIR achieves the strongest cross-run reliability among compressed methods in every main benchmark-scope combination, consistently exceeding the competing prompt-adaptation baseline. Without modifying the downstream agent, PAIR brings compressed execution close to the no-compression baseline and sometimes numerically exceeds it.

Abstract

Prompt Compression

Explore similar work

Adapting Context Compression for Long-Horizon Agents with Counterfactual Continuations

Control Under Compression: Reliability Frontiers for Tool-Using Agents

Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial