ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

Haizhong Zheng; Yizhuo Di; Ranajoy Sadhukhan; Shuowei Jin; Beidi Chen

cs.LGOct 6, 2026

ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

Authors: Haizhong Zheng, Yizhuo Di, Ranajoy Sadhukhan, Shuowei Jin, Beidi Chen

Organizations: Carnegie Mellon University · Amazon

Abstract

Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correct behavior in these environments is often implicit, undisclosed, and subject to change over time. Recent continual-learning harnesses seek to address this challenge by enabling agents to improve from serving experience. Yet the effectiveness and limitations of these methods are not yet well characterized. Existing benchmarks provide only partial coverage: some explicitly provide the target knowledge, others assume a static environment, and those that support continual adaptation remain limited in scale and knowledge diversity. To enable systematic evaluation, we formalize an evolving-environment streaming dataset (EESD), in which agents must infer, apply, and revise latent environment knowledge from interaction and outcome feedback as hidden policies evolve, and introduce ServeLearnBench, spanning retail support, banking, and sales-pitch generation with 53 environment windows and 7,718 tasks. We evaluate five learning harnesses (RAG, Mem0, SkillOpt, Continual Harness, and Prime) across six models (GPT-5.6 Terra, Opus 5, Kimi K3, GLM-5.3, DeepSeek V4.1 Flash, and GLM-5.3 Flash), covering 28 model-harness pairs and 252 learning runs. Our evaluation reveals three main findings: a substantial gap remains between task capability and learning from experience; continual adaptation is costly and can degrade already-correct behavior; and insufficient exploration emerges as a key bottleneck to effective adaptation. Overall, ServeLearnBench provides a controlled testbed for diagnosing these limitations and tracking progress toward agents that continually and reliably improve through serving experience.

Figures & tables

Figure 1: Cost vs. Performance. (a) Hidden-case reward versus token cost per test task (prefill, KV-cache hits and output tokens included). (b) Hidden-case reward versus API cost per test task. Reward is the Avg. H of Table 2 (mean over the nine settings). Lines connect Pareto-efficient pairs, for which no other plotted pair achieves both lower cost and higher reward.

Figure 2: ServeLearnBench protocol. Top: the stream is divided into environment windows, each containing feedback-provided serving tasks followed by feedback-hidden test tasks. The active policy in window $s$ consists of behavioral components $\mathcal{C}(\theta_{s})$ , which may be introduced, withdrawn, or reactivated (dashed lines). Bottom left: the learner receives neither policy-component descriptions nor corrections, and must infer the active policy from serving interactions and outcome feedback. Bottom right: benchmark scale across domains, tiers, windows, and tasks.

Scenario	W	Serving	Test
Retail L1	6	720	384
Retail L2	7	810	765
Retail L3	8	624	537
Banking L1	4	336	178
Banking L2	5	500	387
Banking L3	6	600	551

Table 1: Detailed benchmark statistics (W: number of windows). Appendix C provides more details.

Model	Harness	Retail						Banking						Pitch			Avg.		Cost
		Hidden			Fully Specified			Hidden			Fully Specified			Hidden			H	F
		L1	L2	L3	L1	L2	L3	L1	L2	L3	L1	L2	L3	L1	L2	L3
\cellcolor white	Oracle	100.0	96.0	95.5	100.0	100.0	99.7	100.0	98.9	98.7	100.0	100.0	99.2	89.2	92.1	89.5	95.5	99.8	102
\cellcolor white	Blind	1.7	0.0	2.2	100.0	100.0	99.0	1.2	14.0	16.0	95.7	96.5	96.6	31.7	25.9	18.3	12.3	98.0	120
	RAG	65.4	18.8	15.7	86.8	90.8	95.9	69.0	60.8	35.8	91.5	85.1	86.1	47.9	51.0	40.8	45.0	89.4	1,073
	Mem0	20.4	14.4	23.8	94.4	91.6	94.6	69.0	55.4	35.5	97.9	93.5	87.8	34.4	48.1	38.0	37.7	93.3	829

Table 2: Performance comparison of five harnesses across six models on ServeLearnBench.

Figure 3: Average performance (a) by model and (b) by harness.

Figure 4: Cost vs. performance. Average API cost per task and reward, aggregated (a) by model and (b) by learning harness. Bars show cost, including learning; lines show Hidden and overall reward, averaged equally across the nine settings. Blind and Oracle are excluded.

Figure 5: Learning speed for new policies. (a, b) Correctness on Hidden serving tickets as a function of the number of prior tickets from the same policy, pooled across four open-weight models for Banking and Retail, respectively. (c, d) AULC 10 (%) by model and harness.

Figure 6: Fully Specified accuracy with and without adaptation. Mean over four open-weight models and three difficulty levels.

Figure 7: Exploration across models and harnesses. (a, c) Mean number of distinct answers after

k

Hidden serving examples of the same policy, aggregated by harness and model. (b, d) Exploration AUC versus mean Hidden test reward across the six Retail and Banking settings;

\rho

denotes Spearman rank correlation. Per-domain results are provided in Appendix D.3 .

Figure 8: Knowledge acquisition after first success. Left: correctness on Hidden serving tickets aligned at the first correct response (ticket 0), averaged across Retail and Banking. Right: mean correctness over the next five tickets (

\mathrm{AUC}_{\mathrm{KA}}

) versus mean Hidden test reward;

\rho

denotes Spearman rank correlation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	Serving (incl. learning)			Test (frozen state)
	$/task	Adapt	Total	$/task	Adapt	Total
By model
GLM-5.3 Flash	0.0117	40.4	59.5	0.0086	44.6	58.2
DeepSeek V4.1 Flash	0.0228	41.8	61.0	0.0170	44.9	60.1
GLM-5.3	0.1732	48.5	64.5	0.1153	52.5	64.3
Kimi K3	0.1880	48.9	63.9	0.1298	50.7	62.1

Appendix

Table 3: Average cost per task and reward, by model and by learning harness. Serving: API cost of one serving task including the harness’s learning work (Refiner passes, memory model, optimizer, Prime’s post-score turn), and reward on the serving stream. Test: API cost of one held-out test task under the frozen learned state, and test reward. Cost is total dollars over total tasks; reward is the mean over the column’s model–harness–setting cells of the cell’s Hidden (Adapt) or all-case (Total) reward, so every setting has equal weight as in Table 2 (binary in Retail and Banking, the rubric rating in Pitch). Every model column aggregates 45 complete lanes (five harnesses

\times

nine suites) and every harness column 36 (four models

\times

nine suites); Blind and Oracle are excluded. Fireworks serverless prices of 2026-09-06.

Domain	Task families	Required output	Latent state components
Retail	Cancel a pending order; exchange or return delivered items; modify a pending order; resolve multi-item and ranked-fallback requests.	API actions that produce the correct database state, or a refusal with the correct reason code.	Categorical eligibility scopes over payment form, product category, destination, refund method, order composition, and exception status.
Banking	Authorize a pending card transaction; review a credit-limit increase; screen an outbound transfer; resolve ranked alternatives and payment batches in L2/L3.	Approve, deny, or escalate with the correct reason; execute the first admissible alternative or screen each batch line when applicable.	Numeric thresholds and scoped restrictions over merchant corridor, category limits, daily totals, account standing, requested limits, and payee age.
Pitch	Write a 40–80 word sales pitch from a factual product sheet.	A pitch that foregrounds positively valued attributes, suppresses negatively valued attributes, and introduces no unsupported claims.	Valence over product attribute–value pairs; the active preference table is hidden and only the rubric score is returned.

Appendix

Table 4: Task interfaces and latent environment components.

Tier	Temporal demand	Within-task demand	Primary diagnostic
L1	Discover a latent state or newly introduced rule from outcome feedback; Pitch presents four profiles without re-entry.	Mostly single-rule decisions; Pitch uses five visible attribute dimensions.	Initial acquisition relative to the Blind–Oracle gap.
L2	Revise beliefs after tightening, relaxation, scope migration, or limited profile re-entry.	Scoped rules, correlated changes, ranked fallbacks, multi-line cases, or ten-dimensional preference selection.	Hidden-state adaptation and impact on fully specified cases.
L3	Track oscillation, rule removal, and re-entry of previously observed states.	Multiple simultaneously relevant components and staggered reversals; retail also includes permissive departures from the document.	Returned fully specified behavior, stale beliefs, and interference.

Appendix

Table 5: Cross-domain interpretation of the three difficulty tiers.

Domain	Tier	#W	W0	W1	W2	W3	W4	W5	W6	W7	Total $S/T$
Retail	L1	6	79/44	140/64	140/64	120/64	120/64	121/84	n/a	n/a	720/384
	L2	7	99/81	128/104	135/115	131/128	134/130	90/111	93/96	n/a	810/765
	L3	8	78/60	75/65	84/68	78/67	77/76	74/72	80/64	78/65	624/537
Banking	L1	4	84/41	84/47	84/42	84/48	n/a	n/a	n/a	n/a	336/178
	L2	5	100/74	100/77	100/76	100/80	100/80	n/a	n/a	n/a	500/387
	L3	6	100/72	100/90	100/89	100/97	100/105	100/98	n/a	n/a	600/551

Appendix

Table 6: Sample counts by scenario, difficulty, and environment window. Each entry is serving/test; the final row reports 53 windows and 4,508/3,210 tasks.

Tier	ID	Meaning when active
L1	H1	Block modification of a pending order paid with multiple payment methods.
L1	D1	Block cancellation of a pending order paid with a gift card.
L1	D2	Block exchange of an electronics item.
L1	D3	Block exchange when the replacement destination is Florida.
L1	D4	Block refund to the original payment method.
L1	D5	Permit return of a clearance item.

Appendix

Table 7: Dictionary of atomic retail hidden rules.

Tier	ID	W0	W1	W2	W3	W4	W5	W6	W7
L1	H1	✓	✓	✓	✓	✓	✓	n/a	n/a
L1	D1	✗	✓	✓	✗	✗	✗	n/a	n/a
L1	D2	✗	✗	✓	✓	✓	✓	n/a	n/a
L1	D3	✗	✗	✗	✗	✓	✓	n/a	n/a
L1	D4	✗	✗	✗	✗	✗	✓	n/a	n/a
L1	D5	✓	✓	✓	✓	✓	✓	n/a	n/a

Appendix

Table 8: Retail hidden-rule activation by window. A check mark denotes active, a cross denotes inactive, and n/a denotes an unavailable window. Rule meanings are defined in Table 7 .

Tier	ID	Meaning when active
L1	S1	Basic verification is insufficient for a gift-card purchase.
L1	S2-650	Set the electronics instant-approval ceiling to $650.
L1	D1-J	Apply the overseas-merchant restriction to jewelry.
L1	D2-3	Use the three-payee restricted-payee roster.
L1	D2-5	Use the five-payee restricted-payee roster.
L2	S1-2000	Set the same-day transaction-total cap to $2,000.

Appendix

Table 9: Dictionary of atomic banking hidden rules.

Tier	ID	W0	W1	W2	W3	W4	W5
L1	S1	✓	✓	✓	✓	n/a	n/a
L1	S2-650	✓	✓	✓	✓	n/a	n/a
L1	D1-J	✗	✓	✓	✓	n/a	n/a
L1	D2-3	✓	✓	✓	✗	n/a	n/a
L1	D2-5	✗	✗	✗	✓	n/a	n/a
L2	S1-2000	✓	✓	✓	✓	✓	n/a

Appendix

Table 10: Banking hidden-rule activation by window. A check mark denotes active, a cross denotes inactive, and n/a denotes an unavailable window. Rule meanings are defined in Table 9 .

ID	Meaning when active
P-Val	Use the value-for-money buyer profile.
P-Id	Use the identity-expressive buyer profile.
P-AM	Use the anti-marketing rationalist profile.
P-RA	Use the risk-averse buyer profile.
P-Eco	Use the eco-conscious buyer profile.
P-Conf	Use the conformist buyer profile.

Appendix

Table 11: Dictionary of latent pitch preferences.

Tier	ID	W0	W1	W2	W3	W4	W5	W6
L1	P-Val	✓	✗	✗	✗	n/a	n/a	n/a
L1	P-Id	✗	✓	✗	✗	n/a	n/a	n/a
L1	P-AM	✗	✗	✓	✗	n/a	n/a	n/a
L1	P-RA	✗	✗	✗	✓	n/a	n/a	n/a
L2	P-RA	✓	✗	✗	✗	✗	✓	n/a
L2	P-Val	✗	✓	✗	✗	✓	✗	n/a

Appendix

Table 12: Pitch preference activation by window. A check mark denotes active, a cross denotes inactive, and n/a denotes an unavailable window. Preference meanings are defined in Table 11 .

Figure 9: Learning a policy versus dropping the same policy. Solid: correctness on the

k

-th Hidden ticket of policy

S

. Dashed: correctness on the

k

-th ticket after

S

ends that

S

would answer wrongly. Same model, harness and policy on both sides; pairs in which

S

was learned; number of pairs in parentheses; four models, Banking and Retail pooled.

Figure 10: Answer diversity and reward per domain. (a, c, e, g) Mean number of distinct answers given on a policy after

k

of its Hidden serving tickets, by harness (top) and by model (bottom); correctness is ignored. (b, d, f, h) Exploration AUC of those curves against the domain’s average reward on Hidden cases, with the Spearman

\rho

and a least-squares fit.

Figure 11: Exploration and reward per harness, domain and tier. One point per (harness, domain, tier) cell: exploration AUC (Section 5.3 ,

K=8

) over the cell’s policies with the four open-weight models pooled, against the cell’s reward on Hidden cases from Table 2 averaged over the same models. (a) all 30 cells, (b) Retail, (c) Banking; colour is the harness, shape the tier;

\rho

and

r

are the Spearman and Pearson correlations, the dashed line a least-squares fit.

Figure 12: Average cost per task against reward, by phase. Bars (left axis) give the API cost of one task; lines (right axis) give reward on Hidden (adapt) cases and on all cases, averaged over the nine suites with equal weight. (a, b) Serving stream: cost includes the harness’s learning work, reward is measured on the serving cases. (c, d) Held-out test under the frozen learned state. Columns are grouped by model (a, c) and by learning harness (b, d), ordered by serving cost; every column aggregates the same lanes (45 per model column, 36 per harness column). Blind and Oracle are excluded.

Figure 13: Adaptation Cost Analysis. (a) input plus output tokens and (b) API cost per serving task, pooled over all nine suites, split between serving the stream and updating the learned state. Percentages give the learning share of each bar. Prime’s learning part is its post-score turn, recovered from the session records; its serving part is the rest of the wire ledger’s serving bucket. Scored test tasks are excluded. Each panel has its own scale; RAG (no learning-side compute under BM25) is omitted.

Explore similar work

Aug 19, 2026cs.LG

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Replay-Based Continual Learning Continual Learning Catastrophic Forgetting Plasticity Open-World

Ask PDF

Jun 30, 2026cs.LG

AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learning

Can language agents continually learn from experience, turning earlier interactions into reusable capabilities? AhaBench evaluates this ability through exploration after solved hidden-state puzzles, computational transfer after mathematical teaching, and sustained business operation under delayed feedback. The benchmark is agnostic to how an agent learns; the evaluated agents use fixed model weights. Curriculum profiles, teaching contrasts, and daily trajectories reveal a common challenge: using explicit guidance is more reliable than generalizing beyond it or sustaining useful behavior. Across the Puzzle panel, the advantage over matched cold targets is 36.0-53.5 points greater with trace support than at the trace-free endpoint; Qwen 3.6 Plus nevertheless retains a +12.57-point post-curriculum gain. In Euler, worked procedures yield 80.0-100.0% held-out accuracy across models, while question-plus-answer teaching yields 0.0-73.9%. Vending trajectories separate sustained profit, late recovery, and incomplete operation: Doubao Seed 2.0 Pro finishes nominal operation at +495 but averages -10 over the year. Together, these results make continual learning an operational target: experience should yield capabilities that remain effective as guidance, inputs, and business states change. We release tasks, validators, a simulator, records, and analyses for developing agents that turn useful insights into lasting abilities.

Zerui Cheng, Jiawei Xu, Huacan Chai +3

Princeton University · Tencent Hy · Tsinghua University +2

Agentic Benchmarks Continual Learning Reusable Knowledge Curriculum Guidance

Ask PDF

Jul 24, 2026cs.AI

Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents

AI agents encounter learning opportunities in every episode they run, and discard nearly all of them: the underlying models are frozen at deployment, so an agent that resolves a difficult request today starts from zero when it recurs tomorrow. Yet ordinary operation already produces feedback, in the form of outcome verdicts and after-the-fact corrections. We show that this feedback is a sufficient signal for continual learning when the frozen model is paired with an external memory that distils each episode into retrievable natural-language rules. On the banking domain of

τ

-bench, against a static-RAG control retrieving over the complete policy corpus, learning from the one-bit outcome verdict lifts single-trial success to 1.6

\times

the baseline, and learning from corrections to 2.6

\times

, converting 22 of the 84 tasks the baseline never solves. The result spans the deployment spectrum, measured on Mistral Large, an open-weights model that organisations with data sovereignty requirements can self-host, and replicated on a frontier model, Claude Sonnet 5. The accumulated memory also transfers: each model, reading the store built by the other, rises above its own no-memory baseline. The harness, protocol, and data are released.

Valentin Tablan, Scott Taylor, Kristoffer Bernhem

The Memory Company

Continual Learning Feedback Digital Sovereignty Jobs

Ask PDF