CARE: Grounded Recursive Self-Improvement for Production Customer-Service Agents

Yanbei Jiang; Chao Lei; Rena Gao; Xiaochen Zhao; Yuchen Ge; Haohang Liu; Runze Nie; Qitian Chen

cs.AINov 2, 2026

CARE: Grounded Recursive Self-Improvement for Production Customer-Service Agents

Authors: Yanbei Jiang, Chao Lei, Rena Gao, Xiaochen Zhao, Yuchen Ge, Haohang Liu, Runze Nie, Qitian Chen,

Organizations: AI and Data Team, Keeta, Meituan · University of Melbourne · Fudan University

Abstract

Recursive self-improvement (RSI) in production agents involves transforming accumulated experience into reliable model updates while adapting to evolving business policies. We present CARE (Customer-service Agent Retrospective and Evolve), a grounded RSI framework that operationalizes this process in customer service. Its goal is to replace a frontier proprietary incumbent with an in-house mid-sized model that continues learning after deployment. After a one-time cold start on governed production traces, the current model serves production traffic alongside the incumbent, and its own failures drive the next update. A 24/7 inspection process establishes failures through evidence-blinded, independently reviewed judgments. Attribution separates model errors from defects in Skills, tools, or the harness, routing component failures to repair. Model-attributable trajectories are truncated at the first erroneous decision and paired with verified repairs. On-policy self-distillation (OPSD) then updates the model using the same model as teacher, with the diagnosis and repair supplied as privileged information unavailable to the student. To support continued evolution as business requirements change, training mixes failures from different Skill/SOP versions, each judged against the policy in force when it occurred. Candidates run an executable simulation test compiled from the latest SOP and then frozen, followed by offline inspection of the resulting traces. Promotion requires a combined simulation-and-inspection score at least at parity with the deployed version, confirmation through an online comparison with the incumbent, and human approval.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of CARE. Phase I bootstraps an in-house policy $\pi_{\theta_{0}}$ from governed production traces of the incumbent system. Phase II is the recursive loop. The current model $\pi_{\theta_{k}}$ serves part of the production traffic; 24/7 online inspection turns its own conversations into reviewed failures; attribution separates model-attributable failures from component failures, which are repaired outside the model; model-attributable failures are cut at the first erroneous decision and repaired, and the model distills from itself on that decision, with the hindsight visible only to the teacher. The candidate is measured by the evaluation system, where it runs a simulation test compiled from the latest Skill/SOP and then frozen, and its simulation traces are inspected offline; it is promoted only if the combined score of the two is at least at parity with the deployed version and an online comparison with the incumbent confirms it. After cold start, training data come from the model’s own failures and the teacher is the model itself. Stages tagged in red are driven by an external frontier model rather than by the in-house model. The operating context is revised independently of the loop: a revision reaches the deployed model through its input and the evaluation system through a recompiled test (Section 3.3 ).

Method	Updated component	Experience source	Grounding	Teacher	Contexts
STaR [ 10 ]	weights	own rationales	answer correctness	–	–
SPIN [ 11 ]	weights	own generations	human demonstrations	previous self	–
Self-Rewarding [ 12 ]	weights	own responses	self-judgment	self	–
Reflexion [ 13 ]	prompt / memory	own attempts	env. signal + self-reflection	–	–
Voyager [ 14 ]	skill library	own episodes	environment execution	–	–
DGM [ 15 ]	agent code	own code edits	benchmark evaluation	–	–

Table 1: Positioning of CARE among representative self-improvement and distillation methods. “Grounding” denotes the evidence or supervision used to guide improvement, while “Contexts” indicates whether the method explicitly addresses changing policy requirements during iterative improvement. A dash indicates that the corresponding feature is not explicitly addressed.

Panel of Figure 2 — Figure 2: The cold-start data engine. (a) Admission keeps complete, context-keyed conversations that pass language, structural, and evidence-anchored semantic checks; only no_problem conversations enter $D_{s}$ , and rejected or unresolved records are never backfilled. (b) Allocation splits a fixed budget $N$ over region groups, the scenarios of each group, and the regions of GCC4, applying the capacity-constrained equal split of Eq. 8 at every level, and then freezes the selected members $S$ . (c) Refinement borrows tone from linked human conversations for eligible turns while preserving every protected field; a rejected rewrite keeps the original conversation, and only a failed final check removes it.

Panel of Figure 3 — Figure 3: Online inspection and attribution. Production conversations of the in-house model are frozen into a case set $\mathcal{Q}_{t}$ and pass an evidence-blinded inspection (Eq. 18 ) that separates ranking, capacity-aware admission from three strata (Eq. 20 ), analysis, and independent review with reconciliation, and ends in one of four terminal states (Eq. 21 ). Terminal states feed back to the screener (Eq. 23 ). Only confirmed failures reach attribution, which routes model-attributable cases, together with their diagnosis $d$ , to incremental task construction, routes component failures to repair, and holds cases whose cause cannot be isolated.

Cause	Evidence sought	Route
Model behavior	Same context and valid tools; the decision error is reproducible, including in situations that training covered poorly	Incremental task and attribution-guided OPSD
Skill/SOP or prompt	The policy or prompt itself is wrong, ambiguous, or inconsistent	Revise the Skill/SOP or prompt
Tool, harness, or router	Schema, dispatch, projection, or route mismatch	Runtime repair
Judge or inspection	Unsupported verdict or review disagreement	Recalibrate the evaluator; discard the label
Infrastructure	Timeout, missing artifact, failed write	Operational recovery; no quality label

Table 2: Attribution categories and repair routes. Only the model-behavior category produces training data for Phase II; every other category is repaired outside the model.

Panel of Figure 4 — Figure 4: Retrospective repair construction. (a) A confirmed failure is cut at the first decision the model got wrong. Everything the model could see at that moment becomes the student input; the erroneous output, the rest of the conversation, the outcome, and the diagnosis become hindsight. (b) The proposer uses hindsight to write a minimal repair that remains justified by the decision-time input alone, an independent verifier accepts or rejects it, and each accepted tuple can be projected into SFT, DPO, or OPSD data. Every case ends in exactly one terminal state.

View	Student input	Learning signal
SFT	$x$	imitate the reference repair $a^{+}$
DPO	$x$	prefer $a^{+}$ over $a^{-}$
OPSD	$x$	match a self-teacher that sees $(x,I)$ , on the student’s own sampled output

Table 3: Training views of a repair tuple

(x,c,a^{-},a^{+},d)

I=\mathrm{enc}(d,a^{-},a^{+})

is the privileged information.

Panel of Figure 5 — Figure 5: Attribution-guided on-policy self-distillation at the failing decision. The student samples a fresh decision $\hat{y}$ from the decision-time input $x$ . A frozen copy of the same model, additionally given the privileged information $I$ (diagnosis, incident-time SOP, and verified repair), reads the same prefixes, so at every token position both models give a distribution over the vocabulary. The loss compares the two on the teacher’s top- $K$ tokens, in the forward direction with a floor at zero in the deployed configuration, and is backpropagated directly. Gradients flow only into the student, and $I$ stays on the teacher side of the privileged boundary. The direction of the KL divergence, $K$ , and the floor are ablated in Section 8.7 .

Panel of Figure 6 — Figure 6: The executable simulation environment. Before the run, the frozen case is split into one view per module (Eq. 38 ): the user simulator receives the user’s goal, initial knowledge, and signal-gated reactions; the system simulator the initial state and event schedule; the agent only its operating context; and the assessment the branch and the oracle, which no participant sees (Eq. 37 ). During the run, the orchestrator mediates every exchange, only the system simulator changes business state, and each participant receives its own projection of that state. After the run, the trace and the state log are judged against the oracle, and each case ends in pass or fail (Eq. 41 ). The traces are also inspected offline (Section 7.2 ).

Role	May	May not
Scenario compiler	Fix the user situation, initial state, Skill/SOP version, branch, and oracle	Choose an action for the agent during the run
User simulator	Speak, confirm, or operate on what has been shown	Read $x_{t}$ or $\Omega_{j}$
System simulator	Hold $x_{t}$ , execute actions, advance virtual time	Infer a state change from the agent’s wording
Agent under test	Query, decide, reply, and act	Observe $x_{t}$ other than through query results, or see $\Omega_{j}$
Post-run assessment	Judge from $\Omega_{j}$ and the recorded evidence	Steer any participant during the run

Table 4: Roles in a simulation run. Each role has a fixed boundary.

Link	Placed in	Example
Reachable state	System simulator, in $x_{j,0}$	The order is recorded as delivered
Observable signal	Projections $O_{\mathrm{c}}$ and $O_{\mathrm{u}}$	The order query returns delivered to the agent; the user learns it only when the agent says so or a card shows it
User reaction	User simulator, as a gated pair $(\sigma_{\ell},r_{\ell})$ in $\phi_{j}$	Once told that the order was delivered, the user states that nothing was received
Agent decision	Not given to the agent; checked by $\Omega^{\mathrm{req}}_{j}$ and $\Omega^{\mathrm{pro}}_{j}$	Whether the agent queries before answering, and how it then handles a disputed delivery
Post-run evidence	Assessment, from the trace and the state log	The query and its result, the agent’s statements, and any executed action with its state change

Table 5: How the five links of a branch (Eq. 33 ) are placed in the modules, for the branch in which an order is recorded as delivered while the user reports that it has not arrived. The example illustrates the distribution only; the correct handling is whatever the Skill/SOP in force specifies.

		Offline			Online satisfaction
Version	Period	Sim. acc. $\uparrow$	Bad Case $\downarrow$	Combined $S$ $\uparrow$	$\Delta$ $\uparrow$
V0	P1 (Jul 30–Aug 11)	93.00	6.42	93.29	$-4.57$
V1	P2 (Aug 11–18)	93.12	4.35	94.39	$-8.02$
V2	P3 (Aug 18–27)	94.58	4.72	94.93	$-4.87$
V3	P4 (Aug 27–Sep 4)	95.30	4.47	95.42	$-3.13$
V4	P5 (Sep 10–24)	95.17	4.11	95.53	$+0.34$

Table 6: Six versions of CARE-31B. V0 is the cold-start model, and V

k

is obtained from V

(k{-}1)

by the

k

-th Phase II iteration. Offline metrics are measured on CARE-31B in percent, and

S

is the combined score of Eq. ( 45 ) used for promotion; online,

\Delta

is the satisfaction of CARE-31B minus that of the incumbent serving concurrently, in percentage points, in the period in which each version served. V5 is configuration 4 of Table 10 . No evaluation supplies privileged information to the model.

Panel of Figure 7 — Figure 7: Recursive improvement of CARE-31B over six versions. (a) Online satisfaction of each version against the incumbent serving concurrently in the same period; the vertical scale is omitted. (b, c) Offline metrics of CARE-31B, with reference models as dashed lines; Gemma-4-31B-IT (zero-shot, 79.00%) and Qwen3.8-27B (77.08%) fall below the range of (b), and GPT-6-Luna (5.55%) and Qwen3.8-27B (9.17%) are omitted from (c) for legibility.

Model	Role	Sim. accuracy (%) $\uparrow$	Bad Case rate (%) $\downarrow$	Combined $S$ $\uparrow$
GPT-5.4	incumbent	96.80	4.60	96.10
GPT-6-Luna	reference	91.60	5.55	93.03
Gemini-3.5-Flash	reference	94.30	6.44	93.93
DeepSeek-V4-Flash	reference	93.71	4.97	94.37
Qwen3.8-27B	reference	77.08	9.17	83.96
Gemma-4-31B-IT	base, zero-shot	79.00	5.59	86.71

Table 7: Comparison with other models. Simulation accuracy is the share of simulated cases that pass; the inspection Bad Case rate is the confirmed-bad rate of Eq. ( 22 );

S

is the combined score of Eq. ( 45 ). Reference models run without task-specific training.

	Reply latency				Cost per service order
	Mean	P50	P90	P95	30%	50%	100%
Reduction (%) $\uparrow$	18.4	27.7	28.0	28.9	21.7	35.8	70.5

Table 8: Reduction of CARE-31B against the incumbent, in percent. Reply latency : online reply latency over successful replies, with V5 and the incumbent serving at the same time. Cost per service order : by the share of the traffic served by the in-house model.

Objective	Learning signal	On-policy	Sim. acc. (%) $\uparrow$	Bad Case (%) $\downarrow$	$S$ $\uparrow$
SFT	imitate $a^{+}$	$\times$	–	–	–
DPO	prefer $a^{+}$ over $a^{-}$	$\times$	15.1	1.88	56.61
OPSD (CARE)	match self-teacher given $I$	✓	94.1	1.97	96.07

Table 9: Ablation of the learning signal. All objectives start from V4 and use the same repair tuples and training budget. On-policy : the loss is defined on the model’s own samples.

S

is the combined score of Eq. ( 45 ). A dash means unavailable, not zero: SFT collapsed during training and never entered the full test. The OPSD row is configuration 4 of Table 10 .

Config.	Direction	$K$	Tail	Floor	Sim. acc. (%) $\uparrow$	Bad Case (%) $\downarrow$	$S$ $\uparrow$
1	reverse, Eq. ( 30 )	128	✓	–	90.9	1.79 (10)	94.56
2	forward, Eq. ( 29 )	128	–	✓	93.0	2.87 (16)	95.07
3	forward, Eq. ( 29 )	128	–	–	94.1	2.50 (14)	95.80
4	forward, Eq. ( 29 )	32	–	✓	94.1	1.97 (11)	96.07

Table 10: Ablation of the OPSD loss. Bad Case : confirmed Bad Cases over all cases of the report, with their number in parentheses.

S

is the combined score of Eq. ( 45 ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Revision	Transformation
D1 Baseline	Production incumbent trajectories, no human data
D2 Concat	Append complete human-support conversations
D3 Style	Filter, rewrite, and insert human turns
D4 Protect	Preserve business actions and metadata during insertion
D5 Rebalance	Adjust the region and scenario distribution
D6 EvalPatch	Target inspected failures while preserving structure

Appendix

Table 11: Historical revisions of the cold-start data recipe.

Verdict	Meaning	Effect
no_problem	No supported execution defect; the decision does not depend on missing runtime evidence	Admitted to allocation
problem	Visible evidence supports a specific execution defect	Quarantined
inconclusive	No defect found, but the decision depends on missing decisive runtime evidence	Quarantined separately

Appendix

Table 12: Semantic screening verdicts and their downstream effect.

Item	Setting
Base model	Gemma-4-31B-IT
Objective	masked negative log-likelihood on assistant tokens (Eq. 17 )
Maximum sequence length	16,384 tokens
Epochs and steps	one epoch; 467 steps
Batch size	global batch 32; micro-batch 1 per GPU
Learning rate	peak $1\times 10^{-5}$ ; warmup over 5% of steps (23 steps), then cosine decay

Appendix

Table 13: Cold-start SFT configuration.

Tier	Purpose	Supported conclusion
Fixture smoke	Check assets, protocols, state transitions, and the offline loop with a substitute agent	Internal consistency of the environment
Live smoke	Exercise a small slice with the real model, routes, and tools	Integration evidence for that slice
Live end-to-end	Run the real chain on a frozen scenario set	Acceptance within the selected scope

Appendix

Table 14: Simulation execution tiers.

Mode	Input and purpose	Boundary
Online continuous	Mature production sessions, for ongoing discovery	Persistent source, ranked pool, admitted queue
Custom online	A selected experiment, market, window, or user segment	Independent of the main online queue
Offline check	A frozen report or export with a fixed test set	Frozen cases and evidence view
Screening only	Fast candidate diagnosis	No deep analysis or publication
Technical check	Tool-call, log, and infrastructure anomalies	Technical signal only; no semantic verdict

Appendix

Table 15: Inspection modes.

Stage	Required artifact	Stop condition
Freeze	Source, case identities, evidence view	Identity or snapshot not fixed
Screen	Candidate score, evidence anchors, risk tier	Screening failure
Admit	Ranked pool and admitted queue	Capacity or strata unavailable
Deep analysis	Structured case record with claim ledger	Unsupported claims
Review	Independent verdict and correction history	Conflict, missing evidence, malformed output
Gate and audit	Terminal verdict, counts, denominator	Missing field, count mismatch, leakage

Appendix

Table 16: Inspection stage contract.

	Case 1: expected behavior	Case 2: confirmed Bad Case
Customer issue	Refund for an order cancelled because the courier was late	Charged, but the order was never confirmed
Retained window	9 entries over 45 s	15 entries over 5.5 min
Tool calls	6: Skill load, context resolver	8: Skill load, context resolver, order detail
Review	First review FAIL; reconciled; second review PASS as expected behavior, high confidence	First review PASS as confirmed Bad Case, high confidence
Gate and outcome	Upload not allowed; closed as no issue	Notification gate PASS; uploaded as a Bad Case
Inspection time	26 min	18.5 min

Appendix

Table 17: The two inspected cases at a glance.

#	Speaker	Message (translated, condensed)	Action and tools
G01	Agent	Welcome. Is your issue “When will I get my money back?”	Entry FAQ
G02–03	User	No. / The Merchant A order.
G04	Agent	Do you mean an order from Merchant A? Please tell me the issue, e.g. a missing item, a wrong order, cold food, or a late delivery.	INTENT_GUIDE
G05–07	User	I cancelled it and the money has not been returned. / The driver was late. / It has been over an hour and the money has not been returned.
G08	Agent	Since the order was cancelled because the driver was late, please confirm the order in the list and I will check its refund status.	Skill loaded; resolver returns no order ID; ORDER_CONFIRMATION
G09	Card	Suggested order: Merchant B, delivered, 50.40 SAR, awaiting confirmation.

Appendix

Table 18: Case 1 dialogue (expected behavior). Replies the model generated for G02, G05, and G06 were not delivered to the customer and are omitted.

#	Speaker	Message (translated, condensed)	Action and tools
B01	Agent	Welcome. Is your issue “Why did the payment fail?”	Entry FAQ
B02	User	I placed an order; the money was taken, but the order was not confirmed.
B03	Agent	Please select the order you are asking about, and I will check its payment or refund status.	Skill loaded; resolver returns no order ID; ORDER_CONFIRMATION
B04	Card	Suggested order: Merchant B, delivered, 34.68 SAR, awaiting confirmation.
B05	User	I ordered from Merchant A and you took the money. Why was the order not confirmed?
B06 $\times$	Agent	I checked the Merchant B order: it was completed and delivered, and has no refund request. If this is not the right order, please give the order number.	Resolver returns the Merchant B order; order detail: completed on 29 July, no after-sale record; GUIDE

Appendix

Table 19: Case 2 dialogue (confirmed Bad Case). Rows marked

\times

contain the defect.

Axis	Question	Passing value
Disposition	Is the case repaired by the proposed decision?	repaired
Eligibility	May the pair be used for training at all?	eligible
Fault relation	Is the anchored output the model’s own fault?	model fault
Symptom	Is the inspected symptom corrected?	corrected
Boundary	Is the change confined to the anchored decision?	corrected
Policy	Does the repair follow the incident-time SOP?	compliant

Appendix

Table 20: Axes of the independent repair verifier.

Item	Setting
Student initialization	V4, the previous version; full-parameter update, vision tower trainable
Teacher	V4, frozen for the whole run, not a moving average; scores the student’s rollout token by token; targets detached and kept in FP32
Privileged information	appended to the teacher’s system message; never in the student context
Training set	280 repair tuples after micro-scenario capping (Section 6.3 ); 281 candidates, one excluded for exceeding four images per tuple
Objective	forward KL partial sum over the teacher’s top 32 tokens, floored at zero per token (Eq. 29 ); backpropagated directly; loss coefficient 1.0; mean over all generated tokens of the batch (Eq. 31 )
Not used	task reward, policy gradient, tail cell, renormalization within the top- $K$ set, probability floor or smoothing, auxiliary SFT or DPO loss

Appendix

Table 21: OPSD training configuration of the final model.

Recursive Self-Improvement

Recursive self-improvement studies AI systems that use feedback from their own work to modify and evaluate future versions.

See all 99 papers

RSI-Env: An Open-Ended Environment for Recursive Self-Improvement of Agents
Nov 1, 2026Wanying Qu, Zhiyuan Fan, Jiyao Liu +4LLM Agent Self-Improvement Recursive Self-Improvement
Fudan University · The Hong Kong University of Science and Technology · Georgia Institute of Technology +3
MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution
Sep 21, 2026Junde Wu, Jiayuan Zhu, Minghao Hu +2Self-Improving Agents Self-Evolving Agents
University of Oxford · Stanford University
NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness
Sep 8, 2026NeoHorse Team, Guoliang Cao, Guohao Dai +34Agent Harness Optimization LLM Agent Self-Improvement
TokenRhythm

Nov 1, 2026J/K move · Enter open · S save

RSI-Env: An Open-Ended Environment for Recursive Self-Improvement of Agents

Wanying Qu, Zhiyuan Fan, Jiyao Liu, Jiang Zhou+3

Fudan University · The Hong Kong University of Science and Technology · Georgia Institute of Technology+3

LLM Agent Self-Improvement Recursive Self-Improvement Agent Harness Self-Evolution Self-Evolving Agents

Open paper AskPDF arXiv

Recursive self-improvement (RSI) envisions agent systems that autonomously improve their own models and harnesses, with each updated system continuing to improve itself. Existing approaches typically restrict optimization objectives, editable components, or evaluation procedures, limiting open-ended exploration and continuity across generations. We introduce RSI-Env, a persistent environment for coordinated model-harness improvement, supporting autonomous training, harness modification, and retention of research state. An independent Host Controller manages services, evaluation, and versioning. Under Self, each accepted model-harness version takes over the next improvement round. We define three settings that vary goal specificity and access to benchmark feedback: Open-General, Capability-Directed, and Benchmark-Guided. Across these three settings, we compare Self with two fixed external Teachers on seven tasks, starting from Qwen3.8-27B with a minimal harness. Without benchmark feedback, Open-General Self raises GPQA Diamond accuracy from 65.2% to 73.2% and AIME 2026 accuracy from 53.3% to 56.7%. All three Capability-Directed Self runs achieve target-aligned gains. Knowledge Self demonstrates continued improvement across system updates, with SimpleQA accuracy rising from 82.0% to 85.0% and then 88.0%. Benchmark guidance and Teacher benefits vary by task. These results demonstrate RSI-Env's support for open-ended exploration and sustained recursive improvement within the studied runs, while highlighting challenges in preserving gains across capabilities.

Abstract

Recursive Self-Improvement

Explore similar work

RSI-Env: An Open-Ended Environment for Recursive Self-Improvement of Agents

MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution

NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness