Recursive self-improvement (RSI) in production agents involves transforming accumulated experience into reliable model updates while adapting to evolving business policies. We present CARE (Customer-service Agent Retrospective and Evolve), a grounded RSI framework that operationalizes this process in customer service. Its goal is to replace a frontier proprietary incumbent with an in-house mid-sized model that continues learning after deployment. After a one-time cold start on governed production traces, the current model serves production traffic alongside the incumbent, and its own failures drive the next update. A 24/7 inspection process establishes failures through evidence-blinded, independently reviewed judgments. Attribution separates model errors from defects in Skills, tools, or the harness, routing component failures to repair. Model-attributable trajectories are truncated at the first erroneous decision and paired with verified repairs. On-policy self-distillation (OPSD) then updates the model using the same model as teacher, with the diagnosis and repair supplied as privileged information unavailable to the student. To support continued evolution as business requirements change, training mixes failures from different Skill/SOP versions, each judged against the policy in force when it occurred. Candidates run an executable simulation test compiled from the latest SOP and then frozen, followed by offline inspection of the resulting traces. Promotion requires a combined simulation-and-inspection score at least at parity with the deployed version, confirmation through an online comparison with the incumbent, and human approval.
Figures & tables
Figure 1: Overview of CARE. Phase I bootstraps an in-house policy πθ0 from governed production traces of the incumbent system. Phase II is the recursive loop. The current model πθk serves part of the production traffic; 24/7 online inspection turns its own conversations into reviewed failures; attribution separates model-attributable failures from component failures, which are repaired outside the model; model-attributable failures are cut at the first erroneous decision and repaired, and the model distills from itself on that decision, with the hindsight visible only to the teacher. The candidate is measured by the evaluation system, where it runs a simulation test compiled from the latest Skill/SOP and then frozen, and its simulation traces are inspected offline; it is promoted only if the combined score of the two is at least at parity with the deployed version and an online comparison with the incumbent confirms it. After cold start, training data come from the model’s own failures and the teacher is the model itself. Stages tagged in red are driven by an external frontier model rather than by the in-house model. The operating context is revised independently of the loop: a revision reaches the deployed model through its input and the evaluation system through a recompiled test (Section 3.3 ).
Method
Updated component
Experience source
Grounding
Teacher
Contexts
STaR [ 10 ]
weights
own rationales
answer correctness
–
–
SPIN [ 11 ]
weights
own generations
human demonstrations
previous self
–
Self-Rewarding [ 12 ]
weights
own responses
self-judgment
self
–
Reflexion [ 13 ]
prompt / memory
own attempts
env. signal + self-reflection
–
–
Voyager [ 14 ]
skill library
own episodes
environment execution
–
–
DGM [ 15 ]
agent code
own code edits
benchmark evaluation
–
–
Table 1: Positioning of CARE among representative self-improvement and distillation methods. “Grounding” denotes the evidence or supervision used to guide improvement, while “Contexts” indicates whether the method explicitly addresses changing policy requirements during iterative improvement. A dash indicates that the corresponding feature is not explicitly addressed.
Figure 2: The cold-start data engine. (a) Admission keeps complete, context-keyed conversations that pass language, structural, and evidence-anchored semantic checks; only no_problem conversations enter Ds , and rejected or unresolved records are never backfilled. (b) Allocation splits a fixed budget N over region groups, the scenarios of each group, and the regions of GCC4, applying the capacity-constrained equal split of Eq. 8 at every level, and then freezes the selected members S . (c) Refinement borrows tone from linked human conversations for eligible turns while preserving every protected field; a rejected rewrite keeps the original conversation, and only a failed final check removes it.
Figure 3: Online inspection and attribution. Production conversations of the in-house model are frozen into a case set Qt and pass an evidence-blinded inspection (Eq. 18 ) that separates ranking, capacity-aware admission from three strata (Eq. 20 ), analysis, and independent review with reconciliation, and ends in one of four terminal states (Eq. 21 ). Terminal states feed back to the screener (Eq. 23 ). Only confirmed failures reach attribution, which routes model-attributable cases, together with their diagnosis d , to incremental task construction, routes component failures to repair, and holds cases whose cause cannot be isolated.
Cause
Evidence sought
Route
Model behavior
Same context and valid tools; the decision error is reproducible, including in situations that training covered poorly
Incremental task and attribution-guided OPSD
Skill/SOP or prompt
The policy or prompt itself is wrong, ambiguous, or inconsistent
Revise the Skill/SOP or prompt
Tool, harness, or router
Schema, dispatch, projection, or route mismatch
Runtime repair
Judge or inspection
Unsupported verdict or review disagreement
Recalibrate the evaluator; discard the label
Infrastructure
Timeout, missing artifact, failed write
Operational recovery; no quality label
Table 2: Attribution categories and repair routes. Only the model-behavior category produces training data for Phase II; every other category is repaired outside the model.
Figure 4: Retrospective repair construction. (a) A confirmed failure is cut at the first decision the model got wrong. Everything the model could see at that moment becomes the student input; the erroneous output, the rest of the conversation, the outcome, and the diagnosis become hindsight. (b) The proposer uses hindsight to write a minimal repair that remains justified by the decision-time input alone, an independent verifier accepts or rejects it, and each accepted tuple can be projected into SFT, DPO, or OPSD data. Every case ends in exactly one terminal state.
View
Student input
Learning signal
SFT
x
imitate the reference repair a+
DPO
x
prefer a+ over a−
OPSD
x
match a self-teacher that sees (x,I) , on the student’s own sampled output
Table 3: Training views of a repair tuple (x,c,a−,a+,d) . I=enc(d,a−,a+) is the privileged information.
Figure 5: Attribution-guided on-policy self-distillation at the failing decision. The student samples a fresh decision y^ from the decision-time input x . A frozen copy of the same model, additionally given the privileged information I (diagnosis, incident-time SOP, and verified repair), reads the same prefixes, so at every token position both models give a distribution over the vocabulary. The loss compares the two on the teacher’s top- K tokens, in the forward direction with a floor at zero in the deployed configuration, and is backpropagated directly. Gradients flow only into the student, and I stays on the teacher side of the privileged boundary. The direction of the KL divergence, K , and the floor are ablated in Section 8.7 .
Figure 6: The executable simulation environment. Before the run, the frozen case is split into one view per module (Eq. 38 ): the user simulator receives the user’s goal, initial knowledge, and signal-gated reactions; the system simulator the initial state and event schedule; the agent only its operating context; and the assessment the branch and the oracle, which no participant sees (Eq. 37 ). During the run, the orchestrator mediates every exchange, only the system simulator changes business state, and each participant receives its own projection of that state. After the run, the trace and the state log are judged against the oracle, and each case ends in pass or fail (Eq. 41 ). The traces are also inspected offline (Section 7.2 ).
Role
May
May not
Scenario compiler
Fix the user situation, initial state, Skill/SOP version, branch, and oracle
Choose an action for the agent during the run
User simulator
Speak, confirm, or operate on what has been shown
Read xt or Ωj
System simulator
Hold xt , execute actions, advance virtual time
Infer a state change from the agent’s wording
Agent under test
Query, decide, reply, and act
Observe xt other than through query results, or see Ωj
Post-run assessment
Judge from Ωj and the recorded evidence
Steer any participant during the run
Table 4: Roles in a simulation run. Each role has a fixed boundary.
Link
Placed in
Example
Reachable state
System simulator, in xj,0
The order is recorded as delivered
Observable signal
Projections Oc and Ou
The order query returns delivered to the agent; the user learns it only when the agent says so or a card shows it
User reaction
User simulator, as a gated pair (σℓ,rℓ) in ϕj
Once told that the order was delivered, the user states that nothing was received
Agent decision
Not given to the agent; checked by Ωjreq and Ωjpro
Whether the agent queries before answering, and how it then handles a disputed delivery
Post-run evidence
Assessment, from the trace and the state log
The query and its result, the agent’s statements, and any executed action with its state change
Table 5: How the five links of a branch (Eq. 33 ) are placed in the modules, for the branch in which an order is recorded as delivered while the user reports that it has not arrived. The example illustrates the distribution only; the correct handling is whatever the Skill/SOP in force specifies.
Offline
Online satisfaction
Version
Period
Sim. acc. ↑
Bad Case ↓
Combined S↑
Δ↑
V0
P1 (Jul 30–Aug 11)
93.00
6.42
93.29
−4.57
V1
P2 (Aug 11–18)
93.12
4.35
94.39
−8.02
V2
P3 (Aug 18–27)
94.58
4.72
94.93
−4.87
V3
P4 (Aug 27–Sep 4)
95.30
4.47
95.42
−3.13
V4
P5 (Sep 10–24)
95.17
4.11
95.53
+0.34
Table 6: Six versions of CARE-31B. V0 is the cold-start model, and V k is obtained from V (k−1) by the k -th Phase II iteration. Offline metrics are measured on CARE-31B in percent, and S is the combined score of Eq. ( 45 ) used for promotion; online, Δ is the satisfaction of CARE-31B minus that of the incumbent serving concurrently, in percentage points, in the period in which each version served. V5 is configuration 4 of Table 10 . No evaluation supplies privileged information to the model.
Figure 7: Recursive improvement of CARE-31B over six versions. (a) Online satisfaction of each version against the incumbent serving concurrently in the same period; the vertical scale is omitted. (b, c) Offline metrics of CARE-31B, with reference models as dashed lines; Gemma-4-31B-IT (zero-shot, 79.00%) and Qwen3.8-27B (77.08%) fall below the range of (b), and GPT-6-Luna (5.55%) and Qwen3.8-27B (9.17%) are omitted from (c) for legibility.
Model
Role
Sim. accuracy (%) ↑
Bad Case rate (%) ↓
Combined S↑
GPT-5.4
incumbent
96.80
4.60
96.10
GPT-6-Luna
reference
91.60
5.55
93.03
Gemini-3.5-Flash
reference
94.30
6.44
93.93
DeepSeek-V4-Flash
reference
93.71
4.97
94.37
Qwen3.8-27B
reference
77.08
9.17
83.96
Gemma-4-31B-IT
base, zero-shot
79.00
5.59
86.71
Table 7: Comparison with other models. Simulation accuracy is the share of simulated cases that pass; the inspection Bad Case rate is the confirmed-bad rate of Eq. ( 22 ); S is the combined score of Eq. ( 45 ). Reference models run without task-specific training.
Reply latency
Cost per service order
Mean
P50
P90
P95
30%
50%
100%
Reduction (%) ↑
18.4
27.7
28.0
28.9
21.7
35.8
70.5
Table 8: Reduction of CARE-31B against the incumbent, in percent. Reply latency : online reply latency over successful replies, with V5 and the incumbent serving at the same time. Cost per service order : by the share of the traffic served by the in-house model.
Objective
Learning signal
On-policy
Sim. acc. (%) ↑
Bad Case (%) ↓
S↑
SFT
imitate a+
×
–
–
–
DPO
prefer a+ over a−
×
15.1
1.88
56.61
OPSD (CARE)
match self-teacher given I
✓
94.1
1.97
96.07
Table 9: Ablation of the learning signal. All objectives start from V4 and use the same repair tuples and training budget. On-policy : the loss is defined on the model’s own samples. S is the combined score of Eq. ( 45 ). A dash means unavailable, not zero: SFT collapsed during training and never entered the full test. The OPSD row is configuration 4 of Table 10 .
Config.
Direction
K
Tail
Floor
Sim. acc. (%) ↑
Bad Case (%) ↓
S↑
1
reverse, Eq. ( 30 )
128
✓
–
90.9
1.79 (10)
94.56
2
forward, Eq. ( 29 )
128
–
✓
93.0
2.87 (16)
95.07
3
forward, Eq. ( 29 )
128
–
–
94.1
2.50 (14)
95.80
4
forward, Eq. ( 29 )
32
–
✓
94.1
1.97 (11)
96.07
Table 10: Ablation of the OPSD loss. Bad Case : confirmed Bad Cases over all cases of the report, with their number in parentheses. S is the combined score of Eq. ( 45 ).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Revision
Transformation
D1 Baseline
Production incumbent trajectories, no human data
D2 Concat
Append complete human-support conversations
D3 Style
Filter, rewrite, and insert human turns
D4 Protect
Preserve business actions and metadata during insertion
D5 Rebalance
Adjust the region and scenario distribution
D6 EvalPatch
Target inspected failures while preserving structure
Appendix
Table 11: Historical revisions of the cold-start data recipe.
Verdict
Meaning
Effect
no_problem
No supported execution defect; the decision does not depend on missing runtime evidence
Admitted to allocation
problem
Visible evidence supports a specific execution defect
Quarantined
inconclusive
No defect found, but the decision depends on missing decisive runtime evidence
Quarantined separately
Appendix
Table 12: Semantic screening verdicts and their downstream effect.
Item
Setting
Base model
Gemma-4-31B-IT
Objective
masked negative log-likelihood on assistant tokens (Eq. 17 )
Maximum sequence length
16,384 tokens
Epochs and steps
one epoch; 467 steps
Batch size
global batch 32; micro-batch 1 per GPU
Learning rate
peak 1×10−5 ; warmup over 5% of steps (23 steps), then cosine decay
Appendix
Table 13: Cold-start SFT configuration.
Tier
Purpose
Supported conclusion
Fixture smoke
Check assets, protocols, state transitions, and the offline loop with a substitute agent
Internal consistency of the environment
Live smoke
Exercise a small slice with the real model, routes, and tools
Integration evidence for that slice
Live end-to-end
Run the real chain on a frozen scenario set
Acceptance within the selected scope
Appendix
Table 14: Simulation execution tiers.
Mode
Input and purpose
Boundary
Online continuous
Mature production sessions, for ongoing discovery
Persistent source, ranked pool, admitted queue
Custom online
A selected experiment, market, window, or user segment
Independent of the main online queue
Offline check
A frozen report or export with a fixed test set
Frozen cases and evidence view
Screening only
Fast candidate diagnosis
No deep analysis or publication
Technical check
Tool-call, log, and infrastructure anomalies
Technical signal only; no semantic verdict
Appendix
Table 15: Inspection modes.
Stage
Required artifact
Stop condition
Freeze
Source, case identities, evidence view
Identity or snapshot not fixed
Screen
Candidate score, evidence anchors, risk tier
Screening failure
Admit
Ranked pool and admitted queue
Capacity or strata unavailable
Deep analysis
Structured case record with claim ledger
Unsupported claims
Review
Independent verdict and correction history
Conflict, missing evidence, malformed output
Gate and audit
Terminal verdict, counts, denominator
Missing field, count mismatch, leakage
Appendix
Table 16: Inspection stage contract.
Case 1: expected behavior
Case 2: confirmed Bad Case
Customer issue
Refund for an order cancelled because the courier was late
Charged, but the order was never confirmed
Retained window
9 entries over 45 s
15 entries over 5.5 min
Tool calls
6: Skill load, context resolver
8: Skill load, context resolver, order detail
Review
First review FAIL; reconciled; second review PASS as expected behavior, high confidence
First review PASS as confirmed Bad Case, high confidence
Gate and outcome
Upload not allowed; closed as no issue
Notification gate PASS; uploaded as a Bad Case
Inspection time
26 min
18.5 min
Appendix
Table 17: The two inspected cases at a glance.
#
Speaker
Message (translated, condensed)
Action and tools
G01
Agent
Welcome. Is your issue “When will I get my money back?”
Entry FAQ
G02–03
User
No. / The Merchant A order.
G04
Agent
Do you mean an order from Merchant A? Please tell me the issue, e.g. a missing item, a wrong order, cold food, or a late delivery.
INTENT_GUIDE
G05–07
User
I cancelled it and the money has not been returned. / The driver was late. / It has been over an hour and the money has not been returned.
G08
Agent
Since the order was cancelled because the driver was late, please confirm the order in the list and I will check its refund status.
Skill loaded; resolver returns no order ID; ORDER_CONFIRMATION
I ordered from Merchant A and you took the money. Why was the order not confirmed?
B06 ×
Agent
I checked the Merchant B order: it was completed and delivered, and has no refund request. If this is not the right order, please give the order number.
Resolver returns the Merchant B order; order detail: completed on 29 July, no after-sale record; GUIDE
Appendix
Table 19: Case 2 dialogue (confirmed Bad Case). Rows marked × contain the defect.
Axis
Question
Passing value
Disposition
Is the case repaired by the proposed decision?
repaired
Eligibility
May the pair be used for training at all?
eligible
Fault relation
Is the anchored output the model’s own fault?
model fault
Symptom
Is the inspected symptom corrected?
corrected
Boundary
Is the change confined to the anchored decision?
corrected
Policy
Does the repair follow the incident-time SOP?
compliant
Appendix
Table 20: Axes of the independent repair verifier.
Item
Setting
Student initialization
V4, the previous version; full-parameter update, vision tower trainable
Teacher
V4, frozen for the whole run, not a moving average; scores the student’s rollout token by token; targets detached and kept in FP32
Privileged information
appended to the teacher’s system message; never in the student context
Training set
280 repair tuples after micro-scenario capping (Section 6.3 ); 281 candidates, one excluded for exceeding four images per tuple
Objective
forward KL partial sum over the teacher’s top 32 tokens, floored at zero per token (Eq. 29 ); backpropagated directly; loss coefficient 1.0; mean over all generated tokens of the batch (Eq. 31 )
Not used
task reward, policy gradient, tail cell, renormalization within the top- K set, probability floor or smoothing, auxiliary SFT or DPO loss
Appendix
Table 21: OPSD training configuration of the final model.
Recursive self-improvement (RSI) envisions agent systems that autonomously improve their own models and harnesses, with each updated system continuing to improve itself. Existing approaches typically restrict optimization objectives, editable components, or evaluation procedures, limiting open-ended exploration and continuity across generations. We introduce RSI-Env, a persistent environment for coordinated model-harness improvement, supporting autonomous training, harness modification, and retention of research state. An independent Host Controller manages services, evaluation, and versioning. Under Self, each accepted model-harness version takes over the next improvement round. We define three settings that vary goal specificity and access to benchmark feedback: Open-General, Capability-Directed, and Benchmark-Guided. Across these three settings, we compare Self with two fixed external Teachers on seven tasks, starting from Qwen3.8-27B with a minimal harness. Without benchmark feedback, Open-General Self raises GPQA Diamond accuracy from 65.2% to 73.2% and AIME 2026 accuracy from 53.3% to 56.7%. All three Capability-Directed Self runs achieve target-aligned gains. Knowledge Self demonstrates continued improvement across system updates, with SimpleQA accuracy rising from 82.0% to 85.0% and then 88.0%. Benchmark guidance and Teacher benefits vary by task. These results demonstrate RSI-Env's support for open-ended exploration and sustained recursive improvement within the studied runs, while highlighting challenges in preserving gains across capabilities.
Wanying Qu, Zhiyuan Fan, Jiyao Liu +4
Fudan University · The Hong Kong University of Science and Technology · Georgia Institute of Technology +3
Medical agents increasingly combine general reasoning models with specialized clinical tools, yet their capabilities remain largely fixed by what clinicians and engineers design before deployment. Recursive self-improvement (RSI) offers a different paradigm in which agents learn from their own failures and autonomously expand their capabilities, but directly applying RSI to medicine introduces fundamental safety challenges. We introduce MedRSI, the first recursive self-improvement framework for medicine, which continuously transforms diagnostic failures into new clinical capabilities through tool composition and task-specific model training. Inspired by clinical practice, MedRSI introduces two mechanisms for clinically aligned self-evolution. Clinical-cost-aware failure prioritization directs improvement toward errors according to their potential clinical consequences rather than frequency alone. Fast discovery with slow registration separates rapid capability invention from conservative adoption, allowing new tools to enter the persistent agent only after demonstrating sustained benefit across subsequent patient cohorts. Across public glaucoma and heart disease benchmarks and two private clinical tasks, MedRSI progressively develops segmentation, measurement, prediction, multimodal reasoning, and generative capabilities, surpasses manually engineered medical agents, and autonomously discovers solutions to clinical problems not anticipated by its original designers. Our results show that medical agents need not remain constrained by capabilities specified before deployment: with clinically grounded mechanisms governing what to improve and what to retain, they can continuously construct, validate, and accumulate new capabilities from diagnostic experience. Code is available at https://github.com/ImprintLab/MedRSI.
Recursive self-improvement (RSI) requires a concrete mechanism through which an AI system observes its capabilities and converts that evidence into the next round of learning. We present NeoHorse-1, a family of agent-native models developed to explore this path through agentic post-training. Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn. These records are converted into training examples that preserve interleaved reasoning, tool calls, and harness context, and are admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling. Routing signals organize supervised fine-tuning into a three-stage curriculum and extend to routing-guided on-policy distillation, where a teacher supervises student-generated responses under the same progression. Capability-guided allocation then converts evaluation feedback into the next training mixture, closing an evaluation-selection-update loop in which what the system learns to do shapes what it learns from next. Across eleven benchmarks covering harness-based agents, tool use, coding, and instruction following, post-training raises the macro-average from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B, substantially narrowing the aggregate gap between the post-trained 4B model and the 9B base model. We further adapt NeoHorse-1-4B into NeoHorse-Jev-4B, a prefill-only model that returns structured decisions and probabilities for agent workflows. NeoHorse-1 provides an initial prototype of this feedback-driven process and a path toward harness-mediated RSI across successive iterations.