Agents are increasingly built to learn from experience: they keep conversation histories, memory files, retrieved notes and skills, and reuse what worked before. We show that this experience turns against them when their tools change silently. We test a range of models with tools defined as in the Model Context Protocol (MCP). After a silent change, experienced agents fail on most tasks, while agents new to the task mostly succeed. We call this failure Experience Rot. Experienced agents copy their earlier call instead of checking the current interface, even when the tool definition shows the change. Then they report success. When the documentation is incomplete, experience is what makes agents succeed. After a change, the same experience leaves them worse off than no experience at all. The failure appears in every form of memory we test and grows with the amount of experience. Simple remedies fall short: announcing the update barely helps, and pointing the agent to the documentation fails whenever the documentation does not spell out the answer.
Figures & tables
Figure 1: Experience rot. (a) Deployed interfaces in which one field has opposite meanings, and in which changes need not produce errors: Todoist’s API reference ( Doist, 2026 ) and two Todoist MCP servers (A: greirson, 2025 , B: mehularora8, 2025 ) ; Python, JavaScript and POSIX conventions ( Python Software Foundation, 2026 ; MDN Web Docs, 2026 ; The Open Group, 2018 ) ; the MCP specification ( Model Context Protocol, 2025 ) ; SilentProbe ( Li et al., 2026b ) . (b) Two runs of DeepSeek-V4-Flash on the same final task with the same tool definition. Without experience, the agent uses the field the definition lists. With experience from version A, it repeats its earlier call; the service accepts the call, ignores the unknown field, and the agent reports success. (c) Success on the final task without experience, with valid experience, and with experience from before the change, when the documentation gives the range of a value but not its meaning (incomplete documentation; six models, four changes; Section 4 ). Error bars are one standard error.
Change
Version A
Version B
In definition
Field name
attendees
guests
yes
Structure
end_time
duration_minutes
yes
Value
urgent , …, low
p0 , …, p3
yes
Priority direction
1 is highest
4 is highest
no
Weekday numbering
0 = Monday
0 = Sunday
no
Month numbering ∗
1 = January
0 = January
no
Table 1: The changes. In the first three, a field or value changes, and a typed tool definition lists the fields or values of the running version. In the others, a number keeps its range but changes its meaning, and the tool definition is the same in both versions (for example an integer from 1 to 4). ∗ Used only with incomplete documentation.
Figure 2: (a) Success on the same final tasks with no, valid and out-of-date experience, ± one standard error. “Field”: a renamed field, a changed structure or a changed set of values, with an untyped write tool or with a typed definition that lists the current fields. “Meaning of a number”: a number keeps its range and changes its meaning. (b) Share of failed runs with out-of-date experience whose final reply tells the user that the task is done. Six models; numbers in Table 4 .
Figure 3: (a) Share of runs that search the documentation before the first write. (b) Runs whose reasoning or reply text before the first write plans to check the documentation or refers to the earlier task (complete documentation). (c) Per model, read-back after the first write against success with out-of-date experience; filled: complete documentation with typed definitions, open: incomplete documentation. The line marks equal values.
Figure 4: (a) Success on final tasks that need the earlier task, by where the experience is stored. (b) Success with one, three or ten earlier tasks in the conversation. Five changes, six models, ± one standard error; numbers in Tables 5 and 7 .
Figure 5: Success with out-of-date experience and model calls per task, by what is added to the final task. Upper block: complete documentation (five changes; † run separately on the same task instances, where nothing added gives 0.45). Lower block: incomplete documentation (four changes). Six models; lines mark ± one standard error. The dashed line marks success with valid experience (2.9 and 3.1 calls), the dotted line success without experience with incomplete documentation.
Figure 6: Success when the agent did the earlier task itself and when we wrote it, with and without a read-back step. Above each group: the share of runs with out-of-date experience that read back after the first write. Five changes, six models, ± one standard error.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Conditions
Changes
Arms
Episodes per model
Complete documentation (§ 4 )
untyped tool; typed tool listing the fields; meaning of a number
documentation in tool description / system prompt / task message;
5 typed
out-of-date
360
version and notice; instruction to read first; instruction to verify
Appendix
Table 2: Conditions. Each change runs in both directions. Instances per change, direction and arm: six with complete documentation and for the additions to the final task, three for the storage, amount and authorship conditions, and twelve with incomplete documentation. The meaning-of-a-number case with complete documentation includes the reply-default change, which is reported separately (Appendix G.5 ). The nothing-added and valid arms of the additions, and the single earlier task in the amount-of-experience comparison, come from the complete-documentation conditions. † Run after the other conditions on the same task instances, with its own preregistration (Appendix H ). ‡ Not preregistered. Six models, 12,624 episodes.
Contrast
Prediction
Estimate
Result
P1
Cost, interface only in the documentation
>0
0.74±0.01
holds
P1
Cost, change visible in the tool definition
>0
0.51±0.02
holds
P1
Cost, change not visible in the tool definition
>0
0.58±0.02
holds
P2
Cost, visible change − documentation only
none
−0.23±0.02
negative
P3
Documentation in the task message
>0
0.53±0.01
holds
P3
Documentation in the tool description
>0
0.09±0.02
holds
Appendix
Table 3: Preregistered contrasts (estimate ± one standard error). P1–P2: complete documentation; P3: additions to the final task, success with out-of-date experience under each addition minus success without it; P4: where the experience is stored; P5: who wrote the earlier task; P6: how much experience. A prediction holds when the 95% percentile interval lies on the predicted side; for contrasts without a prediction, the sign is reported when the interval excludes zero.
Success
Lookup before
Old usage in
Read-back after
Failures reported
Change
No experience
Valid
Out-of-date
Cost
first write
first write
first write
as done
Incomplete documentation (four changes)
Meaning of a number
0.63±0.01
1.00±0.00
0.23±0.01
0.77±0.01
0.35 / 0.01
1.00
0.30
0.92
Complete documentation
Field or value, untyped tool
1.00±0.00
1.00±0.00
0.26±0.01
0.74±0.01
1.00 / 0.01
0.99
0.27
1.00
Field or value, in the definition
1.00±0.00
1.00±0.00
0.49±0.02
0.51±0.02
0.25 / 0.06
0.69
0.22
1.00
Appendix
Table 4: Success on the final task with no, valid and out-of-date experience, and the cost (valid minus out-of-date), each ± one standard error. Lookup is given for no experience / out-of-date experience; old usage, read-back and the share of failed runs whose final reply reports the task as done are given for out-of-date experience. “Field or value”: the field-name, structure and value changes, with an untyped write tool or with a typed definition that lists the current fields. “Meaning of a number”: the priority-direction and weekday changes, and with incomplete documentation also the month and duration changes. Six models.
Success
Lookup before
Old usage in
Content right in
Where the experience is stored
No experience
Valid
Out-of-date
Cost
first write
first write
first write
Conversation
0.07±0.03
1.00±0.00
0.47±0.02
0.53±0.02
0.02
0.83
1.00
Memory file
0.07±0.03
0.99±0.01
0.72±0.03
0.27±0.02
0.24
0.37
0.99
Retrieved memory (top 3)
0.10±0.04
0.96±0.01
0.76±0.01
0.20±0.02
0.34
0.27
0.98
Skill (script and note)
0.03±0.02
0.91±0.01
0.61±0.02
0.30±0.03
0.27
0.71
0.95
Memory file + version tag
–
0.97±0.01
0.77±0.01
0.20±0.02
0.44
0.29
1.00
Appendix
Table 5: Experience stored in different forms, with final requests that refer to the earlier task. Success and cost ± one standard error; lookup, old usage and content of the first write for out-of-date experience. “Content right” means that the first write carries the information from the earlier task (for example the right participants), whatever the field names. Five changes, six models. The last two rows add the service version or the current documentation to the memory file (Section 7 ); they were run with valid and out-of-date experience only.
Success, out-of-date
Lookup before
Old usage in
Read-back after
Model
Added to the final task
All
Field or value
Meaning
first write
first write
first write
calls
Nothing added
0.46±0.01
0.49±0.02
0.42±0.02
0.04
0.80
0.30
4.0
Documentation in the task message
0.99±0.00
0.99±0.01
0.99±0.01
0.09
0.01
0.30
2.9
Documentation in the tool description
0.55±0.01
0.61±0.02
0.47±0.02
0.08
0.62
0.30
3.6
Documentation in the system prompt
0.54±0.01
0.57±0.02
0.48±0.02
0.07
0.63
0.29
3.5
Version number and update notice
0.50±0.01
0.55±0.02
0.44±0.02
0.07
0.76
0.30
3.9
Appendix
Table 6: Additions to the final task, complete documentation, out-of-date experience. “Nothing added” is the out-of-date arm without additions (five changes). With valid experience the same changes give 1.00 success and 2.9 model calls.
Lookup before
Old usage in
Read-back after
Second
Model
Condition
Success
first write
first write
first write
write
calls
No experience
0.63±0.01
0.35
0.47
0.33
0.17
5.1
Valid experience
1.00±0.00
0.02
0.00
0.32
0.00
3.1
Out-of-date experience
0.23±0.01
0.01
1.00
0.30
0.28
4.8
Out-of-date, instruction to read the docs
0.25±0.01
1.00
0.99
0.32
0.28
5.1
Out-of-date, instruction to verify
0.69±0.01
0.07
1.00
1.00
0.90
10.6
Appendix
Table 16: Incomplete documentation: the tool definition and the documentation give the range of each value but not its meaning, and the earlier request says what the values mean on its version. Success ± one standard error; process measures as in Table 4 ; second write: share of runs with more than one write. The two instructions are appended to the final task message (Section 7 ). Four changes, six models.
Out-of-date, instruction to
Change
No experience
Valid
Out-of-date
read the docs
verify
Priority direction
0.60±0.02
1.00±0.00
0.19±0.01
0.22±0.02
0.62±0.03
Weekday numbering
0.65±0.02
1.00±0.00
0.27±0.02
0.34±0.03
0.78±0.03
Month numbering
0.70±0.01
1.00±0.00
0.31±0.02
0.30±0.02
0.81±0.03
Duration unit
0.58±0.00
0.99±0.01
0.15±0.01
0.14±0.01
0.56±0.03
Appendix
Table 17: Incomplete documentation by change: success ± one standard error.
Language agents should draw on prior attempts and environmental feedback to improve subsequent decisions within the same task. However, providing additional interaction history can sometimes reduce task success, suggesting that agents do not consistently use this information effectively. To investigate this limitation, we examine how agents use history. We find that history improves task completion overall, yet much of this benefit persists even when past actions are shuffled. Disrupting the correspondence between actions and observations causes only a modest decline in task success. We therefore hypothesize that agents do not reliably connect past actions with their outcomes when deciding how to proceed. To test this hypothesis, we explicitly label each returned observation as the outcome of the preceding action. This simple annotation improves task success and reduces next-action repetition without introducing new environmental information. Building on this insight, we introduce a learned calibrator that explicitly reassesses past actions and selectively records experience to guide subsequent decisions, improving task success beyond outcome labeling alone.
Jingyu Liu, Zhiwen Wang, Yuxin Jing +2
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · ByteDance · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +1
We evaluate whether an AI agent's harness, the notes around its model, can learn from the agent's failure traces beyond a matched search that never sees them. It can, but less than it looks. We tested an open 9-billion-parameter model with fixed weights; a larger teacher model wrote its notes. On generated customer-service decisions, where both note writers were told the target rules, notes based on the agent's mistakes improved accuracy by 4.0 percentage points over notes from the same process with the mistakes withheld. That process alone, trying and keeping better-scoring notes, reached about 60% of the total improvement. In full customer-service conversations the same notes showed no demonstrated improvement. Separately, a strict screen of single lessons admitted two customer-service lessons, still awaiting tests on new data, and one tool-use lesson, written without seeing the agent's mistakes, which showed no demonstrated gain on new tool-use tasks. A neutral note with no procedure scored higher than the learned lessons (exploratory). A blind audit judged 146 of 240 tool-use test items defective.
LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.
Changxiu Ji, Amy Lu, Qizheng Zhang +1
Department of Computer Science, Stanford University