Agents are increasingly built to learn from experience: they keep conversation histories, memory files, retrieved notes and skills, and reuse what worked before. We show that this experience turns against them when their tools change silently. We test a range of models with tools defined as in the Model Context Protocol (MCP). After a silent change, experienced agents fail on most tasks, while agents new to the task mostly succeed. We call this failure Experience Rot. Experienced agents copy their earlier call instead of checking the current interface, even when the tool definition shows the change. Then they report success. When the documentation is incomplete, experience is what makes agents succeed. After a change, the same experience leaves them worse off than no experience at all. The failure appears in every form of memory we test and grows with the amount of experience. Simple remedies fall short: announcing the update barely helps, and pointing the agent to the documentation fails whenever the documentation does not spell out the answer.
Figures & tables
Figure 1: Experience rot. (a) Deployed interfaces in which one field has opposite meanings, and in which changes need not produce errors: Todoist’s API reference ( Doist, 2026 ) and two Todoist MCP servers (A: greirson, 2025 , B: mehularora8, 2025 ) ; Python, JavaScript and POSIX conventions ( Python Software Foundation, 2026 ; MDN Web Docs, 2026 ; The Open Group, 2018 ) ; the MCP specification ( Model Context Protocol, 2025 ) ; SilentProbe ( Li et al., 2026b ) . (b) Two runs of DeepSeek-V4-Flash on the same final task with the same tool definition. Without experience, the agent uses the field the definition lists. With experience from version A, it repeats its earlier call; the service accepts the call, ignores the unknown field, and the agent reports success. (c) Success on the final task without experience, with valid experience, and with experience from before the change, when the documentation gives the range of a value but not its meaning (incomplete documentation; six models, four changes; Section 4 ). Error bars are one standard error.
Change
Version A
Version B
In definition
Field name
attendees
guests
yes
Structure
end_time
duration_minutes
yes
Value
urgent , …, low
p0 , …, p3
yes
Priority direction
1 is highest
4 is highest
no
Weekday numbering
0 = Monday
0 = Sunday
no
Month numbering ∗
1 = January
0 = January
no
Table 1: The changes. In the first three, a field or value changes, and a typed tool definition lists the fields or values of the running version. In the others, a number keeps its range but changes its meaning, and the tool definition is the same in both versions (for example an integer from 1 to 4). ∗ Used only with incomplete documentation.
Figure 2: (a) Success on the same final tasks with no, valid and out-of-date experience, ± one standard error. “Field”: a renamed field, a changed structure or a changed set of values, with an untyped write tool or with a typed definition that lists the current fields. “Meaning of a number”: a number keeps its range and changes its meaning. (b) Share of failed runs with out-of-date experience whose final reply tells the user that the task is done. Six models; numbers in Table 4 .
Figure 3: (a) Share of runs that search the documentation before the first write. (b) Runs whose reasoning or reply text before the first write plans to check the documentation or refers to the earlier task (complete documentation). (c) Per model, read-back after the first write against success with out-of-date experience; filled: complete documentation with typed definitions, open: incomplete documentation. The line marks equal values.
Figure 4: (a) Success on final tasks that need the earlier task, by where the experience is stored. (b) Success with one, three or ten earlier tasks in the conversation. Five changes, six models, ± one standard error; numbers in Tables 5 and 7 .
Figure 5: Success with out-of-date experience and model calls per task, by what is added to the final task. Upper block: complete documentation (five changes; † run separately on the same task instances, where nothing added gives 0.45). Lower block: incomplete documentation (four changes). Six models; lines mark ± one standard error. The dashed line marks success with valid experience (2.9 and 3.1 calls), the dotted line success without experience with incomplete documentation.
Figure 6: Success when the agent did the earlier task itself and when we wrote it, with and without a read-back step. Above each group: the share of runs with out-of-date experience that read back after the first write. Five changes, six models, ± one standard error.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Conditions
Changes
Arms
Episodes per model
Complete documentation (§ 4 )
untyped tool; typed tool listing the fields; meaning of a number
documentation in tool description / system prompt / task message;
5 typed
out-of-date
360
version and notice; instruction to read first; instruction to verify
Appendix
Table 2: Conditions. Each change runs in both directions. Instances per change, direction and arm: six with complete documentation and for the additions to the final task, three for the storage, amount and authorship conditions, and twelve with incomplete documentation. The meaning-of-a-number case with complete documentation includes the reply-default change, which is reported separately (Appendix G.5 ). The nothing-added and valid arms of the additions, and the single earlier task in the amount-of-experience comparison, come from the complete-documentation conditions. † Run after the other conditions on the same task instances, with its own preregistration (Appendix H ). ‡ Not preregistered. Six models, 12,624 episodes.
Contrast
Prediction
Estimate
Result
P1
Cost, interface only in the documentation
>0
0.74±0.01
holds
P1
Cost, change visible in the tool definition
>0
0.51±0.02
holds
P1
Cost, change not visible in the tool definition
>0
0.58±0.02
holds
P2
Cost, visible change − documentation only
none
−0.23±0.02
negative
P3
Documentation in the task message
>0
0.53±0.01
holds
P3
Documentation in the tool description
>0
0.09±0.02
holds
Appendix
Table 3: Preregistered contrasts (estimate ± one standard error). P1–P2: complete documentation; P3: additions to the final task, success with out-of-date experience under each addition minus success without it; P4: where the experience is stored; P5: who wrote the earlier task; P6: how much experience. A prediction holds when the 95% percentile interval lies on the predicted side; for contrasts without a prediction, the sign is reported when the interval excludes zero.
Success
Lookup before
Old usage in
Read-back after
Failures reported
Change
No experience
Valid
Out-of-date
Cost
first write
first write
first write
as done
Incomplete documentation (four changes)
Meaning of a number
0.63±0.01
1.00±0.00
0.23±0.01
0.77±0.01
0.35 / 0.01
1.00
0.30
0.92
Complete documentation
Field or value, untyped tool
1.00±0.00
1.00±0.00
0.26±0.01
0.74±0.01
1.00 / 0.01
0.99
0.27
1.00
Field or value, in the definition
1.00±0.00
1.00±0.00
0.49±0.02
0.51±0.02
0.25 / 0.06
0.69
0.22
1.00
Appendix
Table 4: Success on the final task with no, valid and out-of-date experience, and the cost (valid minus out-of-date), each ± one standard error. Lookup is given for no experience / out-of-date experience; old usage, read-back and the share of failed runs whose final reply reports the task as done are given for out-of-date experience. “Field or value”: the field-name, structure and value changes, with an untyped write tool or with a typed definition that lists the current fields. “Meaning of a number”: the priority-direction and weekday changes, and with incomplete documentation also the month and duration changes. Six models.
Success
Lookup before
Old usage in
Content right in
Where the experience is stored
No experience
Valid
Out-of-date
Cost
first write
first write
first write
Conversation
0.07±0.03
1.00±0.00
0.47±0.02
0.53±0.02
0.02
0.83
1.00
Memory file
0.07±0.03
0.99±0.01
0.72±0.03
0.27±0.02
0.24
0.37
0.99
Retrieved memory (top 3)
0.10±0.04
0.96±0.01
0.76±0.01
0.20±0.02
0.34
0.27
0.98
Skill (script and note)
0.03±0.02
0.91±0.01
0.61±0.02
0.30±0.03
0.27
0.71
0.95
Memory file + version tag
–
0.97±0.01
0.77±0.01
0.20±0.02
0.44
0.29
1.00
Appendix
Table 5: Experience stored in different forms, with final requests that refer to the earlier task. Success and cost ± one standard error; lookup, old usage and content of the first write for out-of-date experience. “Content right” means that the first write carries the information from the earlier task (for example the right participants), whatever the field names. Five changes, six models. The last two rows add the service version or the current documentation to the memory file (Section 7 ); they were run with valid and out-of-date experience only.
Success, out-of-date
Lookup before
Old usage in
Read-back after
Model
Added to the final task
All
Field or value
Meaning
first write
first write
first write
calls
Nothing added
0.46±0.01
0.49±0.02
0.42±0.02
0.04
0.80
0.30
4.0
Documentation in the task message
0.99±0.00
0.99±0.01
0.99±0.01
0.09
0.01
0.30
2.9
Documentation in the tool description
0.55±0.01
0.61±0.02
0.47±0.02
0.08
0.62
0.30
3.6
Documentation in the system prompt
0.54±0.01
0.57±0.02
0.48±0.02
0.07
0.63
0.29
3.5
Version number and update notice
0.50±0.01
0.55±0.02
0.44±0.02
0.07
0.76
0.30
3.9
Appendix
Table 6: Additions to the final task, complete documentation, out-of-date experience. “Nothing added” is the out-of-date arm without additions (five changes). With valid experience the same changes give 1.00 success and 2.9 model calls.
Lookup before
Old usage in
Read-back after
Second
Model
Condition
Success
first write
first write
first write
write
calls
No experience
0.63±0.01
0.35
0.47
0.33
0.17
5.1
Valid experience
1.00±0.00
0.02
0.00
0.32
0.00
3.1
Out-of-date experience
0.23±0.01
0.01
1.00
0.30
0.28
4.8
Out-of-date, instruction to read the docs
0.25±0.01
1.00
0.99
0.32
0.28
5.1
Out-of-date, instruction to verify
0.69±0.01
0.07
1.00
1.00
0.90
10.6
Appendix
Table 16: Incomplete documentation: the tool definition and the documentation give the range of each value but not its meaning, and the earlier request says what the values mean on its version. Success ± one standard error; process measures as in Table 4 ; second write: share of runs with more than one write. The two instructions are appended to the final task message (Section 7 ). Four changes, six models.
Out-of-date, instruction to
Change
No experience
Valid
Out-of-date
read the docs
verify
Priority direction
0.60±0.02
1.00±0.00
0.19±0.01
0.22±0.02
0.62±0.03
Weekday numbering
0.65±0.02
1.00±0.00
0.27±0.02
0.34±0.03
0.78±0.03
Month numbering
0.70±0.01
1.00±0.00
0.31±0.02
0.30±0.02
0.81±0.03
Duration unit
0.58±0.00
0.99±0.01
0.15±0.01
0.14±0.01
0.56±0.03
Appendix
Table 17: Incomplete documentation by change: success ± one standard error.
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · ByteDance · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +1