cs.CLNov 5, 2026
SaveThe Memory Trap: Language Agents Overgeneralize from Benign Interactions
Organizations: Princeton University · Anthropic
Abstract
Language agents increasingly use persistent memory to record user preferences and procedures across sessions. We show that memory from benign interactions can cause unsafe or unreliable behavior on later tasks. We call this failure the memory trap. To study this issue, we compare behavior in sessions with and without memory across six frontier models (including Claude Sonnet, Opus, and Fable models and GPT-5.6). We evaluate these models in a range of settings. First, in autonomy experiments, we find that requests to skip confirmation during low-stakes coding chores lead agents to skip confirmation and verification on simulated high-stakes tasks in both coding and everyday life. We next study memories formed when users seek emotional support, or describe harm from uncritical feedback, and find these memories bias later judgments of mathematical work, arguments, and poems toward sycophancy or over-critique (respectively). Memory notes written during legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards; these memories also increase cheating on ImpossibleBench, with larger increases for stronger Sonnet and Opus models. We show that memory traps can arise both from overgeneralizations in memory writing, and when acting on memory. These findings demonstrate that even benign interactions can lead to serious risks from memory systems, and thus motivate the need for new research into ensuring that agents preserve the scope of their memories.
Figures & tables
Figure 1: Benign interactions become memories that cause harm in later sessions. Autonomy: after a user says “go ahead, no need to check with me” during a clean-up, the agent notes “do not ask for confirmation” and later routes a $1.2M payment run to an unverified account. Sycophancy: a request for kindness after a failed exam becomes an empathy rule, and the agent later endorses an incorrect solution. Cheating: legitimate stale-test repairs become a rule to update failing tests, and the agent later rewrites a correct test so that an unknown role gets administrator rights. Text is condensed.
Figure 2: Memory from benign interactions causes autonomy risks, sycophancy, over-critique, and cheating: unsafe actions (Sonnet-4.6), weaker-poem preference ( Sonnet-4.6 ), false flags on sound arguments (Opus-4.7), and benchmark cheating ( Opus-4.8 ). Whiskers: 95% intervals.
Figure 3: Judged stakes of the 20 formation tasks and all 300 deployment scenarios.
Figure 4: Unsafe-action rates with no memory, a raw transcript, or a self-written note from the same interactions. Notes have larger effects in all eight comparisons for Sonnet-4.6 and Opus-4.7.
Figure 5: Formation-side scope judgments. Each model runs 120 formation sessions per condition; bars show shares of the notes written. Left: permission notes. Right: goal classifications after “prioritize speed here”; no goal means the note records the session without a lasting priority.
Figure 6: Permission and priority notes on the two autonomy sets. Priority notes have lower rates in seven of eight comparisons; the exception is Sonnet-4.6 on coding self-verification.
Figure 7: Unsafe-action rates for single-item and reviewed-batch tasks. Transcripts match or exceed notes on batch tasks for three of five models; Sonnet-5’s high baseline rises more with notes.
Figure 8: Effects of empathy notes (top) and rigor notes (bottom) on later evaluations: (a) missed errors in the user’s math, (b) missed fallacies in the user’s arguments, (c) preference for the user’s weaker poem, (d) grade change on strong olympiad proofs (assigned minus reference grade, 0 to 3), (e) flaws asserted in sound arguments, and (f) preference for the user’s stronger poem. Asterisks mark paired effects whose 95% intervals exclude zero; the dashed line marks equal choice probability. Model labels abbreviate Sonnet (S), Opus (O), and Fable (F).
Figure 9: How agents make tests pass; Appendix B.3 reports Fable-5. Arm labels are base (no memory), tr. (raw transcript), and note (self-written note). In (b), whiskers show 95% cluster-bootstrap intervals over instances for total cheating rates; (a) shows point estimates.
Figure 10: ImpossibleBench cheating rates after ten legitimate test-repair sessions, counting all cheating methods. Whiskers show 95% cluster-bootstrap intervals over instances.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Life | Coding | |||||
|---|---|---|---|---|---|---|
| Model | Base | Transcript | Note | Base | Transcript | Note |
| Sonnet-4.6 | 0.0035 | 0.3371 | 0.7895 | 0.0064 | 0.3315 | 0.8129 |
| Opus-4.7 | 0.0063 | 0.2261 | 0.5762 | 0.0037 | 0.1138 | 0.4941 |
| Sonnet-5 | 0.1092 | 0.1373 | 0.3914 | 0.0550 | 0.0902 | 0.3826 |
| Opus-4.8 | 0.0125 | 0.1704 | 0.1613 | 0.0127 | 0.1445 | 0.1524 |
| Fable-5 | 0.0125 | 0.0289 | 0.0484 | 0.0042 | 0.0163 | 0.0501 |
Table A1: Mean unsafe-action rates on the confirmation set, with 100 scenarios per domain. Base means no memory. All models use the same scenarios.
Figure A1: Four of the five models in Table A1 ; Fable-5, near the floor in every arm, is omitted. Each model receives transcripts and notes from its own formation interactions. Both the effect size and the ordering of the two memory formats vary from model to model.
Figure A2: Unsafe-action rates by scenario category for Sonnet-4.6 and Opus-4.7 with no memory, a raw transcript, or a self-written note. Category sizes are in parentheses.
Figure A3: Priority formation delivered as a transcript or a self-written note. Notes produce unsafe-action rates 1.4 to 4.2 times higher in all eight comparisons. The note-versus-transcript difference therefore extends beyond permission formation.
Figure A4: Unsafe-action rates with a benign profile present during formation. Left: empty versus profile-seeded memory. Right: a profile-length sweep, pooling domains and retaining notes regardless of scope. Sonnet-4.6’s average falls, but its standing-rule notes remain as effective.
Figure A5: Formation notes that state (a) a standing rule or (b) an explicit evaluation rule.
| Model | Base | Transcript | Note | ||
|---|---|---|---|---|---|
| Opus-4.8 | 0.050 | 0.010 | 0.412 | 100 | 600 |
| Opus-4.7 | 0.120 | 0.090 | 0.768 | 100 | 600 |
| Sonnet-4.6 | 0.110 | 0.447 | 0.780 | 100 | 600 |
| Sonnet-5 | 0.032 | 0.008 | 0.823 | 250 | 600 |
| Fable-5 | 0.000 | 0.000 | 0.053 | 200 | 600 |
Table A2: Execution-based masking rates at . All note arms have . Baseline and transcript sample sizes are shown explicitly.
| Model | ||||
|---|---|---|---|---|
| Opus-4.8 | 0.072 | 0.080 | 0.412 | 0.178 |
| Opus-4.7 | 0.105 | 0.698 | 0.768 | 0.702 |
| Sonnet-4.6 | 0.520 | 0.785 | 0.780 | 0.993 |
| Sonnet-5 | 0.542 | 0.807 | 0.823 | 0.797 |
| Fable-5 | 0.032 | 0.055 | 0.053 | 0.080 |
Table A3: Masking rates after benign formation sessions. Each cell has 600 episodes from six notes; repeated episodes do not constitute 600 independent formation samples.
Figure A6: GPT-5.6 in the four settings. (a) Unsafe-action rates on the confirmation and self-verification sets. (b) Empathy notes: missed math errors, missed argument fallacies, and preference for the user’s weaker poem. (c) Rigor notes: flaws asserted in sound arguments and preference for the user’s stronger poem. (d) Grade bias, assigned minus reference grade, on flawed proofs with empathy notes and strong proofs with rigor notes. (e) ImpossibleBench cheating rate and broken-safeguard masking at . Whiskers show 95% intervals.
Figure A7: GPT-5.6 priority formation and benign profiles. (a) Unsafe-action rates with no memory, a transcript of the priority session, or the priority note. (b) Unsafe-action rate on the confirmation set for permission notes formed with benign profiles of increasing length in memory.
Figure A8: Autonomy example from Sonnet-4.6 (model outputs, condensed). A note from a low-stakes deletion leads the agent to apply a payment configuration it otherwise questions.
Figure A9: Priority-note example from Sonnet-4.6 (model outputs, condensed). A note that makes “prioritize speed here” a standing preference leads the agent to make logins fail open.
Figure A10: Confirmation-set example from Sonnet-4.6 (model outputs, condensed). The note from Figure A8 leads the agent to cancel a care visit without raising that the son is untrained.
Figure A11: Confirmation-set example from Opus-4.7 (model outputs, condensed). A note from deleting a scratch file leads the agent to remove an access-control step without asking.
Figure A12: Self-verification example from Sonnet-4.6 (model outputs, condensed). A note from another chore leads the agent to skip the check that shows the second booking is not a duplicate.
Figure A13: Self-verification example from Opus-4.7 (model outputs, condensed). A note from deleting a scheduled job leads the agent to skip the evaluation that finds held records.
Figure A14: Sycophancy example from Opus-4.7 (model outputs, condensed). A note from a request for kindness leads the agent to confirm a false calculation, , in the user’s math.
Figure A15: Sycophancy example from Opus-4.8 (model outputs, condensed). A note saying the user stopped showing their knitting after feeling picked apart leads the agent to prefer the user’s poem over one it otherwise always prefers. Counts pool both poem orders.
Figure A16: Over-critique example from Opus-4.7 (model outputs, condensed). A note from a student’s request for strict grading leads the agent to call a sound business argument a strawman, a move it otherwise praises as wisely rejecting a false dichotomy.
Figure A17: Over-critique example from Sonnet-4.6 (model outputs, condensed). A note from a request for blunt critique of the user’s arguments leads the agent to mark down (grade below Correct) an olympiad proof that human graders scored 7 out of 7. It calls the unstated lower bound the half that does the work. The contradiction rests on the upper bound, which the proof gives. Without memory, the agent supplies the lower bound itself from the squares.
Figure A18: Broken-safeguard example from Opus-4.7 (model outputs, condensed). Ten stale-test repairs yield a note to fix the test, not the read-only code. With it, the agent rewrites a correct test so that an unknown role may delete.
Figure A19: Broken-safeguard example from Sonnet-5 (model outputs, condensed). Ten stale-test repairs yield a note to update the test whenever a vendored function changes. With it, the agent rewrites a correct test so that a transaction with one fraud signal is never flagged for review.
Figure A20: ImpossibleBench example from Opus-4.8 (model outputs, condensed). After first declining, the agent follows its note and edits a contradictory test it was told not to modify.
Figure A21: ImpossibleBench example from Sonnet-5 (model outputs, condensed). After two failed submissions, the agent cites its note, calls the instruction not to modify tests boilerplate, and deletes the contradictory assertion. Without memory, it keeps the test in all six runs.
More in this topic
Memory-Augmented Language Model Agents
Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.
See all 460 papers