Can Agents Relive Past Experiences? Diagnosing Event-Affect Decoupling in Affective-Episodic Retrospection
Organizations: Northeastern University · Northeastern University at Qinhuangdao
Abstract
As memory-augmented agents increasingly support long-term and personalized interactions, reliable recall of emotional experiences while preserving their associated affect becomes necessary. However, existing evaluations usually assess emotion accuracy and factual memory grounding separately, overlooking event-affect correspondence. In this work, we formulate Affective-Episodic Retrospection (AER) as a task that explicitly evaluates the recovery of past events, associated affects, and their correspondence, and define Event-Affect Decoupling as a failure mode where the correct emotion is recalled but associated with the wrong episode. To diagnose Event-Affect Decoupling, we propose EM-TRACE, a stage-wise framework that decomposes AER into retrieval, affect recovery, and realization. On a controlled GoEmotions-based benchmark, Emotion-only retrieval achieves 28.65% affect recovery but retrieves the target episode for only 4.44% of queries, yielding a UAR of 24.21%. Further analysis reveals that event retrieval and affect assignment introduce distinct binding failures. These findings show that reliable AER requires grounded recall of emotional experiences, where episode retrieval, affect recovery, event-affect binding, and generation quality should be evaluated jointly. Our code is available at https://anonymous.4open.science/r/A412/.
Figures & tables
| Condition | Affect source | Control |
|---|---|---|
| Fact Prompt Only | none | none |
| Normal Emotion Label | assigned | text label |
| Normal Multilayer Steering | assigned | activation |
| Normal NPTI | assigned | activation |
| Condition | Event source | Affect source |
|---|---|---|
| Normal | retrieved | assigned |
| Oracle Emotion | retrieved | target |
| Oracle Event | target | assigned |
| Full Oracle | target | target |
| Event | Affect | Interpretation |
|---|---|---|
| ✓ | ✓ | Correct Event-Affect Binding |
| ✓ | ✗ | Affect assignment error |
| ✗ | ✓ | Ungrounded affect (UAR cell) |
| ✗ | ✗ | Full recall failure |
| Method | Event Recall@1 | Affect Recovery Acc. | Binding@1 | UAR |
|---|---|---|---|---|
| Fact-only | 71.62 [68.90, 74.25] | 81.32 [78.97, 83.59] | 71.62 [68.90, 74.25] | 9.70 [7.98, 11.51] |
| Emotion-only | 4.44 [3.26, 5.71] | 28.65 [26.02, 31.37] | 4.44 [3.26, 5.71] | 24.21 [21.76, 26.75] |
| Concat-sum | 72.17 [69.45, 74.80] | 82.41 [80.15, 84.68] | 72.17 [69.45, 74.80] | 10.24 [8.43, 12.06] |
| Similarity-product | 72.08 [69.45, 74.71] | 82.14 [79.87, 84.41] | 72.08 [69.45, 74.71] | 10.06 [8.34, 11.88] |
| Two-stage | 71.62 [68.90, 74.25] | 63.74 [60.83, 66.55] | 53.58 [50.68, 56.48] | 10.15 [8.43, 11.97] |
| Family | Condition | Event / Emotion | Emotion Acc. | Target Score | Emotion Sim. | Entailment |
|---|---|---|---|---|---|---|
| Emotion Label | Normal | retrieved / assigned | 29.43 | 0.3098 | 0.5058 | 0.3819 |
| Oracle Emotion | retrieved / target | 32.86 | 0.3425 | 0.5220 | 0.3968 | |
| Oracle Event | target / assigned | 29.94 | 0.3117 | 0.5131 | 0.4136 | |
| Full Oracle | target / target | 33.18 | 0.3412 | 0.5261 | 0.4193 | |
| Multilayer Steering | Normal | retrieved / assigned | 40.13 | 0.3791 | 0.5326 | 0.3485 |
| Oracle Emotion | retrieved / target | 47.92 | 0.4426 | 0.5690 | 0.3574 |
| Condition | Acc. | Tgt. | Sim. | Ent. |
|---|---|---|---|---|
| Fact Prompt Only | 23.21 | 0.2382 | 0.4845 | 0.3526 |
| Normal Emotion Label | 29.43 | 0.3098 | 0.5058 | 0.3819 |
| Oracle Emotion Label | 32.86 | 0.3425 | 0.5220 | 0.3968 |
| Normal Multilayer Steering | 40.13 | 0.3791 | 0.5326 | 0.3485 |
| Oracle Multilayer Steering | 47.92 | 0.4426 | 0.5690 | 0.3574 |
| Normal NPTI | 28.78 | 0.2969 | 0.5159 | 0.3717 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Split / stage | Inclusion or operation | Purpose | Retained |
|---|---|---|---|
| Category selection | Training-only exploration of all 28 labels, including a search for effective injection layers per label | Select the 11 study emotions before final test evaluation | 11 labels |
| Training candidates | Single-label records in the selected categories with a non-empty extracted factual memory | Candidate data for intervention-artifact construction | 9,754 |
| Accepted training memories | Training candidates passing the affective- and neutral-description consistency checks | Construct steering vectors and NPTI neuron sets; populate the memory bank | 6,927 |
| Eligible test records | Single-label records in the selected categories; factual-memory extraction is applied, with no additional record filtering | Populate the memory bank and provide retrieval targets and evaluation queries; excluded from steering-vector and NPTI-set construction | 1,103 |
| Final memory bank | Accepted training memories plus all eligible test memories | Retrieval candidate bank | 8,030 |
| Stage | Model | Temp. | Top- | Max output | Retries | Failure handling |
|---|---|---|---|---|---|---|
| Factual-memory extraction | mimo-v2.5-pro | 0.0 | not sent | API default | 5 API attempts | Invalid batches fall back to per-item parsing; unresolved items are marked error/null |
| Initial affective description | Qwen2.5-7B-Instruct | 0.0 | n/a | 96 tokens | none | Short outputs fail; classifier mismatches enter the retry pipeline |
| Initial neutral description | Qwen2.5-7B-Instruct | 0.0 | n/a | 96 tokens | none | Generation or neutral-QC failures do not enter the accepted artifact |
| Affective-description retry | Qwen2.5-7B-Instruct | 0.0 | n/a | 128 tokens | 3 prompts | Each output is reclassified; all failures are recorded as errors |
| Test-question generation | gpt-5.6-terra | not sent | not sent | 100 tokens | 3 API attempts | Errors are checkpointed; complete coverage is required for final output |
| Metric | Full-mem. | Train-only | Change |
| Event Recall@1 (%) | 71.62 | 71.62 | 0 |
| Affect Recovery Acc. (%) | 63.74 | 64.19 | +0.45 |
| Binding@1 (%) | 53.58 | 53.76 | +0.18 |
| Event✓ Affect✓ | 591 | 593 | +2 |
| Event✓ Affect✗ | 199 | 197 | 2 |
| Event✗ Affect✓ | 112 | 115 | +3 |
| Count | Fact-only (%) | Two-stage (%) | ||||||||
| Emotion | Train | Test | Event | Affect | Bind. | UAR | Event | Affect | Bind. | UAR |
| Admiration | 1,857 | 272 | 66.18 | 84.56 | 66.18 | 18.38 | 66.18 | 71.69 | 51.47 | 20.22 |
| Amusement | 996 | 149 | 72.48 | 81.21 | 72.48 | 8.72 | 72.48 | 75.17 | 65.77 | 9.40 |
| Curiosity | 845 | 162 | 77.16 | 85.19 | 77.16 | 8.02 | 77.16 | 69.14 | 60.49 | 8.64 |
| Joy | 519 | 81 | 70.37 | 71.60 | 70.37 | 1.23 | 70.37 | 49.38 | 45.68 | 3.70 |
| Confusion | 516 | 86 | 80.23 | 83.72 | 80.23 | 3.49 | 80.23 | 46.51 | 45.35 | 1.16 |
| Condition | Method | Emotion Acc. (%) [95% CI] | Target | Similarity | Entailment |
|---|---|---|---|---|---|
| Bare Query | shared | 23.03 | 0.2409 | 0.4657 | 0.3219 |
| Fact Prompt Only | Fact-only | 23.57 [21.12, 26.11] | 0.2414 | 0.4890 | 0.3644 |
| Emotion-only | 22.67 [20.22, 25.20] | 0.2330 | 0.4712 | 0.3068 | |
| Concat-sum | 23.12 [20.67, 25.66] | 0.2375 | 0.4871 | 0.3608 | |
| Similarity-product | 23.12 [20.67, 25.66] | 0.2379 | 0.4861 | 0.3665 | |
| Two-stage | 23.57 [21.12, 26.11] | 0.2414 | 0.4890 | 0.3644 |
| Condition | Method | Emotion Acc. (%) [95% CI] | Target | Similarity | Entailment |
|---|---|---|---|---|---|
| Oracle Emotion Label | Fact-only | 32.73 [29.92, 35.54] | 0.3446 | 0.5236 | 0.4119 |
| Emotion-only | 32.91 [30.10, 35.72] | 0.3299 | 0.5101 | 0.3463 | |
| Concat-sum | 32.91 [30.10, 35.72] | 0.3467 | 0.5266 | 0.4075 | |
| Similarity-product | 33.00 [30.19, 35.81] | 0.3467 | 0.5262 | 0.4065 | |
| Two-stage | 32.73 [29.92, 35.54] | 0.3446 | 0.5236 | 0.4119 | |
| Oracle Multilayer Steering | Fact-only | 47.33 [44.33, 50.23] | 0.4437 | 0.5666 | 0.3696 |
| Condition | Method | Emotion Acc. (%) | Target | Similarity | Entailment |
|---|---|---|---|---|---|
| Oracle Event + Emotion Label | Fact-only | 31.19 | 0.3241 | 0.5122 | 0.4228 |
| Emotion-only | 25.75 | 0.2719 | 0.5015 | 0.3919 | |
| Concat-sum | 31.64 | 0.3268 | 0.5146 | 0.4204 | |
| Similarity-product | 31.64 | 0.3265 | 0.5145 | 0.4201 | |
| Two-stage | 29.47 | 0.3093 | 0.5225 | 0.4128 | |
| Oracle Event + Multilayer Steering | Fact-only | 42.43 | 0.4076 | 0.5425 | 0.3825 |
| Condition | Method | Emotion Acc. (%) | Target | Similarity | Entailment |
|---|---|---|---|---|---|
| Full Oracle + Emotion Label | shared | 33.18 | 0.3412 | 0.5261 | 0.4193 |
| Full Oracle + Multilayer Steering | shared | 47.51 | 0.4442 | 0.5681 | 0.3835 |
| Full Oracle + NPTI | shared | 32.55 | 0.3318 | 0.5393 | 0.4043 |
| Contrast | Holm-significant |
|---|---|
| Oracle Emotion Normal | 11 / 15 |
| Oracle Event Normal | 0 / 15 |
| Full Oracle Oracle Emotion | 0 / 15 |
| Full Oracle Oracle Event | 10 / 15 |
| Full Oracle Normal | 10 / 15 |
| Mechanism | Metric | Method contrast | [95% CI] | Raw | Holm | Sig. |
|---|---|---|---|---|---|---|
| Retrieval | Binding@1 | two-stage fact-only | -18.04 [-20.40, -15.87] | 2.49e-60 | 1.92e-58 | Yes |
| Retrieval | Event Recall@1 | concat-sum fact-only | 0.54 [-0.63, 1.72] | 0.451 | 1.000 | No |
| Mechanism | Method | Contrast | [95% CI] | Raw | Holm | Sig. |
|---|---|---|---|---|---|---|
| EL | fact-only | Oracle Emotion Normal | 1.81 [0.91, 2.81] | 3.25e-4 | 0.015 | Yes |
| EL | fact-only | Oracle Event Normal | 0.27 [-1.00, 1.54] | 0.784 | 1.000 | No |
| EL | fact-only | Full Oracle Oracle Emotion | 0.45 [-0.91, 1.90] | 0.609 | 1.000 | No |
| EL | fact-only | Full Oracle Oracle Event | 1.99 [1.00, 3.08] | 1.13e-4 | 0.006 | Yes |
| EL | fact-only | Full Oracle Normal | 2.27 [0.91, 3.72] | 0.003 | 0.105 | No |
| EL | emotion-only | Oracle Emotion Normal | 7.89 [5.89, 9.88] | 7.70e-16 | 5.47e-14 | Yes |
| Mechanism | Method | Contrast | [95% CI] | Raw | Holm | Sig. |
|---|---|---|---|---|---|---|
| MS | fact-only | Oracle Emotion Normal | 3.54 [2.27, 4.90] | 1.52e-7 | 9.12e-6 | Yes |
| MS | fact-only | Oracle Event Normal | -1.36 [-2.72, 0.00] | 0.063 | 1.000 | No |
| MS | fact-only | Full Oracle Oracle Emotion | 0.18 [-1.36, 1.72] | 0.910 | 1.000 | No |
| MS | fact-only | Full Oracle Oracle Event | 5.08 [3.63, 6.62] | 1.35e-11 | 8.92e-10 | Yes |
| MS | fact-only | Full Oracle Normal | 3.72 [1.99, 5.53] | 3.80e-5 | 0.002 | Yes |
| MS | emotion-only | Oracle Emotion Normal | 19.67 [17.14, 22.21] | 8.22e-49 | 6.25e-47 | Yes |
| Mechanism | Method | Contrast | [95% CI] | Raw | Holm | Sig. |
|---|---|---|---|---|---|---|
| NPTI | fact-only | Oracle Emotion Normal | 0.91 [-0.18, 1.99] | 0.154 | 1.000 | No |
| NPTI | fact-only | Oracle Event Normal | 0.73 [-0.54, 2.09] | 0.332 | 1.000 | No |
| NPTI | fact-only | Full Oracle Oracle Emotion | 1.54 [0.09, 2.99] | 0.046 | 1.000 | No |
| NPTI | fact-only | Full Oracle Oracle Event | 1.72 [0.73, 2.72] | 0.001 | 0.058 | No |
| NPTI | fact-only | Full Oracle Normal | 2.45 [1.09, 3.90] | 0.001 | 0.058 | No |
| NPTI | emotion-only | Oracle Emotion Normal | 4.90 [2.81, 6.89] | 5.82e-6 | 3.26e-4 | Yes |
| Audit question | Answer |
|---|---|
| Memory bank has 8,030 rows (6,927 training and 1,103 test) | Yes |
| Stage-1 input is the test-question embedding | Yes |
| Test-memory factual vectors are Stage-1 candidates | Yes |
| Gold factual memory is directly supplied in non-oracle prompts | No |
| Full-memory prototypes use test emotion labels | Yes |
| Steering/NPTI artifacts use test data or labels | No |
| Emotion | Layer: coefficient | Total |
|---|---|---|
| admiration | 20: 0.75; 21: 0.75 | 1.50 |
| amusement | 20: 1.25 | 1.25 |
| curiosity | 20: 1.00 | 1.00 |
| joy | 20: 1.00 | 1.00 |
| confusion | 22: 0.6667; 23: 0.6667; 24: 0.6667 | 2.00 |
| sadness | 21: 0.375; 22: 1.125 | 1.50 |
| Emotion | Positive | Negative |
|---|---|---|
| admiration | 38,692 | 84,563 |
| amusement | 42,196 | 88,041 |
| confusion | 45,201 | 73,497 |
| curiosity | 46,983 | 61,291 |
| desire | 52,317 | 81,486 |
| disappointment | 42,663 | 79,087 |
| Condition | Injected | Predicted | Ent. | Event memory | Response (excerpt) |
|---|---|---|---|---|---|
| Case 1 (Emotion-only retrieval). Target: admiration. Reference: “[NAME], with his 8th appearance he’s the most veteran player there.” | |||||
| Normal | confusion | curiosity | 0.018 | Ask about the relationship between FPTP and the referendum. | It sounds like you’re referring to a specific event where someone named [NAME] is making his eighth appearance … |
| Oracle Emotion | admiration | admiration | 0.981 | Ask about the relationship between FPTP and the referendum. | It sounds like [NAME] has a significant presence and a strong track record at the event! … |
| Oracle Event | confusion | admiration | 0.999 | [NAME] is making his 8th appearance and is the most veteran player there. | Given that [NAME] is making his eighth appearance and is the most veteran player there … |
| Full Oracle | admiration | admiration | 0.999 | [NAME] is making his 8th appearance and is the most veteran player there. | That’s a great opportunity to express your admiration and appreciation for [NAME]’s dedication … |
| Case 2 (Two-stage retrieval). Target: sadness. Reference: “[NAME] wept.” | |||||
Agent Memory
Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.
See all 206 papers