The Right Source Is Not Enough: Understanding History Interference in Speech-LLM Memory
Organizations: Beijing Institute of Technology · Northeastern University, China · Institute of Science Tokyo · Ant Group ·
Abstract
Speech large language models retain audio history to support multi-turn conversations. Yet remembering an event does not ensure that its information is used only when relevant. We investigate why irrelevant historical audio changes answers even when the queried event remains represented. Controlled experiments on MOSS-Audio and MiMo-Audio show that historical values direct errors toward specific competing answers, beyond a generic effect of context length. Causal interventions distinguish two paths. In open-ended question answering, most history-value errors disappear when subsequent computation cannot directly access the irrelevant source. In multiple-choice answering, many errors persist: early historical influence is carried by question-candidate states even after direct source access is blocked. Low-dimensional interventions further separate source-role information from answer-value information. Their effects peak at different depths, and changing the source-role signal does not eliminate the causal influence of competing values. On naturally occurring errors, strengthening the queried role and counteracting the historical value each improve answers; their joint intervention gives the greatest recovery. These findings support a failure of downstream evidence use beyond source identification alone: source-role information can influence a decision without exclusively controlling which historical values affect it. Reliable conversational memory therefore requires not only retention and retrieval, but also control over how retained information enters answer computation.
Figures & tables
| Model | Recordings and current question | Target only | Full history |
|---|---|---|---|
| MOSS | History: When asked about the backpack, Camille Russell said its color is orange. Target: When asked about the suitcase, Camille Russell said its color is purple. Question: What color is Camille Russell’s suitcase? | A: purple | C: orange |
| MiMo | History: To keep it safe, Abigail Thomas put the boarding pass inside the locker. Target: To keep it safe, Abigail Thomas put the passport inside the box. Question: Where does Abigail Thomas keep the passport? | A: box | B: locker |
| Both | History: Amelia Campbell’s backpack has a white color. Target: Amelia Campbell’s suitcase has a green color. Question: What color is Amelia Campbell’s suitcase? | A: green | A: green |
| Open-ended | Multiple-choice | |||
|---|---|---|---|---|
| Condition | MOSS | MiMo | MOSS | MiMo |
| Full history ( ) | 151 | 22 | 175 | 94 |
| Strict late block ( ) | 2 | 3 | 124 | 49 |
| Replace with early-blocked state ( ) | — | — | 6 | 1 |
| Restore full-access state ( ) | — | — | 125 | 49 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Set | Dev / test | Main use |
|---|---|---|
| Relation-content | 128 / 512 | Behavior by historical relation; natural errors and correct controls; population and module tests. |
| Reading-structure | 128 / 512 | Question-role transfer; strict source block and bidirectional state bridge; matched answer formats. |
| Coherent role reversal | 128 / 512 | Same history queried about old versus current facts; all-layer role–value factorial scan. |
| Intervention | MOSS | MiMo |
|---|---|---|
| Role enhancement | 43.0 [34.8, 51.6] | 62.2 [45.9, 77.0] |
| Counterfactual value shift | 68.4 [60.7, 76.2] | 67.6 [54.1, 81.1] |
| Both | 79.1 [72.1, 85.7] | 83.8 [73.0, 93.2] |
| Case | Model | Options in recorded order | History answer |
|---|---|---|---|
| Backpack (error) | MOSS | A: purple; B: black; C: orange; D: red | C: orange |
| Boarding pass (error) | MiMo | A: box; B: locker; C: suitcase; D: drawer | B: locker |
| Backpack (control) | Both | A: green; B: blue; C: white; D: yellow | C: white |
| Model | Input | Neutral | Competing | Change | 95% CI |
|---|---|---|---|---|---|
| MOSS | Audio | 84.23 | 78.73 | [ ] | |
| MOSS | Text | 98.10 | 96.57 | [ ] | |
| MiMo | Audio | 48.23 | 39.67 | [ ] | |
| MiMo | Text | 70.43 | 52.50 | [ ] |
Conversational Memory
Conversational memory equips AI assistants to retain, update, and retrieve useful information from past interactions for coherent, personalized dialogue.
See all 77 papers