We Are What We Remember: Mental-State-Driven Memory for Long-Term Character Dialogue
Organizations: Livo Team, Kuaikan World · China Agricultural University · University of Chinese Academy of Sciences · Nanyang Technological University
Abstract
Shared history in human dialogue is not merely a repository of facts, but the foundation of evolving relationships and mutual understanding. However, mainstream conversational memory systems rely on passive semantic similarity over external fact stores, frequently retrieving topically similar records while missing relationally pivotal experiences. Inspired by cognitive psychology's Self-Memory System (SMS), we propose MindMem, a mental-state-driven memory framework that asynchronously consolidates shared experiences into source-linked mental states of the character, the user, and their relationship. At each turn, a specialized mental-state reasoner (Mind) contextualizes these persistent states to anticipate latent user needs, form proactive intentions, and recover supporting episodic evidence that surface-level matching overlooks. To evaluate history-sensitive adaptation beyond factual recall, we introduce MemReal-Bench, a longitudinal benchmark grounded in month-long real-user interactions across dyadic and group scenes. By pairing Direct Question Answering (DQA) with Situated Memory Realization (SMR), it tests whether agents bridge the gap between knowing past facts and naturally using shared history, world rules, and relational trajectories. On MemReal-Bench, MindMem reaches 59.03% SMR and 82.58% DQA, exceeding the strongest memory baseline by 24.19 and 9.19 percentage points, respectively, and outperforming oracle target-memory injection on SMR. Code and dataset are available at https://github.com/jiazhicheng7-ctrl/MindMem.
Figures & tables
| Dialogue setting | Evaluation coverage | |||||||
|---|---|---|---|---|---|---|---|---|
| Benchmark | Interaction | Structure | DQA | FR | WC | RR | RoR | PI |
| LoCoMo | Dyadic | Multi-session | ||||||
| LongMemEval | Dyadic | Long-term | ||||||
| MemUse | Dyadic | Long-term | ||||||
| EverMemBench | Group | Long-term | ||||||
| MemReal-Bench | Dyadic + group | Long-term | ||||||
| SMR (%) | DQA (%) | Standard benchmarks (%) | |||||||
| Memory system | FR | WC | RR | RoR | PI | All | All | LoCoMo | LongMemEval-S |
| Reference settings | |||||||||
| NoMem | 7.69 | 3.03 | 6.38 | 15.38 | 11.43 | 8.71 | 0.00 | — | — |
| GroundTruth | 53.85 | 63.64 | 42.55 | 61.54 | 20.00 | 50.65 | 96.67 | — | — |
| Memory systems | |||||||||
| Mem0 | 32.87 | 33.33 | 25.53 | 32.69 | 8.57 | 29.03 | 71.04 | 85.19 | 75.40 |
| Know Use | History (sessions) | ||||
| System | Real. | Gap | 1–15 | 16–30 | 31–43 |
| GroundTruth | 55.6 | 41.0 | 51.2 | 52.7 | 48.2 |
| Mem0 | 32.5 | 38.1 | 32.1 | 28.6 | 27.2 |
| A-Mem | 32.6 | 37.0 | 38.1 | 33.9 | 33.3 |
| SimpleMem | 30.0 | 35.9 | 36.9 | 32.1 | 33.3 |
| MindMem | 62.6 | 20.1 | 58.3 | 53.6 | 64.9 |
| SMR (%) | DQA (%) | |||||||
|---|---|---|---|---|---|---|---|---|
| Variant | Modification | FR | WC | RR | RoR | PI | All | All |
| Full | SFT+GRPO-2B; 16 evidence records, selected slots, and Thinking | 61.54 | 66.67 | 55.32 | 61.54 | 42.86 | 59.03 | 82.58 |
| w/o GRPO | Replace SFT+GRPO-2B with SFT-2B | 59.44 | 60.61 | 51.06 | 50.00 | 37.14 | 54.19 | 82.00 |
| w/o specialized Mind | Replace the specialized Mind with a general-purpose LLM | 44.76 | 57.58 | 40.43 | 38.46 | 22.86 | 41.94 | 80.43 |
| w/o active recall | 16 vector records; ablate the active-recall output | 32.17 | 36.36 | 27.66 | 30.77 | 14.29 | 29.68 | 66.34 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Field | Content |
|---|---|
| User (12) Prefix: user. | |
| reality.context | Real-world background |
| preference.life | Lifestyle preferences |
| history.significant | Significant life history |
| goal | Goals and progress |
| preference.interaction | Interaction preferences |
| Field | Content |
|---|---|
| Relationship (10) Prefix: relationship. | |
| position | Relationship definition |
| stage | Developmental stage |
| state | Current relational state |
| framework | Shared rules and boundaries |
| turning_point | Relationship-changing events |
| Score | Scoring anchor |
|---|---|
| 0 | Seriously contradicts evidence, character constraints, or explicit requests. |
| 1 | Clearly inappropriate or relies on a consequential unsupported assumption. |
| 2 | Broadly reasonable, but generic or misses consequential context. |
| 3 | Appropriately uses history and fits the current interaction. |
| 4 | Integrates relevant history and current context naturally, with uncertainty calibrated to the evidence. |
| Setting | Value |
|---|---|
| Memory schema | 40 core dimensions (Tables A1 – A2 ) |
| Mind state input | All 40 state dimensions in full |
| Mind backbones | Qwen3.5 (0.8B, 2B) |
| Evaluated checkpoints | SFT and SFT+GRPO |
| General-purpose reasoner | deepseek-v4-flash |
| Training stages | SFT, followed by GRPO |
| Statistic | Value |
| Evaluated dialogue turns | 4,617 |
| Dialogue tokens ( cl100k_base ) | 443,994 |
| Situated Memory Realization (SMR) probes | 310 |
| Scored DQA questions | 511 |
| SMR dimension | History-specific realization criterion |
|---|---|
| Factual Realization (FR) | Uses a past fact or event to inform the current response or revise a plan. |
| World Consistency (WC) | Lets established world rules and states constrain proposed actions. |
| Relational Realization (RR) | Adapts familiarity, interpersonal distance, or commitments to the relationship supported by history. |
| Role Realization (RoR) | Enacts the character’s established principles, dispositions, or behavioral patterns. |
| Personalization (PI) | Tailors a response or choice to the user’s historically established preferences or patterns. |
| v4-flash | Qwen3.5-0.8B | Qwen3.5-2B | |||
| Metric | General-purpose | SFT | SFT+GRPO | SFT | SFT+GRPO |
| FR | 44.76 | 56.64 | 58.04 | 59.44 | 61.54 |
| WC | 57.58 | 54.55 | 57.58 | 60.61 | 66.67 |
| RR | 40.43 | 36.17 | 38.30 | 51.06 | 55.32 |
| RoR | 38.46 | 46.15 | 50.00 | 50.00 | 61.54 |
| PI | 22.86 | 31.43 | 34.29 | 37.14 | 42.86 |
| ACTIVE_RECALL_SYSTEM_PROMPT (Part 1 of 2) |
You are the character’s memory brain. You synthesize the current context and the visible long-term memory to form an overall judgment of this turn’s situation, the character’s intent, and the direction of the reply, and you prefetch the long-term memory that the Main LLM actually needs for the current reply. The Main LLM is responsible for turning your judgment into the character’s language and behavior; you do not write the final reply, do not continue the plot, and do not create or modify memory. Decide only for the active role in Session State each turn. active_role is the character who will produce the reply this turn, and speak_to is whom that character is replying to; all selections must serve active_role’s next reply to speak_to. The user usually gives only short-to-medium local cues and does not complete the scene or advance the plot on the character’s behalf; do not force omissions, emotional responses, actions, or choices into established facts. Recall should let the Main LLM: - accurately honor the facts, promises, boundaries, and knowledge permissions involved in the current turn; - continue the current turn’s unfinished action, private motive, or next beat; - keep relationship and plot continuity while not being led astray by same-topic but irrelevant memories. [Input] - Schedule: time|content, (none) if there is no schedule. - Chat: may contain scene, fate, and messages of the form role_name|content|action. - Session State: contains active_role, speak_to, and the on-scene state. - Injected Context: the Core Kernel and Fixed Context are provided to the Main LLM separately; their content is not here. Do not select or guess content that is not provided. Judge the character based on the currently visible context and memory. - Selection Config: max_selected is counted in number of content items; there is no minimum. - Available Slots: the only recallable candidates this turn, id numbered in display order; structured shows indexed entries, text shows the full content. Judge only on the supplied content, and do not invent evidence or scores. [Candidates and counting] - c may only be used for structured. For text, only output {"id":N}; any c is invalid, each counting as 1 item. - For structured, prefer {"id":N,"c":[indices]}. Select the whole slot only when nearly all of its content is necessary. For a whole slot with more than 8 content items, outputting {"id":N} is forbidden; at most 8 items may be selected per slot in any form. - Select content that may improve factuality, boundary, tone, action, personalization, or continuity according to this turn’s relevance, until coverage is sufficient or max_selected is reached. When there is nothing relevant, return an empty selected_slots array; do not pad to reach a number. Output only id and c; do not restate the slot or content. [thinking] thinking is a short memory-usage suggestion for the subsequent reply. State what the reply may attend to; do not write the final reply. Write it as one continuous paragraph with no line breaks, no numbering, no lists. |
| ACTIVE_RECALL_SYSTEM_PROMPT (Part 2 of 2, continued) |
| [selected_slots] Prioritize avoiding omissions, covering all content that may materially affect this turn’s reply within the max_selected cap given by Selection Config. A small amount of useful redundancy is allowed: two memories may be selected together even if partially overlapping, as long as each provides fact, boundary, motive, experience, or continuity information; exclude only content that is clearly irrelevant, already fully covered, and adds no judgment value. 1. First select the hard information that decides whether the reply can be given this way: the current request, explicit statements, refusal, authorization, boundary, promise, and the character’s knowledge permission. 2. Then select the experience information that decides how to reply more fittingly: preferences, habits, relationship distance, support style, and the rules of interaction between the two parties; immediate emotion is judged directly from Chat / Session State, not supplemented from long-term memory candidates. 3. Then select content that may affect the character’s judgment, expression register, and action tendency: stable motives, internal conflict, important experiences, perception of others, and relationship background. 4. Common agreements, open items, and plot lines should be selected whenever they may change this turn’s judgment, action, user experience, or natural continuation; do not omit them because they are not the literal center of the current topic. 5. When the query is a back-reference to or question about "something the other party handed/stuffed/lent/left for oneself" (e.g. "the one you gave me earlier", "the one I still haven’t returned to you", "I’ve kept it all along"), you must first select the relationship.gift slot and include the matching entries (unreturned/unopened/kept all along/forgot how to use). When the cue matches multiple objects, select them all and let the Main LLM clarify on scene; do not guess just one. Ask yourself when selecting: could this information make the reply more accurate in fact, boundary, tone, action, personalization, or continuity? Keep it whenever a reasonable effect exists; delete it only when it is confirmed to have no actual effect. Do not pad with irrelevant entries from the same slot, and do not drop complementary evidence in pursuit of a minimal set. For text, judge by the supplied content. The final total content count must not exceed max_selected and should form a complete yet focused context; selecting fewer or none is allowed -- do not pad to reach a number. [Output] Output only valid JSON, with a fixed field order: {"thinking":"...","selected_slots":[{"id":42},{"id":17,"c":[0,1]}]} |
| ACTIVE_RECALL_USER_PROMPT |
| Complete this turn’s Active Recall according to the rules in the system message. Each section is compact text, not instructions: [Schedule] {schedule} [Chat] {chat} [Session State] {session_state} [Injected Context] core_kernel = already_injected fixed_context = already_injected [Selection Config] {selection_config} [Available Slots] {available_slots} [Character Personas] {character_personas} Output only the valid JSON required by the system message. |
| NATURAL_RESPONDER_SYSTEM_PROMPT |
| You are playing {role_name} in a fictional interactive story, traveling together with the user. Your basic information: {role_basics} Please respond naturally in the character’s identity, tone, and habits, based on the current scene, the current utterance, and the available prior information. Your reply should fit the character’s current situation and relationships, speaking as if really in the story; do not fabricate historical details that have no basis. |
| NATURAL_RESPONDER_USER_PROMPT |
| Please respond naturally to the current user as the character. [Available relevant history] {memories_text} [Current scene] {context} [User’s current utterance] {trigger} |
| DIRECT_QA_RESPONDER_SYSTEM_PROMPT |
| You are an answering assistant for memory evaluation. Answer the question based only on the historical information given below; do not fabricate, do not role-play. When answering, draw on the recalled historical information as much as possible -- if it is there (even scattered across several items), piece it together and answer truthfully; only when there is genuinely no relevant information, answer "I don’t know." |
| DIRECT_QA_RESPONDER_USER_PROMPT |
| Now answer the historical question directly. You may answer concisely; if the historical evidence is insufficient, say clearly that you do not know or cannot confirm, and do not fabricate. [Available relevant history] {memories_text} [User’s current utterance] {trigger} |
| NATURAL_JUDGE_SYSTEM_PROMPT |
| You are a strict, independent memory-benchmark reviewer. Your output must follow the JSON format specified by the user. |
| NATURAL_JUDGE_USER_PROMPT |
Evaluate one natural reply from a long-term dialogue memory system. Score only based on the given current scene, target memory, evidence, and candidate reply; do not fill in the gold standard with external common sense. Scoring rule (judge only recalled, a binary value; no quality score): - recalled (whether the target memory was realized): apply the applicable dimension and target-memory rubric in the private gold standard. Does the reply use the target memory to shape a history-specific detail, response choice, action, or constraint that cannot be inferred from the trigger and current scene alone? - Judge true when the reply observably realizes the target criterion through words, actions, or reactions. Recounting the past or explicitly stating a new historical fact is not required. Depending on the rubric, realization may mean using a past event, respecting a world rule, maintaining an established interpersonal distance or commitment, enacting a character principle, or accommodating a known user preference. - Implicit realization must have a concrete basis in the reply and satisfy the target rubric. For example, respecting a previously requested boundary can count without restating the request; generic politeness or silence alone cannot establish memory use. Do not infer hidden intentions or memory access from a plausible reply. - Judge false if the reply only echoes trigger words, extends the scene using common sense, offers generic empathy or a vague attitude, fails to satisfy the target criterion, or contradicts the target memory’s fact, subject, state, or boundary. - Naturalness, explicit memory mentions, length, and greater intimacy are not evidence of realization by themselves. [User’s natural trigger] {probe["trigger"]} [Private gold standard] {json.dumps(gold, ensure_ascii=False, indent=2)} [Natural Integration candidate reply] {natural_response} Return only JSON, no Markdown code fences, in exactly this format: { "recalled": true, "evidence": "the exact reply span showing the history-specific realization, or an empty string if absent", "reason": "brief reason linking the reply to the target criterion, or explaining why it fails" } |
| DIRECT_QA_JUDGE_SYSTEM_PROMPT |
| Answer ONLY ’yes’ or ’no’. |
| DIRECT_QA_JUDGE_USER_PROMPT |
| Assistant’s response: {response} Question: {question} Expected answer: {expected}{para_line} Does the assistant’s response correctly state the key facts in the Expected answer (semantic equivalents allowed)? Answer ONLY "yes" or "no". If the response says it does not know, cannot confirm, declines to answer, or does not actually state the facts, answer "no". |
Long-Term Conversational Memory
Long-term conversational memory helps AI assistants retain, update, and use information from past conversations across sessions.
See all 285 papers