cs.CYNov 5, 2026

Remembering Is Not Enough: Evaluating Timing and Omission in Longitudinal AI Mental Health Support

Authors: Eli Min, Mohit Chandra, Aamogh Sawant, Mengting Tu, Freya Lin, Ridwan Shams Khan, Reigna El-Yashruti, Christopher L. Smith, 

Organizations: Georgia Institute of Technology, USA · University of Virginia, USA · Beth Israel Deaconess Medical Center, USA

Abstract

Evaluations of large language models (LLMs) for mental health support commonly focus on the clinical appropriateness of individual responses, potentially overlooking failures that emerge across extended interactions. As AI assistants accumulate memory of users' circumstances and prior disclosures, this raises an accountability question: Do AI chatbots provide appropriate support as users' needs evolve? We investigate by analyzing 14,256 longitudinal simulations grounded in donated chatbot histories from 22 users, examining how a memory-equipped assistant responds to suicidal ideation, self-harm, and anxiety crises across recovery trajectories and communication styles. We introduce the Context-Aware and Readiness-Evaluative (CARE) framework, developed with mental health professionals, to assess the timing and quality of advice, including omissions. Advice was less aligned with users' readiness after a crisis than in matched control stretches of the same conversation, largely because of premature recommendations, which were most common when recovery stalled. Revisiting a crisis reduced premature advice but not lagging advice, and users' communication style changed how often and when the assistant gave recommendations. Although explicitly unproductive advice was rare, the assistant frequently omitted clinician-expected support such as follow-up and planning for next steps. These findings show the limits of single response-level safety evaluation and the need to assess timing, omission, and consistency across users when holding AI systems accountable for longitudinal support.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. VERA-MH: Validation of Ethical and Responsible AI in Mental Health

    May 13, 2026Luca Belli, Kate H. Bentley, Josh Gieringer +6HealthcareMental Health

  2. Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

    Feb 23, 2026Ian Steenstra, Paola Pedrelli, Weiyan Shi +2LLM Safety EvaluationClinical Language Model Evaluation

  3. MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support

    Oct 15, 2026John Mendonça, Daniel Tschernutter, José Pombal +6Clinical Language Model EvaluationMulti-Turn Dialogue Evaluation