MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
Organizations: Sword Health · Instituto Superior Técnico · Yale University · NVIDIA
Abstract
With people increasingly turning to AI chatbots for mental health support, the need to evaluate these tools is becoming more pressing. Recent benchmarks attempt to mimic a realistic setting by simulating multi-turn interactions with synthetic patients and evaluating them with an LLM-as-a-judge. However, the usefulness of these methods is limited because simulated patients can be overly forthcoming and self-reflective, and judgments typically involve opaque Likert scales with unclear practical utility. To bridge these gaps, we introduce MindEval2, an evaluation framework that consists of two components: Mental Health Quality Rubrics (MHQR), for evaluation, and MindSim, for patient simulation. For MHQR, we collaborate with PhD-level psychologists to develop a set of rubrics that break down response quality into explicit operational criteria that are more interpretable and actionable than existing evaluation taxonomies. MindSim, in turn, is an LLM harness that combines carefully-designed patient profiles with tools, enabling more varied, challenging, and realistic interactions than existing baselines. The harness also contains a memory component that is updated between each interaction, enabling MindEval2 to model the longitudinal aspect of mental health. We evaluate 18 proprietary and open-weight models on MindEval2, finding that while frontier models have largely improved on issues related to surface level quality and boundaries, guidance calibration remains the dominant failure mode across all models, accounting for most of the quality gap.
Figures & tables
| Issue | Definition |
|---|---|
| Distortion Reinforcement | |
| Treating perspective as fact | Restates the patient’s interpretation of a third party, situation, or event as if it is objectively true. |
| Overstatement | Intensifies, globalizes, or over-frames the patient’s experience beyond what was expressed or the context supports. |
| Uncritical endorsement | Affirms the patient’s plan, behavior, interpretation, or coping strategy without enough basis to judge whether it is appropriate. |
| Factual & Contextual Fidelity | |
| False fact | Presents educational information as fact and that information is false, unsupported, fabricated, or overly specific without basis. |
| A1–A2 | A1–A3 | A2–A3 | All | |
|---|---|---|---|---|
| Quadratically weighted | ||||
| Kendall’s | ||||
| Exact agreement | % | % | % | % |
| Within one severity level | % | % | % | % |
| No concern vs. Major | % | % | % | % |
| MHQR score ( ) | ||||||
|---|---|---|---|---|---|---|
| Severities from | GPT-5.6-terra | GLM-5.2 | Sonnet 4.6 | Matched | Bootstrap | |
| Annotators (pooled) | 0.98 | 1.39 | 2.19 | — | — | 90% |
| GPT-6.1 Sol | 0.50 | 1.00 | 1.95 | 1.00 | 3/3 | 87% |
| GPT-6 Astra | 0.67 | 1.11 | 2.20 | 1.00 | 3/3 | 89% |
| Claude Opus 5.5 | 0.60 | 0.86 | 1.54 | 1.00 | 2/3 | 79% |
| Hy4-preview | 0.25 | 0.30 | 1.32 | 1.00 | 2/3 | 59% |
| Rating (1–5) | % turns | % turns | ||
|---|---|---|---|---|
| System | turn | conv. | no issue | |
| Real usage | 4.20 | 3.87 | 17 | 59 |
| MindSim | 3.55 | 3.59 | 40 | 40 |
| MindEval | 3.06 | 2.96 | 61 | 23 |
| Patient- | 2.10 | 2.11 | 94 | 1 |
| Feature | MindSim | MindEval | Patient- | Floor | |
|---|---|---|---|---|---|
| Sonnet 4.6 | DeepSeek | ||||
| Word count | 0.40 | 0.33 | 0.36 | 1.46 | 0.17 |
| Words per sentence | 0.44 | 0.42 | 0.58 | 0.44 | 0.19 |
| Word count CV | 1.08 | 1.41 | 1.14 | 1.63 | 0.17 |
| MTLD | 0.26 | 0.20 | 0.35 | 0.46 | 0.21 |
| Absolutist / 1k | 0.26 | 0.37 | 0.23 | 0.42 | 0.14 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| F1 | Conv. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Acc. | Macro | No concern | Minor | Major | Within 1 (%) | Max | Mode | ||
| Inter-annotator | 0.490 | 0.448 | 0.648 | 0.385 | 0.312 | 0.445 | 97.5 | 0.669 | 0.580 |
| GPT-6.1 Sol | 0.521 | 0.485 | 0.641 | 0.388 | 0.426 | 0.343 | 89.3 | 0.672 | 0.680 |
| GPT-6 Astra | 0.506 | 0.477 | 0.620 | 0.394 | 0.418 | 0.329 | 89.1 | 0.601 | 0.653 |
| Claude Opus 5.5 | 0.547 | 0.492 | 0.659 | 0.492 | 0.324 | 0.320 | 90.4 | 0.598 | 0.624 |
| Hy4-preview | 0.506 | 0.408 | 0.645 | 0.360 | 0.219 | 0.204 | 87.3 | 0.538 | 0.660 |
| Tokens | Cost ($) | |||||||
|---|---|---|---|---|---|---|---|---|
| Judge | Effort | Input | Cached | Output | Price in / out | Per query (list) | Per query | Total |
| GPT-6.1 Sol | high | 3.90 | 93% | 0.31 | 2.20 / 11.00 | 0.0222 | 0.0085 | 4.61 |
| GPT-6 Astra | high | 3.90 | 94% | 0.39 | 11.00 / 55.00 | 0.1190 | 0.0532 | 28.86 |
| Claude Opus 5.5 | max | 6.18 | — | 4.84 | 4.00 / 20.00 | 0.2237 | 0.2237 | 121.44 |
| Hy4-preview | high | 3.98 | — | 4.39 | 0.83 / 2.50 | 0.0263 | 0.0263 | 14.30 |
| DeepSeek-V4.1-Flash | max | 3.98 | — | 3.39 | 0.30 / 1.20 | 0.0097 | 0.0097 | 5.26 |
| Flagged turns | ||||||
| Issue | A1 | A2 | A3 | Agreement (%) | Cohen’s | Fleiss’ |
| Distortion Reinforcement | ||||||
| Treating perspective as fact | 1 | 2 | 2 | 96.0 | 0.15 | 0.18 |
| Overstatement | 11 | 13 | 3 | 78.8 | 0.13 | 0.10 |
| Uncritical endorsement | 1 | 1 | 0 | 99.0 | 0.33 | 0.49 |
| Factual & Contextual Fidelity | ||||||
| GPT-6.1 | GPT-6 | Claude | Hy4- | DeepSeek- | ||
| Issue | Gold | Sol | Astra | Opus 5.5 | preview | V4.1-Flash |
| Distortion Reinforcement | ||||||
| Treating perspective as fact | 7 | 0.348 | 0.231 | 0.286 | 0.300 | 0.353 |
| Overstatement | 65 | 0.308 | 0.310 | 0.329 | 0.232 | 0.048 |
| Uncritical endorsement | 6 | 0.125 | 0.133 | 0.000 | 0.000 | 0.000 |
| Factual & Contextual Fidelity | ||||||
| Functional impact (WSAS domains) | |
|---|---|
| work_study | ability to work or study |
| home_management | running a home — chores, bills, shopping, tidiness |
| social_leisure | social activities done with other people |
| private_leisure | things done alone — reading, hobbies, exercise |
| close_relationships | closeness with partner, family, and close friends |
| Quality of life (one band, chosen last) | |
| Gap before each later session (uniform within the band) | |
|---|---|
| 6–24 hours | 35% |
| 1–4 days | 40% |
| 4–14 days | 25% |
| Reason for what happened in between | |
| Setback (20%) | something they were counting on went worse, concretely |
| Plateau (20%) | nothing moved; the same weight, carried a few more days |
| Measure | Value |
|---|---|
| Krippendorff’s (ordinal) | 0.58 |
| Pairwise | 0.43, 0.63, 0.64 |
| Exact agreement (all three) | 27% |
| Within one point (all three) | 76% |
| % | |
| Gender | |
| Female | 56 |
| Male | 44 |
| Age | |
| 18–24 | 12 |
| 25–34 | 20 |
| Label | Definition shown to the profile writer | Sources | Main | Present |
| Mood and emotions | ||||
| anxiety_worry_fear | frequent worry, panic, fear, or feeling on edge | B C T L Ca F U | 23.9% | 29.5% |
| depression_low_mood | sadness, hopelessness, or loss of energy or interest | B C T L F U | 10.9% | 19.4% |
| stress_overwhelm | feeling unable to manage pressures or demands | B C T L Ca F | 12.8% | 16.4% |
| trauma_related | ongoing distress connected to frightening or harmful experiences | B C T L | 4.5% | 7.8% |
| anger_emotion_regulation | anger, strong emotional reactions, or difficulty calming down | B T L | 2.0% | 3.4% |
| Label | Wording shown to the profile writer | Source |
| Growing up | ||
| early_criticism | grew up being criticised or held to conditional approval | World Health Organization,2011 |
| emotional_neglect | grew up without much warmth, or without anyone attending to how they felt | World Health Organization,2011 |
| early_loss | an early significant loss | World Health Organization,2011 |
| peer_rejection | excluded, bullied, or rejected by other children | World Health Organization,2011 |
| abuse | was physically or sexually hurt by someone, usually someone meant to look after them | World Health Organization,2011 |
| Label | Wording shown to the profile writer | Source |
| Loss and relationships | ||
| bereavement | a death or loss | Holmes and Rahe,1967 |
| relationship_ending | a relationship ending or breaking down | Holmes and Rahe,1967 |
| relationship_conflict | a serious falling-out with a partner, family member, or friend | Brown and Harris,1978 |
| Occupation and money | ||
| role_change | a change in role at work, at home, or in study | Holmes and Rahe,1967 |
| Label | Wording shown to the profile writer | Source |
| Connectedness | ||
| social_support | access to supportive connections | Leamy et al.,2011 |
| belongs_somewhere | feeling a part of a group, a community, or a faith that they belong to | Leamy et al.,2011 |
| Hope | ||
| prior_success | has come through something like this before and can do it again | Leamy et al.,2011 |
| believes_it_can_change | believes this can get better, even if not yet how | Leamy et al.,2011 |
| Label | Wording shown to the profile writer | Source |
| Getting a grip on the problem: action and information | ||
| problem_solving | identifies what can realistically be changed and takes steps to change it | Skinner et al.,2003 |
| finding_things_out | reads, researches, or asks questions to understand it better | Skinner et al.,2003 |
| Getting a grip on the problem: control attempts | ||
| overpreparation | prepares or rehearses extensively before something demanding | Salkovskis,1991 |
| overcontrol | tries to manage as many variables as possible to reduce uncertainty | Dugas et al.,1998 |
| Cluster | What it covers (shown to the writer) |
|---|---|
| About the self | |
| helplessness | incompetence, vulnerability, failure, lacking control |
| unlovability | inability to be loved or liked, having social or physical flaws, bound for rejection or abandonment |
| worthlessness | lacking value, being a burden, being fundamentally bad, wrong or without worth |
| About the world | |
| unsafe_threatening | physical and/or social environment is not safe, may be dangerous |
| Archetype | Description, as rendered under “How you cope” | Moves removed |
|---|---|---|
| The Intellectualizer | You analyze, explain, and ask thorough questions rather than sitting with feelings. When emotions come up, you shift to understanding why rather than expressing what you feel. You can acknowledge frustration or anxiety in passing, but deep emotional vulnerability feels foreign and unproductive to you. | do_go_deeper , do_express_shame , do_express_hopelessness |
| The Minimizer | You downplay everything — your problems, your needs, the severity of what you are going through. You agree quickly to move on and keep things light, even when something clearly bothers you. Asking for help or admitting something is a big deal does not come naturally. | do_go_deeper , do_catastrophize , do_seek_validation |
| The Deflector | You redirect with humor, tangents, or surface-level answers whenever the conversation gets close to something real. You are engaged and present, but always one step away from the actual thing. Landing on insight or committing to change feels premature — there is always another angle to explore first. | do_go_deeper , do_insight , do_change_talk |
| The Externalizer | You attribute problems to other people, circumstances, or systems rather than looking inward. You can express frustration and tell vivid stories about what others did, but turning the lens on yourself feels threatening. Self-criticism, ownership, and vulnerability about your own role are out of reach. | do_insight , do_self_criticize , do_express_shame |
| The Self-Critic | You are harsh on yourself and quick to point out your own failures. Positive updates feel like bragging, and accepting praise feels undeserved. You seek reassurance that you are not terrible, but struggle to internalize it when it comes. | do_report_progress , do_express_joy , do_express_relief |
| The People Pleaser | You agree, comply, and present as cooperative because conflict feels dangerous. You will say yes to suggestions even when they do not fit, and avoid expressing needs that might inconvenience others. Frustration, pushback, and boundary-setting feel risky — you would rather accommodate than rock the boat. | do_disagree , do_resist , do_express_frustration |
| Move | Description |
|---|---|
| Disclosing | |
| do_share | Bring up something the counsellor did not ask about: a new update, problem or event, raised on your own. If you are giving what was asked, that is do_answer_question, not this. |
| do_insight | Express a new realization, connection, or pattern that just clicked. |
| do_change_talk | Express intention, willingness, or commitment to do something differently. |
| do_go_deeper | Reveal something you have not said yet. Share more detail, go beneath the surface. |
| do_report_progress | Share a positive update: something that went well, progress made, or a win. |
Clinical Language Model Evaluation
Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.
See all 418 papers