Organizations: Centre for Addiction Medicine, Department of Psychiatry, National Institute of Mental Health and Neuro Sciences (NIMHANS), Bengaluru, India. · Department of Computer Science and Engineering, Indian Institute of Technology (IIT) Kharagpur, India. · Department of Psychiatry, National Institute of Mental Health and Neuro Sciences (NIMHANS), Bengaluru, India.
Motivational interviewing (MI) promotes behavioural change in substance use disorders. Its fidelity is measured using the Motivational Interviewing Treatment Integrity (MITI) framework. While large language models (LLMs) can potentially generate MI-consistent therapist responses, their competence using MITI is not well-researched, especially in real world clinical transcripts. We aim to benchmark MI competence of proprietary and open-source models compared to human therapists in real-world transcripts and assess distinguishability from human therapists. Methods: We shortlisted 3 proprietary and 7 open-source LLMs from LMArena, evaluated performance using MITI 4.2 framework on two datasets (96 handcrafted model transcripts, 34 real-world clinical transcripts). We generated parallel LLM-therapist utterances iteratively for each transcript while keeping client responses static, and ranked performance using a composite ranking system with MITI components and verbosity. We conducted a distinguishability experiment with two independent psychiatrists to identify human-vs-LLM responses. Results: All 10 tested LLMs had fair (MITI global scores >3.5) to good (MITI global scores >4) competence across MITI measures, and three best-performing models (gemma-3-27b-it, gemini-2.5-pro, grok-3) were tested on real-world transcripts. All showed good competence, with LLMs outperforming human-expert in Complex Reflection percentage (39% vs 96%) and Reflection-Question ratio (1.2 vs >2.8). In the distinguishability experiment, psychiatrists identified LLM responses with only 56% accuracy, with d-prime: 0.17 and 0.25 for gemini-2.5-pro and gemma-3-27b-it respectively. Conclusion: LLMs can achieve good MI proficiency in real-world clinical transcripts using MITI framework. These findings suggest that even open-source LLMs are viable candidates for expanding MI counselling sessions in low-resource settings.
Large language models (LLMs) can generate fluent dialogue, but prior works lack situational grounding, dynamic strategy control, and evaluation aligned with clinical standards in motivational interviewing (MI). We introduce StoryMI, a multi-LLM agent framework for controllable MI dialogue generation, where questionnaire-based client profiles are expanded into situational stories that provide narrative context for the dialogue. Therapist and client agents generate MI-coded utterances guided by MI codes selected by the interaction agent, while an interaction agent dynamically coordinates exchanges to control MI strategies during a multi-turn conversation. We propose a two-level evaluation protocol: lexical metrics and MI-specific measures of macro-level counseling strategies, alongside LLM-as-judge and human expert assessments. We construct a dataset of 6K simulated MI dialogues grounded in 1K questionnaire-story pairs, covering 12 MI codes and 13 symptom domains, and benchmark six open- and closed-source LLMs. Our results show that situational grounding and macro-level control can improve MI adherence and clinical plausibility, demonstrating the effectiveness of a structured multi-agent workflow for psychotherapy dialogue generation. We provide code and data for reproducibility.
Qingyu Meng, Min Chen, Dingming Liu +5
Vrije Universiteit Amsterdam · NII, Tokyo Institute of Technology
Reasoning large language models (LLMs) have recently made much progress in complex problem-solving, leveraging internal reasoning (or thought) to guide their solution generation. However, existing LLM-based counseling agents, including those using Motivational Interviewing (MI), generate responses without explicitly aligning thoughts with counseling techniques, limiting their effectiveness. We propose MIThinker, a lightweight thinking model that generates therapeutic thoughts to guide MI counseling agents in strategy selection and response generation. To overcome the lack of annotated thought data, we introduce AugR1-MI, an automated pipeline that reverse-engineers counselor's thoughts from observed responses. Through two-stage training combining supervised fine-tuning and reinforcement learning, MIThinker demonstrates improved theory-of-mind assessment and strategy alignment. Comprehensive evaluations show that MindfulMI, our agent leveraging MIThinker, achieves MI competency comparable to state-of-the-art systems with an order of magnitude less computation.
Yizhe Yang, Palakorn Achananuparp, Heyan Huang +2
Beijing Institute of Technology · Singapore Management University · Australian National University
Conversational dynamics describe how rapport, emotion, and goals change over a dialogue. These changes are important in therapeutic conversations, where they can influence when and how therapists intervene. However, large language model (LLM)-based synthetic therapeutic dialogue generators rarely track how these properties evolve together, which may limit the quality of longer conversations. We introduce CALM-IT, a dual-agent framework that tracks client states and therapist-side inferences to guide strategy selection and utterance generation. We evaluate synthetic Motivational Interviewing (MI) dialogues using automated Motivational Interviewing Treatment Integrity (MITI) ratings, conversational metrics, and comparisons across dialogue lengths, with ratings from four licensed psychologists providing support for key findings. Across 8,232 dialogues covering 686 grounded scenarios, four frameworks, and three target lengths, CALM-IT achieves the highest mean scores on all four MITI 4.2 global ratings and shows the smallest aggregate change in quality between short- and long-form conditions. CALM-IT also achieves the highest simulated client redirection acceptance rate (64.3%), with its strongest redirections followed by increased change talk and reduced sustain talk. Ablation experiments show that removing conversational dynamics components reduces dialogue quality. Together, these results suggest that explicitly modeling conversational dynamics can improve the quality and consistency of long-form synthetic MI dialogues.
Viet Cuong Nguyen, Nhi Yen Nguyen, Kristin A. Candan +5
Georgia Institute of Technology · Northwell Health · Columbia University