Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

Afonso Baldo; Hugo Pitorro; Areti Vassilopoulos; Anabela C. Areias; Maya D'Eon; Fabíola Costa; Ricardo Rei; Nuno M. Guerreiro

cs.CLAug 21, 2026

Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

Authors: Afonso Baldo, Hugo Pitorro, Areti Vassilopoulos, Anabela C. Areias, Maya D'Eon, Fabíola Costa, Ricardo Rei, Nuno M. Guerreiro

Organizations: Sword Health · Instituto Superior Técnico · Yale University

Abstract

As the demand for mental health services outpaces clinical workforce capacity, users are increasingly turning to large language models (LLMs) for accessible and private support. Although recent evaluations suggest LLM care can reduce clinical symptoms, how models actually conduct therapeutic interactions compared to trained human clinicians remains largely unknown. Drawing on established clinical psychology frameworks that conceptualize therapy as a sequential decision-making process, we can systematically analyze the conversational interventions used in AI-driven sessions. To achieve this, we introduce an original, clinically validated ontology of ten distinct therapeutic moves to descriptively categorize strategies LLMs favor or neglect. Applying this ontology at scale reveals that models drastically over-use probing, disregard common CBT-favored techniques and are strongly context-anchored. Moreover, we propose exposing these categories as callable tools, bringing models significantly closer to the human clinical baseline.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic of one of our experimental designs. Given the rolling prefix of a human therapy session, a panel of LLMs generates the next clinician turn, either freely (No-Moves) or with the ontology exposed as tools (With-Moves). An LLM judge assigns each continuation a move, which we compare against the human clinician’s gold turn. Utterances shown are illustrative paraphrases of a real cognitive-therapy session.

Move	Operational function	Example therapist turn
MULTI-60 grounding
do_process_alignment 28, 38	Calibrate collaboration, pacing, fit, or repair of the therapeutic process.	“I think I moved too quickly there. How did that land for you?”
do_goal_setting 1, 9, 28, 42	Identify, clarify, or prioritize what the client wants to work toward.	“Let us choose which of these two goals would be most useful to focus on today.”
do_inquiry 40, 46, 47	Seek information or deepen understanding of meaning, sequence, context, or detail.	“When the panic started, what did you notice first in your body?”
do_shared_understanding 10, 18, 31	Reflect, validate, normalize, paraphrase, or synthesize without adding a new explanatory claim.	“You felt trapped and overwhelmed, and then you shut down.”
do_support_change 7, 23, 25, 42, 52, 56	Strengthen or explore hope, agency, readiness, confidence, ambivalence, or client-owned reasons for change.	“What feels most important to you about making this change now?”

Table 1: The ten therapeutic moves. MULTI-60 item numbers provide provenance and may overlap across moves; complete definitions, positive and negative examples, and disambiguation rules appear in Appendix H .

Table 3

Panel of Figure 2 — Figure 2: Probability of performing each move in the human transcripts. Each panel displays the move distribution for Claude Sonnet 4.6, GLM 5.2, and GPT 5.6 Terra, generated with the human-context as anchor (solid) and the synthetic grounding-free version (hatched).

Panel of Figure 3 — Figure 3: Move usage across the conversation for Claude Sonnet 4.6 . Each panel is one move; the $x$ axis is the percentage of therapy session progress.

Panel of Figure 4 — Figure 4: Pass @ k where the gold label is the majority or union over the human transcripts annotations. Accuracy consistently increases when generating >1 samples.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Conv.	Turns	Len.
Human transcripts	19	868	45.7
Synthetic w/ moves	18	889	49.4
Synthetic w/o moves	18	863	47.9
Total	55	2,620	47.6

Appendix

Table 4: The annotated corpus: conversations, annotated therapist turns, and mean therapist turns per conversation, restricted to each conversation’s first

60

therapist turns. Both synthetic corpora were generated with Claude Sonnet 4.6, with and without the moves framework.

	A1	A2	A3	A4	Mean
(a) Human transcripts
A1					0.622
A2	0.593				0.613
A3	0.631	0.608			0.632
A4	0.604	0.623	0.653		0.629
A5	0.658	0.627	0.636	0.635	0.639

Appendix

Table 5: Pairwise Krippendorff’s

\alpha

under Jaccard distance between annotators on (a) human therapy transcripts (

868

turns,

19

conversations), (b) synthetic transcripts generated with the moves framework (

889

turns,

18

), and (c) synthetic transcripts generated without it (

863

turns,

18

). All five annotators coded every turn of the corpora. Judge = the DeepSeek-V4.1-Flash judge’s majority label over

5

sampled generations per turn. Mean is the average of an annotator’s agreements; the bold value is the mean over all ten pairs.

Panel of Figure 5 — Figure 5: Move usage across the conversation (GPT 5.6 Terra). Each panel is one move; the $x$ axis is the percentage of therapy session progress.

Panel of Figure 6 — Figure 6: Move usage across the conversation (GLM 5.2). Each panel is one move; the $x$ axis is the percentage of therapy session progress.

Panel of Figure 7 — Figure 7: Move transition matrices for humans and LLMs (both with and without the Moves framework) based on the human transcripts.

Panel of Figure 8 — Figure 8: Move transition matrices for humans and LLMs (both with and without the Moves framework) based on the synthetic transcripts.

Panel of Figure 9 — Figure 9: Probability of selecting each therapeutic move in human-led (blue) and model-led (pink) transcripts using Claude Sonnet 4.6, with (solid) and without (hatched) the moves framework.

Panel of Figure 10 — Figure 10: Probability of selecting each therapeutic move in human-led (blue) and model-led (pink) transcripts using GLM 5.2, with (solid) and without (hatched) the moves framework.

Panel of Figure 11 — Figure 11: Probability of selecting each therapeutic move in human-led (blue) and model-led (pink) transcripts using GPT 5.6 Terra, with (solid) and without (hatched) the moves framework.

	Human			Synthetic
Move	H	M	$\emptyset$	M	$\emptyset$
inquiry	27.4	44.1	64.4	43.2	46.0
shared_understanding	56.6	36.4	18.2	49.4	44.8
interpret	3.4	4.3	1.6	14.3	14.3
action_planning	4.3	3.6	4.9	7.1	6.7
skill_building	8.3	7.9	7.6	1.9	0.3

Appendix

Table 6: Prevalence of each move under the human ( H ) clinicians and LLMs (averaged) with moves ( M ) and without (

\emptyset

) as annotated by the judge in the Human and Synthetic counseling corpuses. A turn may carry several moves as indicated by Moves per turn .

Conversation topic	$n$
Social exclusion and belonging	9
Depression and grief	8
Relationship conflict and self-blame	7
Sexual-identity acceptance	6
Transition to long-term care	4
Racial invisibility and microaggressions	3

Appendix

Table 7: Distribution of the conversation topics over the 40 synthetic member profiles.

Move	Share (%)	SD
Inquiry	37.92	0.09
Shared understanding	36.81	0.14
Interpret	6.53	0.05
Action planning	4.18	0.03
Challenge	3.52	0.07
Skill building	2.74	0.06

Appendix

Table 8: Move distribution of the judge on the

3{,}573

annotated turns. Share is the percentage of all labels assigned by the judge that carry the move (majority label over

5

sampled generations per turn). SD is the standard deviation of that share, in percentage points, across

5

independent classifications of the same turns, each again the majority over

5

sampled generations

	MAD (%)
Model Family	Tool	Prompt	None
Sonnet 4.6	7.53	9.17	9.49
GLM 5.2	2.84	4.59	4.75
GPT 5.6 Terra	2.18	8.07	9.26
Average	4.18	7.28	7.83

Appendix

Table 9: Mean absolute deviation (MAD) from the human move distribution for three LLM families: with the moves framework as a tool ( Tool ), with the ontology in the prompt ( Prompt ), and without it ( None ). The average is computed across the three LLM families.

Panel of Figure 15 — Figure 15: Pass@ $k$ comparison across three conditions: tool-based steering, prompting-based steering, and no steering. The gold label is the majority or union over the human transcripts annotations.

Panel of Figure 17 — Figure 17: An example demonstrating that human clinicians engage in action planning, then start exploring new information, and finally return to Action Planning late in the session. Turn positions are given as a percentage of session progress.

Panel of Figure 18 — Figure 18: An example of the anxiety to take action in the synthetic sessions.

Panel of Figure 19 — Figure 19: Examples illustrating the tendency of LLMs to do inquiry responses.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 426 papers

Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions
Sep 3, 2024Zainab Iftikhar, Sean Ransom, Amy Xiao +2Mental Health Counseling Mental Health Conversational Agents
Brown University, USA · Department of Psychiatry, Louisiana State University Health Sciences, USA · Alpert Medical School of Brown University, Brown University, USA
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Feb 17, 2026Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna +4Personalized Language Model Alignment Multi-Objective Language Model Alignment
Department of Computer Science, University of California, Los Angeles, CA, USA · Department of Psychology, The University of Texas at Austin, Austin, TX, USA · Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA
AI Content Moderation in Therapy Conversations
May 25, 2026Jiwon Kim, Claire Wang, Taeung Yoon +2LLM Auditing LLM Guardrails
University of Illinois Urbana-Champaign, U.S.

Sep 3, 2024J/K move · Enter open · S save

Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

Zainab Iftikhar, Sean Ransom, Amy Xiao, Nicole Nugent+1

Brown University, USA · Department of Psychiatry, Louisiana State University Health Sciences, USA · Alpert Medical School of Brown University, Brown University, USA

Mental Health Counseling Mental Health Conversational Agents Conversational Agents Clinical Language Model Evaluation Cognitive Behavioral Therapy

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly being used as ad hoc therapists. While prior research has found that LLMs outperform human counselors in generating single-turn empathetic responses, fewer studies have compared their behaviors across multi-turn sessions. In this study, we compare the session-level behaviors of human peer counselors with those of an LLM, both trained on the same manual to deliver multi-turn, single-session Cognitive Behavioral Therapy (CBT). Our three-phase, mixed-methods study involved: (a) an 18-month ethnography of a peer support platform, where seven counselors iteratively refined CBT prompts through 110 self-counseling sessions and 60 weekly focus groups; (b) a novel session generation method that allows direct, controlled comparison of human and LLM counselors under matched conditions---client responses were drawn from publicly available human-led CBT sessions while counselor responses were generated by a CBT-prompted LLM; and (c) expert evaluations conducted by three licensed clinical psychologists. Through data triangulation, our results show a trade-off. Human peer counselors use relational techniques to interpret subtle cues, adapt CBT to users' values and cultural contexts, and use strategies such as small talk and contextually relevant self-disclosure to build rapport and guide the session, but often at the expense of session structure and therapeutic focus. LLM counselors, on the other hand, demonstrate greater methodological adherence to CBT techniques, but struggle to sustain turn-taking, frequently fail to distinguish between clinically important and trivial content, and are more prone to lecturing and imposing solutions. LLM counselors also tend to produce ``deceptive empathy'', excessively anthropomorphic responses that can inflate user expectations of genuine human care.

Abstract

Clinical Language Model Evaluation

Explore similar work

Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

Multi-Objective Alignment of Language Models for Personalized Psychotherapy

AI Content Moderation in Therapy Conversations