Scaling Therapy Supervision: Assessing Psychotherapy Fidelity Using Large Language Models

Arvind Pillai; Alexis Roach; Jason Holden; Varsha D Badal; Subigya Nepal; Yu Yvonne Wu; Yuliang Chen; Sudarshan Regmi

cs.CLOct 20, 2026

Scaling Therapy Supervision: Assessing Psychotherapy Fidelity Using Large Language Models

Authors: Arvind Pillai, Alexis Roach, Jason Holden, Varsha D Badal, Subigya Nepal, Yu Yvonne Wu, Yuliang Chen, Sudarshan Regmi,

Organizations: Dartmouth College · University of California, San Diego · University of Virginia

Abstract

Psychotherapy fidelity assessment measures how faithfully a therapist adheres to a treatment protocol, a critical determinant of positive patient outcomes. However, fidelity assessment is bottlenecked by manual expert review, rendering the process labor-intensive and unscalable. To scale therapy supervision and support clinician self-learning, we present an integrated application of large language models (LLMs) that automates psychotherapy fidelity workflows and generates actionable supervisory feedback from authentic, long-form clinical transcripts. We evaluate three open-weight models across multiple task decomposition configurations and reasoning strategies, demonstrating that evidence-first reasoning improves fidelity estimation. Furthermore, we establish the framework's external construct validity on an independent randomized controlled trial (RCT), where it reliably distinguishes Cognitive Behavioral Therapy (CBT) from non-CBT sessions across core therapeutic skills. Through a user study that evaluates the generated feedback, mental health experts indicate that the pipeline synthesizes actionable clinical feedback. Ultimately, our work offers a practical, scalable solution toward bridging the community mental health supervision gap.

Figures & tables

Figure 1: Scaling therapy supervision. Left: Manual review of a 60-minute session is highly time-intensive, limiting scalability. Right: The proposed automated pipeline leverages LLM reasoning over de-identified transcripts to automatically generate fidelity ratings and actionable feedback.

Decomposition

Metric

Qwen3.5-9B

Gemma4-E4B-it

Gemma4-26B-A4B-it

Appendix

Category	Item	Description
a) Agenda	1	Therapist noted client’s current emotional state regarding agenda setting.
	2	Therapist and client established agenda for session.
	3	Priorities for agenda items were established.
	4	Agenda was appropriate for time allotment (neither too ambitious nor too limited).
	5	Client discussed salient events or problems occurring since last session.
	6	The agenda was adhered to during the session.

Appendix

Table 2: CTS-Psy Section I: General Items

Category	Item	Description
f) Guided	1	Used questions to determine meaning client attached to an event.
Discovery	2	Used questions to show incongruities without demeaning the person.
	3	Used questions to help client explore various facets of a problem.
	4	Used questions to examine client’s arbitrary conclusions or assumptions.
	5	Used questions to elicit alternative ways of solving a problem.
	6	Used questions to consider alternative explanations.

Appendix

Table 3: CTS-Psy Section II: Specific Items. Note that we do not use the overall score category j).

Study	#Dialogues	#Turns (Form)	Type	#Ratings
Cactus ( Lee et al., 2024 )	31577	16.6 (Short)	Synthetic	0 (unrated)
TherapyGym ( Huang et al., 2026 )	116	10 (Short)	Synthetic	1276
Ours	30 (420 full)	$\sim$ 180 $\pm$ 81 (Long)	Real	1800

Appendix

Table 4: Comparison of prior evaluations and data for dialogue-based therapy.

Decomposition	Qwen3.5-9B			Gemma4-E4B-it			Gemma4-26B-A4B-it
	RZ	EV	RF	RZ	EV	RF	RZ	EV	RF
Category	0.31	0.46	0.26	0.26	0.29	0.15	0.17	0.44	0.35
Clinical	0.32	0.33	0.32	0.32	0.37	0.36	0.20	0.47	0.42
Section	0.32	0.28	0.36	0.36	0.36	0.36	0.14	0.33	0.43
Obs	0.36	0.31	0.30	0.30	0.38	0.28	0.09	0.45	0.47
Full	0.26	0.26	0.36	0.36	0.33	0.33	0.16	0.44	0.50

Appendix

Table 5: Comparing LLM fidelity estimation performance vs. human performance (T1). Performance reports Cohen’s

\kappa

. Human performance upper bound is

\kappa=0.71

. Reasoning strategies evaluated include Rationalization (RZ), Evidential Estimation (EV), and Reflection (RF).

Category	BA	MCC
Agenda	0.68 [0.61–0.75]	0.39 [0.24–0.54]
Feedback	0.54 [0.47–0.62]	0.08 [-0.06–0.23]
Understanding	0.63 [0.48–0.78]	0.26 [-0.04–0.54]
Interpersonal	0.78 [0.44–0.96]	0.22 [-0.04–0.44]
Collaboration	0.53 [0.48–0.59]	0.11 [-0.06–0.28]
Discovery	0.55 [0.46–0.63]	0.08 [-0.06–0.21]

Appendix

Table 6: Qwen3.5-9B with evidential reasoning and category level decomposition performance across categories.

Figure 6: Prediction Powered Inference of overall session fidelity prevalence using Gemma4-E4B-it (T2). Red crosses denote the biased, uncorrected raw population averages (

{\mu}_{\text{unlabeled}}

) produced by each model configuration. The colored dots and corresponding horizontal brackets represent the unbiased PPI rectified means (

\hat{\mu}_{\text{PPI}}

) and their 95% confidence intervals. Grey dashed lines illustrate the direction and magnitude of the statistical rectifier, defined as

\Delta=\hat{\mu}_{\text{PPI}}-{\mu}_{\text{unlabeled}}

Figure 7: Prediction Powered Inference of overall session fidelity prevalence using Gemma4-26B-A4B-it (T2). Red crosses denote the biased, uncorrected raw population averages (

{\mu}_{\text{unlabeled}}

) produced by each model configuration. The colored dots and corresponding horizontal brackets represent the unbiased PPI rectified means (

\hat{\mu}_{\text{PPI}}

) and their 95% confidence intervals. Grey dashed lines illustrate the direction and magnitude of the statistical rectifier, defined as

\Delta=\hat{\mu}_{\text{PPI}}-{\mu}_{\text{unlabeled}}

#	Role	Degree	SMI	CBT	Cog.
1	Therapist	Master’s	Extremely Familiar	Extremely Familiar	Extremely Familiar
2	Clinical Psychologist	Doctorate	Extremely Familiar	Extremely Familiar	Extremely Familiar
3	Therapist	Doctorate	Very Familiar	Extremely Familiar	Extremely Familiar

Appendix

Table 7: Expertise information of mental health experts.

Panel of Figure 8 — Figure 8: Post-Hoc Estimation (Direct-then-Rationalize). The LLM immediately assigns a binary score $y_{i}$ per item, then provides a brief rationale as post-hoc evidence. The user turn injects the transcript $\langle$ T $\rangle$ , the decomposition-specific instruction (Table 8 ), and the CTS-Psy item set $Q$ .

Panel of Figure 9 — Figure 9: Evidential Estimation (Extract-Assess-Predict). A grounded three-step chain: the model first extracts verbatim relevant_spans from the transcript, then assesses them against the criterion’s operational definition, and only then assigns the binary score $y_{i}$ . This enforces an evidence-first reasoning order.

Panel of Figure 10 — Figure 10: Self-Reflection (Critic-in-the-Loop). After an initial Evidential Estimation pass (Step 1), a separate reflection LLM instantiated as an expert clinical supervisor critiques each rating across rubric adherence, evidence coverage, and confidence calibration (Step 2). The consolidated critique is fed back to the primary generator, which then revises its binary decisions and supporting evidence (Step 3). This loop may be applied for $k\geq 1$ iterations.

Decomposition	Partition	$\|Q\|$	Task Instruction
Full	—	60	Rate the transcript based on all 60 CTS-Psy items below, organized by section:
Observability	Explicit	24	Rate the explicitly observable items below (Agenda, Feedback, Homework, Social):
Observability	Implicit	36	Rate the implicitly observable items below (Understanding, Interpersonal Effectiveness, Collaboration, Discovery, Cognition, Intervention):
Section	General	30	Rate the 30 General CTS-Psy items below (Agenda, Feedback, Understanding, Interpersonal Effectiveness, Collaboration):
Section	Specific	30	Rate the 30 Specific CTS-Psy items below (Discovery, Cognition, Intervention, Homework, Social Skills Training):
Clinical	Relationship	18	Rate the Therapeutic Relationship items below (Understanding, Interpersonal Effectiveness, Collaboration):

Appendix

Table 8: Decomposition-specific task instructions injected into the user turn of each prompt (Figures 8 – 10 ). Category-level decomposition issues 10 independent inference passes, one per CTS-Psy category (Agenda, Feedback, Understanding, Interpersonal Effectiveness, Collaboration, Discovery, Cognition, Intervention, Homework, Social).

Panel of Figure 11 — Figure 11: Feedback Generation. Given the three lowest-scoring CBT competency categories from the fidelity-estimation stage (§4.2), the LLM produces one positive observation and one actionable, session-specific suggestion per weak competency area. The user turn injects $\langle$ bottom-3 fidelity evidence $\rangle$ : the per-item fidelity statement, score, relevant transcript spans, and criterion assessment for each of the 18 items spanning the three weakest categories. In the RAG condition, retrieved therapy-manual passages relevant to this evidence are additionally prepended to the user turn before generation.

Panel of Figure 12 — Figure 12: LLM-as-a-Judge (Paired Comparison). For each dimension (Actionability, Clarity), a judge LLM independently compares RAG-grounded and non-RAG feedback generated for the same session (Figure 11 ), presented order-randomized as Feedback A/B, and returns a single comparative score $\in\{-1,0,+1\}$ . Each dimension is scored by a separate model call so that no single judge call sees more than one dimension at once.

Clinical Language Model Evaluation

Clinical language model evaluation measures whether AI systems are accurate, safe, fair, and useful across medical tasks and care settings.

See all 419 papers

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
Feb 23, 2026Fangrui Huang, Souhad Chbeir, Arpandeep Khatua +8Mental Health Conversational Agents Clinical Language Model Evaluation
Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA · The University of Hong Kong, Hong Kong +2
MyMentorLLM: A psychotherapy GenAI environment with multimodal voice/text patients, trainees and experts for deliberate practice
Jul 28, 2026Rodolfo Rizzi, Alessandro Grecucci, Massimo StellaMental Health Counseling Multi-Agent LLM Systems
CogNosco Lab, Department of Psychology and Cognitive Science, University of Trento, Italy. · Department of Education, Psychology and Communication Sciences, University of Bari, Italy.
Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health Conversation
Apr 7, 2026Abdullah Mazhar, Het Riteshkumar Shah, Aseem Srivastava +2Clinical Language Model Evaluation LLM Evaluation
IIIT Delhi · MBZUAI · Wysa

Feb 23, 2026J/K move · Enter open · S save

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang+7

Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA · The University of Hong Kong, Hong Kong+2

Mental Health Conversational Agents Clinical Language Model Evaluation Conversational Agents LLM Safety Benchmarks Cognitive Behavioral Therapy

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for mental-health support; yet prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy. We introduce THERAPYGYM, a framework that evaluates and improves therapy chatbots along two clinical pillars: fidelity and safety. Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks (e.g., failing to address harm or abuse). To mitigate bias and unreliability in LLM-based judges, we further release THERAPYJUDGEBENCH, a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians. THERAPYGYM also serves as a training harness: CTRS and safety-based rewards drive RL with configurable patient simulations spanning diverse symptom profiles. Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). Our work enables scalable development of therapy chatbots that are faithful to evidence-based practice and safer in high-stakes use.

Abstract

Clinical Language Model Evaluation

Explore similar work

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

MyMentorLLM: A psychotherapy GenAI environment with multimodal voice/text patients, trainees and experts for deliberate practice

Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health Conversation