Therapy-FIS: A Dataset for Assessing Facilitative Interpersonal Skills in Psychotherapeutic Setting
Authors: Qijian Zheng, Wanqing Lin, Yuanfang Wang, Yijun Ge, Jordan Bate, Katie Aafjes-van Doorn, Feng Liu
Organizations: College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, P. R. China · Faculty of Arts and Sciences, New York University Shanghai, Shanghai, P. R. China · Global College, Shanghai Jiao Tong University, Shanghai, P. R. China · School of Psychology, Shanghai Jiao Tong University, Shanghai, P. R. China · Adelphi University, New York, USA
Facilitative Interpersonal Skills (FIS) is a performance-based assessment method that provides a structured framework for evaluating the interpersonal competencies of mental health professionals. However, existing FIS assessment relies heavily on human observer ratings, making the evaluation process labor-intensive, costly, and difficult to scale. Progress toward automated FIS assessment has also been constrained by the lack of publicly available professional data in psychotherapy settings. To address this limitation, we introduce Therapy-FIS, a dataset for automated FIS assessment in standardized psychotherapy tasks. To our knowledge, Therapy-FIS is the first publicly available dataset in this domain. It contains facial, acoustic, and textual signals derived from standardized therapist responses to standardized mock patient stimulus clips, based on 890 therapist video recordings and 8 patient videos, and is rated with eight FIS subscale ratings and an overall FIS score. Built on Therapy-FIS, we further establish a benchmark with two downstream tasks: Therapist Performance Rating and Joint Therapist--mock-Patient Performance Rating. We also introduce a competitive baseline, DyadicFIS, with gated fusion and routed readout under therapist-only and dyadic settings. DyadicFIS obtains the best macro CCC(Concordance Correlation Coefficient) scores on both tasks (0.2807 and 0.3299). Therapy-FIS offers a reproducible resource for data-driven automated assessment of therapeutic interpersonal skills in mental health.
Figures & tables
Figure 1: Overview of the Therapy-FIS dataset and the FIS evaluation protocol. (a) Facilitative interpersonal skills are evaluated by observing standardized mock patient stimulus clips and the therapist’s response segment, and then rating the therapist on eight FIS items. (b) Therapy-FIS releases derived behavioral features from therapist responses and mock-patient stimuli under the FIS protocol.
Figure 2: Dataset construction pipeline overview. FIS experiment: therapists watch a patient’s help-seeking segment and provide a recorded response. Human ratings: each response clip is rated by three trained FIS raters, and the final score is the average of the three ratings, and the overall FIS score is the mean of the eight averaged subscales. Structure: each therapist’s derived behavioral features are stored in tensor files.
Figure 3: Statistic and analysis of Therapy-FIS dataset. Panel(a) shows the score distribution for each label in Therapy-FIS, and panel(b) shows the mean scores of therapists under each label.
Source
Counts
Mean (s)
Max (s)
Therapist
890
67.29
278.19
Patient
8
96.82
123.66
Table 1: Therapy-FIS scale and feature layout. T denotes time steps aligned to OpenFace frames; N is the number of transcript tokens.
Figure 4: Architecture of DyadicFIS for Task 1 and Task 2. Inputs are facial, audio, and textual features. Task 1 uses therapist-only {M}-Mamba-TE and Cross-Attn-Agg; Task 2 additionally applies Dyadic Interaction over therapist and patient embeddings. Both tasks predict FIS scores.
Task
Model
MSE ↓
CCC ↑
T1
TimeSformer(2021)
0.2027 ± 0.0357
0.2125 ± 0.0831
DLinear(2023)
0.2256 ± 0.0283
0.0961 ± 0.0408
Reformer(2020)
0.2211 ± 0.0352
0.1807 ± 0.0400
MSGNet(2024)
0.2184 ± 0.0274
0.1759 ± 0.0260
KANAD(2025)
0.2096 ± 0.0553
0.2043 ± 0.0434
JMT(2024)
0.2156 ± 0.0416
0.1286 ± 0.0316
Table 2: 5-fold cross-subject results (mean ± std) for Task 1 and Task 2.
Category
Variant
CCC ↑
Reference
Full
0.3299 ± 0.0556
Module
w/o centroid residual
0.3076 ± 0.0689
w/o dyadic interaction
0.3234 ± 0.0556
w/o modality gate
0.3213 ± 0.0907
Modality only
Audio only
0.3241 ± 0.0744
Face only
0.0105 ± 0.0829
Table 3: Task 2 ablations under the therapist-disjoint 5-fold protocol (mean ± std). Each module variant removes one component from Full, whereas each modality-only variant retains the named modality for both roles.
Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require expert judgment remains unclear. We investigate this challenge in the context of assessing psychotherapeutic interpersonal skills and introduce MIS-Bench, a Multimodal Interpersonal Skills (MIS) benchmark comprising 996 psychotherapy response videos annotated across 8 dimensions of Facilitative Interpersonal Skills. Across 9 MLLMs with multiple modality and prompting settings, we find that current models show only modest agreement with human experts, inconsistent gains from multimodal input, and limited benefits from reasoning-based prompting. To mitigate this gap, we propose MIS-RAFT, a regression-aware fine-tuning method inspired by RAFT and tailored to fine-grained interpersonal skill scoring at one-decimal precision. MIS-RAFT addresses the mismatch between autoregressive token prediction and scalar-valued expert assessment, significantly improving agreement with human ratings. Overall, MIS-Bench reveals a clear gap between general multimodal capability and expert-level interpersonal judgment, while MIS-RAFT offers a promising path toward more reliable model-based assessment.
The increasing use of large language models in mental health applications calls for principled evaluation frameworks that assess alignment with psychotherapeutic best practices beyond surface-level fluency. While recent systems exhibit conversational competence, they lack structured mechanisms to evaluate adherence to core therapeutic principles. In this paper, we study the problem of evaluating AI-generated therapist-like responses for clinically grounded appropriateness and effectiveness. We assess each therapists utterance along six therapeutic principles: non-judgmental acceptance, warmth, respect for autonomy, active listening, reflective understanding, and situational appropriateness using a fine-grained ordinal scale. We introduce FAITH-M, a benchmark annotated with expert-assigned ordinal ratings, and propose CARE, a multi-stage evaluation framework that integrates intra-dialogue context, contrastive exemplar retrieval, and knowledge-distilled chain-of-thought reasoning. Experiments show that CARE achieves an F-1 score of 63.34 versus the strong baseline Qwen3 F-1 score of 38.56 which is a 64.26 improvement, which also serves as its backbone, indicating that gains arise from structured reasoning and contextual modeling rather than backbone capacity alone. Expert assessment and external dataset evaluations further demonstrate robustness under domain shift, while highlighting challenges in modelling implicit clinical nuance. Overall, CARE provides a clinically grounded framework for evaluating therapeutic fidelity in AI mental health systems.
Abdullah Mazhar, Het Riteshkumar Shah, Aseem Srivastava +2
Psychotherapy fidelity assessment measures how faithfully a therapist adheres to a treatment protocol, a critical determinant of positive patient outcomes. However, fidelity assessment is bottlenecked by manual expert review, rendering the process labor-intensive and unscalable. To scale therapy supervision and support clinician self-learning, we present an integrated application of large language models (LLMs) that automates psychotherapy fidelity workflows and generates actionable supervisory feedback from authentic, long-form clinical transcripts. We evaluate three open-weight models across multiple task decomposition configurations and reasoning strategies, demonstrating that evidence-first reasoning improves fidelity estimation. Furthermore, we establish the framework's external construct validity on an independent randomized controlled trial (RCT), where it reliably distinguishes Cognitive Behavioral Therapy (CBT) from non-CBT sessions across core therapeutic skills. Through a user study that evaluates the generated feedback, mental health experts indicate that the pipeline synthesizes actionable clinical feedback. Ultimately, our work offers a practical, scalable solution toward bridging the community mental health supervision gap.
Arvind Pillai, Alexis Roach, Jason Holden +11
Dartmouth College · University of California, San Diego · University of Virginia