CAIA in Practice: Field Evaluation of an AI-Assisted Support System for Text-Based Online Counselling

Philipp Steigerwald; Nico Bienlein; Jennifer Burghardt; Mara Stieler; Robert Lehmann; Jens Albrecht

cs.HCAug 23, 2026

CAIA in Practice: Field Evaluation of an AI-Assisted Support System for Text-Based Online Counselling

Authors: Philipp Steigerwald, Nico Bienlein, Jennifer Burghardt, Mara Stieler, Robert Lehmann, Jens Albrecht

Organizations: Faculty of Computer Science, Technische Hochschule Nürnberg Georg Simon Ohm, Nürnberg, Germany · Faculty of Social Sciences, Institute for E-Counselling, Technische Hochschule Nürnberg Georg Simon Ohm, Nürnberg, Germany

Abstract

Rising global demand for mental health support creates significant service delivery challenges, with asynchronous email counselling serving as a crucial low-threshold channel for accessing care. This paper presents CAIA, a co-designed AI-based tool suite that demonstrates responsible AI integration into counselling practice through seven LLM-driven functions enhanced by retrieval-augmented generation. A field evaluation involved 34 professional counsellors conducting authentic sessions with trained student counsellees (36 threads, 321 messages, 1,257 AI outputs). User behaviour analysis confirms substantial adoption, revealing that professional autonomy and information accuracy are decisive for sustained acceptance, with counsellors particularly valuing interpretive functionalities that provide new perspectives and stimulate professional reflection.

Figures & tables

Fig. 2: CAIA user interface displaying the split-screen design with email conversation thread (left) and expandable AI functionality panels (right).

Fig. 3: Counselling History functionality panel in edit mode, displaying the editing toolbar with formatting options, AI-powered text manipulation functions (replace word, rephrase sentence) and reference symbols with hover tooltips showing source passages.

Fig. 4: Boxplots of each functionality’s relative open-time per email thread (labels show total opens $n$ ).

Fig. 5: Functionality usage trends across counselling progression showing relative functionality engagement patterns with 95% confidence intervals. Each subplot displays the percentage of functionality opens at different counsellee message indices (1-6), revealing how functionality preferences evolve throughout the counselling process.

Fig. 6: Functionality transition heat-map showing counsellor navigation between CAIA components. Counts are printed in each cell, with asterisks denoting significantly over-represented transitions (* $p_{\text{adj}}{<}.05$ , ** $p_{\text{adj}}{<}.01$ , *** $p_{\text{adj}}{<}.001$ ).

Fig. 7: Distribution of functionality panel states during counsellor typing sessions showing active consultation patterns

AI Functionality	Positive	Negative
Counselling Brief	5	4
Counselling History	5	1
Current Situation Overview	3	2
Social Network Graph	1	14
Hypotheses	3	7
Methodological Suggestions	8	5

TABLE I: AI Functionality Feedback Distribution

Explore similar work

CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models
Feb 24, 2026Anqi Li, Chenxiao Wang, Yu Lu +3Mental Health Counseling
Zhejiang University · Westlake University · Department of Psychology, Jing Hengyi School of Education, Hangzhou Normal University
CounselReflect: Opportunities and Challenges for Designing Tools to Support Self-Reflection on Mental Health and Well-Being Conversations with AI
Mar 31, 2026Yahan Li, Chaohao Du, Christopher Chun Kuizon +7Mental Health Counseling Emotional Support Conversation
University of Southern California, United States · The University of Texas at Austin, Austin, Texas, United States · Department of Psychiatry and Behavioral Sciences, University of Southern California, Los Angeles, California, United States
Beyond Reflection: Affirmation as a Promising Behavioral Marker Associated with Quality in Text-Based Counseling
Aug 27, 2026Michimasa InabaMental Health Counseling Conversational Agents
The University of Electro-Communications 1-5-1, Chofugaoka, Chofu, Tokyo, Japan

Feb 24, 2026J/K move · Enter open · S save

CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models

Anqi Li, Chenxiao Wang, Yu Lu, Renjun Xu+2

Zhejiang University · Westlake University · Department of Psychology, Jing Hengyi School of Education, Hangzhou Normal University

Mental Health Counseling

Open paper AskPDF arXiv

Client perceptions of the therapeutic alliance are critical for counseling effectiveness. Accurately capturing these perceptions remains challenging, as traditional post-session questionnaires are burdensome and often delayed, while existing computational approaches produce coarse scores, lack interpretable rationales, and fail to model holistic session context. We present CARE, an LLM-based framework to automatically predict multi-dimensional alliance scores and generate interpretable rationales from counseling transcripts. Built on the CounselingWAI dataset and enriched with 9,516 expert-curated rationales, CARE is fine-tuned using rationale-augmented supervision with the LLaMA-3.1-8B-Instruct backbone. Experiments show that CARE outperforms leading LLMs and substantially reduces the gap between counselor evaluations and client-perceived alliance, achieving over 70% higher Pearson correlation with client ratings. Rationale-augmented supervision further improves predictive accuracy. CARE also produces high-quality, contextually grounded rationales, validated by both automatic and human evaluations. Applied to real-world Chinese online counseling sessions, CARE uncovers common alliance-building challenges, illustrates how interaction patterns shape alliance development, and provides actionable insights, demonstrating its potential as an AI-assisted tool for supporting mental health care.

Abstract

Explore similar work

CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models

CounselReflect: Opportunities and Challenges for Designing Tools to Support Self-Reflection on Mental Health and Well-Being Conversations with AI

Beyond Reflection: Affirmation as a Promising Behavioral Marker Associated with Quality in Text-Based Counseling