Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training

Tobias Hallmen; Elisabeth André

cs.CLOct 6, 2026

Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training

Authors: Tobias Hallmen, Elisabeth André

Organizations: Chair for Human-Centered Artificial Intelligence University of Augsburg

Abstract

Automatic assessment of communication quality in dyadic counseling conversations is bottlenecked by data: expert-rated corpora are small and expensive to grow. We study cross-domain transfer of expert overall-impression prediction across three German corpora of simulated counseling (two general-practice medical, one school-related parent-teacher; $n=195$ expert-rated sessions, one corpus after scale equating). Training on the other domains beats training in-domain: leave-one-domain-out transfer reaches nested Spearman $ρ= 0.54$ against $\le 0.48$ within the target domain, a paired session-level gap of $+0.15$ that holds at $+0.12$ when the training-set sizes are matched, so it is not simply data volume. The decisive features are session-level construct scores from small open-weight LLMs reading the two-speaker transcript, with the constructs largely derived from the experts' rating instruments: the instrument-derived battery lifts a single judge from $0.32$ to $0.41$ over generic dialogue qualities, judges from three model families ensemble to $0.51$ language-only, and a nonverbal-dyadic block adds $+0.03$ more, not separable from noise at this sample size. We also price the recording setup: one corpus lost its per-speaker audio, 16% of its diarised segments carry the wrong speaker, and repair is worth $+0.07$ there. At practically attainable corpus sizes, the expert's overall impression is carried by what is said, and by other communication programs' data more than by one's own.

Figures & tables

Panel of Figure 1 — Figure 1: Pipeline. Both speakers’ transcripts feed LLM judges whose constructs are largely derived from the expert rating instruments; nonverbal-dyadic features come from the recordings. All low-dimensional blocks are z-calibrated per domain (features only), then a model trained on the two other domains predicts the held-out domain’s expert rating.

	Ticks	Vacc	Teach
sessions	57	53	85
duration (min)	7.1 $\pm$ 0.6	6.9 $\pm$ 0.9	9.5 $\pm$ 2.3
assessed role	doctor	doctor	teacher
interlocutor role	patient	patient	parent
target instrument	BGR	BGR	global item
rating	1–5	1–5	1–5 / 1–4

Table 1: Corpora. Berlin Global Rating overall impression (BGR; inverted so 5 = best). ICC = between-expert agreement on the double-rated subsets for Ticks / Vacc , and single-rater/average-rater ICC(2,1)–ICC(2,

k

) on the four-rater panel for Teach , which rated a parallel overall item rather than the target one (Appendix A ). Teach switched its overall item from five to four points after the first 24 sessions (Appendix A ).

Figure 2: Score distributions over the 195 sessions. Row blocks are construct groups by source instrument (the inventory is Table 4 ); the generic battery appears twice, once per model, and the cross-family BGR+OSCE runs are omitted. Most constructs occupy a 3–4 band, but discriminative signal survives per-domain calibration.

Figure 3: Modality attribution under LODO transfer: the language-based judges dominate; every nonverbal family, and the surface transcript statistics grouped with them, stays below the weakest instrument-anchored judge. Initial = four judges before the cross-family runs and the repair of Section 6 ; final = five after both, which also drops the phase

+

technique judge, so the sets are not nested. Stack = those judges plus the compact block, its deltas and prosody, not the nonverbal-dyadic set.

Language: LLM judges
	LODO $\rho$		CV $\rho$
Feature set	fixed	best	best
final ensemble (5 judges)	.528	.528	.453
initial ensemble (4 judges)	.450	.465	.416
granite-30B judge	.448	.464	.405
BGR+OSCE judge	.418	.431	.293

Table 2: Main results (Spearman

\rho

). fixed = one configuration for all feature sets (PCA-10

+

elastic net; a plateau width, not a tuned one, Appendix D ); best = best selector

\times

regressor per feature set (Appendix F ). LODO = mean over three held-out domains, CV = 5-fold within domain. Bold = best cell in the column; underlined = best language-only cell in the column: it is within

.03

of the bold everywhere, while the whole nonverbal block stays below

.30

. Ensemble membership among the final judges is chosen by nested LODO (

\rho=.513

language-only,

.542

with the nonverbal-dyadic block, Section 6 ). Nonverbal blocks are quoted at their best cell throughout, so the modality comparison is stacked against our own conclusion.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Placement of the four-point era	$\rho$	ties
$\{1,3,4,5\}$ as stored	.44	33.2%
$\{1,2,4,5\}$ midpoint-skipping	.47	30.8%
$\{1,2.33,3.67,5\}$ equal-interval	.53	21.0%
$\{1,2,3,4\}$ shifted below	.58	30.4%
equipercentile †	.40	50.3%
within-era reference	.51	—

Appendix

Table 3: Sensitivity of the Teach fold to how the four-point era is placed on the five-point range, at one fixed reference configuration ( lowdim_all234_z , elastic net on 10 principal components). The within-era reference is invariant to the placement, since every candidate is monotone within an era; a pooled value below it indicates destroyed cross-era ordering, above it manufactured separation. † The equipercentile map degenerates at

n{=}24

, sending two four-point categories to the same value, which is why it also depresses the within-era reference (to

.44

) rather than only the pooled score.

Battery	Model (family)	#c	ens.
generic dialogue	gemma3-12B (gemma)	5	$\bullet$
generic dialogue	qwen3-14B, think (qwen)	5	$\bullet$
BGR $+$ OSCE	gemma3-12B (gemma)	8	$\bullet$
anti-band wording	gemma3-12B (gemma)	8
anti-band wording	qwen3.6-27B (qwen)	8	$\bullet$
anti-band wording	granite4.1-30B (granite)	8	$\bullet$

Appendix

Table 4: The eight full judge runs: three construct batteries (one of them in two promptings, Appendix B ) across five models from four open-weight families ( Gemma Team, 2025 ; Qwen Team, 2025 ; IBM Research, 2026 ; Abdin et al., 2024 ) . #c = constructs returned per session; ens. marks the five judges nested selection keeps in the final ensemble (Section 6 ). Each run scores all 195 sessions. The two generic-dialogue runs share a prompt and differ only in model, which is what makes them ensemble partners with decorrelated errors; the phase

+

technique judge overlaps the BGR

+

OSCE battery in construct content and is not selected.

Prompt	LODO $\rho$	folds
generic, 5 constructs	.321	.34/.32/.30
experts’ 8 constructs, bare	.414	.45/.36/.43
$+$ quoted level descriptions	.418	.40/.43/.42

Appendix

Table 5: Anchoring control: same model, transcripts, seeds, construct keys and JSON contract; only the instrument-derived level descriptions differ.

Construct (judge)	Tic	Vac	Tea	min
phase completeness (P)	.33	.44	.40	.33
coherence (B)	.33	.35	.38	.33
overall grade (B)	.32	.29	.41	.29
clarity (think)	.31	.27	.28	.27
structure (generic)	.26	.31	.27	.26
empathy (generic)	.30	.25	.38	.25

Appendix

Table 6: Per-construct Spearman

\rho

with the expert target, per domain (B = BGR+OSCE judge, P = phase+technique judge, think = thinking variant); the six most domain-consistent constructs (top, ranked by their weakest domain) and the three least stable (bottom). Bold = best, underlined = second best per column, over all constructs of all judges.

Held out	selected inside the fold	$\rho$
Ticks	final stack, PCA-2, ridge	.565
Vacc	final stack w/o prosody, PCA-5, ridge	.489
Teach	judges $+$ nonverbal, PCA-3, SVR	.570
mean		.542

Appendix

Table 7: Fully nested LODO: feature set, selector and regressor chosen inside each fold on an inner leave-one-training-domain-out split.

$k$	PCA	$F$ -score	RFE	mut. inf.	headline
1	.240	.219	.191	.168	.251
2	.269	.248	.223	.204	.552
3	.277	.248	.224	.206	.551
5	.278	.263	.242	.224	.551
8	.289	.269	.251	.246	.551
12	.285	.267	.270	.253	.551

Appendix

Table 8: Selector families on one matched grid (LODO Spearman

\rho

, mean over all feature sets of Table 2 and all five regressors;

k

capped per feature set at its width). The sweep of Appendix F gave the families different grids, and

F

-score and mutual information carried collapse-prone

k\leq 3

cells that RFE never ran, so its family means were not comparable; these are. PCA wins at every

k

up to 12 and degrades most gracefully at

k=1

. headline = the paper’s configuration (PCA

+

elastic net on the final stack), which returns

\rho=.551

for every dimensionality from 3 to 50 and

.552

at 2: the choice of PCA-10 is arbitrary within that plateau, not a tuned value.

final stack
Held-out	$m$	LODO	LODO m	CV m	$>$ CV
Ticks	46	.562	.529 $\pm$ .048	.384 $\pm$ .052	0.99
Vacc	42	.483	.442 $\pm$ .046	.306 $\pm$ .049	0.98
Teach	68	.607	.592 $\pm$ .025	.527 $\pm$ .023	0.98
mean		.551	.521	.405
final judges (language only)

Appendix

Table 9: Training-set-size control (Spearman

\rho

, PCA-10

+

elastic net). LODO = the paper’s transfer setting, trained on all

110

–

142

pooled foreign sessions; LODO m = the same, with the foreign training set subsampled to

m\approx 0.8n

sessions, the size the in-domain reference trains on (mean

\pm

sd over 200 draws); CV m = in-domain 5-fold CV, which trains on

m

sessions per fold by construction, scored on the same sessions and averaged over 20 random splits, since a single split is unstable at this

n

>

CV = fraction of draws above CV m . Removing the size advantage costs

.03

on the stack and

.01

language-only; the remaining gap of

+.12

and

+.06

is provenance, not volume.

	foreign	$+$ own	own only
final stack	.551	.526	.392
final judges	.528	.507	.423

Appendix

Table 10: Training on foreign data only, on foreign plus the domain’s own training fold, and on the domain’s own data only, scored on identical held-out sessions.

	$\rho$	MAE	mean-only	QWK	sd(pred)
Ticks	.562	.699	.908	.347	0.43
Vacc	.498	.728	.816	.361	0.49
Teach	.566	.652	.754	.375	0.38
pooled	.545	.687	.815	.364	0.43

Appendix

Table 11: Absolute-error and agreement metrics for the final stack, per domain. mean-only is the predict-the-mean MAE baseline.

Identifier	Meaning
llm $N$ _only	judge $N$ alone: 2 generic (gemma3-12B), 3 thinking (qwen3-14B), 4 BGR $+$ OSCE (gemma3-12B), 5 phase $+$ technique, 6 anti-band BGR $+$ OSCE (Appendix B ), 7 qwen3.6-27B, 7m its 3-seed mean, 8 phi4-14B, 9 granite4.1-30B
llm $N_{1}N_{2}\ldots$ _z	per-domain z-scored scores of those judges, concatenated
lowdim_all $\ldots$ _z	the same judges plus the compact block and the dyadic deltas
c / f suffix	judge scores from the AU50-repaired vaccines transcripts (Section 6 ); c = every judge of the set repaired, plus the seed-mean qwen3.6
pro / prosody2	prosody block (semitone $F_{0}$ , spread/slope, jitter/shimmer/HNR)
shallow_ling7	shallow transcript statistics (7 surface counts)

Appendix

Table 12: Feature-set identifiers used in the result grids (Tables 13 and 14 ) and in the released CSVs.

Feature set	$F$ -score	MI	RFE	PCA
lowdim_all23479c_pro_z	0.518	0.461	0.467	0.553
lowdim_all2349c_z	0.518	0.512	0.536	0.549
lowdim_all23479c_z	0.499	0.478	0.449	0.545
lowdim_all234789c_z	0.526	0.487	0.501	0.536
lowdim_all23479_z	0.493	0.479	0.500	0.534
lowdim_all2349f_z	0.511	0.512	0.519	0.529

Appendix

Table 13: Full LODO grid: Spearman

\rho

of the best cell per feature set

\times

selector family (best regressor and best

k

/dimensionality;

k\in\{1,2,3,5,8,12,20,50\}

for

F

-score/MI,

\{5,20\}

for RFE, 2–10 components for PCA). Bold = best family per feature set. LODO = mean over three held-out domains. Identifiers are decoded in Table 12 .

Feature set	$F$ -score	MI	RFE	PCA
lowdim_all234789c_z	0.356	0.317	0.323	0.482
lowdim_all23479c_z	0.374	0.372	0.386	0.471
lowdim_all23479c_pro_z	0.383	0.339	0.356	0.465
llm234789c_z	0.390	0.390	0.309	0.456
llm2349c_z	0.371	0.342	0.321	0.453
lowdim_all2349c_z	0.368	0.361	0.386	0.453

Appendix

Table 14: Full CV grid: Spearman

\rho

of the best cell per feature set

\times

selector family (best regressor and best

k

/dimensionality;

k\in\{1,2,3,5,8,12,20,50\}

for

F

-score/MI,

\{5,20\}

for RFE, 2–10 components for PCA). Bold = best family per feature set. CV = 5-fold within domain, mean. Identifiers are decoded in Table 12 .

Feature set	Ticks	Vacc	Teach
final stack, nested	.562	.498	.566
compact-block variant, nested	.501	.483	.607
final judges only, nested	.484	.464	.589
final stack (23479c $+$ pro)	.562	.483	.607
final judges only (23479c)	.530	.464	.589
initial stack, 3 judges (234)	.470	.395	.531

Appendix

Table 15: Per-fold LODO Spearman

\rho

. The nested rows select ensemble membership inside each fold (Section 4.4 ); the remaining rows hold one configuration fixed across all folds (PCA-10

+

elastic net, except nonverbal-dyadic at its best cell, PCA-7

+

SVR).

Explore similar work

Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues
Jun 28, 2025Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto +5Counseling Prompt Engineering
Japan National Institute of Occupational Safety and Health, Kanagawa, Japan · Kaze To Taiyo, Tokyo, Japan · Saga Occupational Health Association, Saga, Japan +5
SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences
Sep 28, 2026Mingyue Huo, Shivam Mehta, Bhavin Jawade +2Large Audio Language Models Critic
University of Illinois Urbana-Champaign · Netflix
A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents
Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Rater Full-Duplex Voice Agents
Salesforce Applied AI Research, eVerse team

Jun 28, 2025, cs.CLJ/K move · Enter open · S save

Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues

Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto, Tomonori Hosokawa+4

Japan National Institute of Occupational Safety and Health, Kanagawa, Japan · Kaze To Taiyo, Tokyo, Japan · Saga Occupational Health Association, Saga, Japan+5

Counseling Prompt Engineering

Open paper AskPDF arXiv

Large language models (LLMs) may support counseling training, yet evidence from Japanese-language interactions and automated quality ratings remains limited. We examined 18 fixed Japanese-language counseling transcripts generated through artificial intelligence (AI)-to-AI interactions under three counselor conditions: GPT-minimal (GPT-4-turbo with a minimal role instruction), GPT-SMDP (GPT-4-turbo with the Structured Multi-step Dialogue Prompt [SMDP]), and Claude-SMDP (Claude-3-Opus with SMDP). Fifteen counseling experts rated transcripts on four adapted global scales from the Motivational Interviewing Treatment Integrity coding manual and an overall-quality item; three newer LLMs independently rated the same transcripts in three iterations. In this fixed stimulus set, SMDP-condition dialogues received higher expert ratings for cultivating change talk, partnership, empathy, and overall quality than GPT-minimal dialogues; the two SMDP counselor models did not differ. LLM ratings were reproducible but generally more lenient than expert-reference ratings, particularly for softening sustain talk and overall quality. Simulated-client naturalness was below the scale midpoint. These findings provide an expert-referenced benchmark for Japanese-language AI counseling simulations and show that reproducible LLM ratings should not be treated as calibrated counseling-quality evidence without expert validation. This study does not test clinical effectiveness or human-client outcomes.

Abstract

Explore similar work

Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues

SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences

A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents