cs.CLAug 27, 2026
SaveBeyond Reflection: Affirmation as a Promising Behavioral Marker Associated with Quality in Text-Based Counseling
Organizations: The University of Electro-Communications 1-5-1, Chofugaoka, Chofu, Tokyo, Japan
Abstract
While AI-assisted text-based counseling is gaining attention, it remains empirically unclear which counselor behaviors are associated with higher dialogue quality. Existing research often focuses heavily on Reflection, borrowing frameworks from Motivational Interviewing. To address this gap, we conduct a multi-layered analysis using KokoroChat, a large-scale Japanese text counseling dataset conducted by professional counselors and trainees, newly annotated with counselor strategy tags and client distress levels. Our results show that, under the quality indicators used in this study, Affirmation is more consistently associated with session quality than Reflection among the analyzed strategies. Cross-dataset transfer experiments further suggest that this quality signal can be observed to some extent on ESConv, an English dataset with non-expert supporters. These findings provide empirical implications for counselor training and emotional support system design. We release the additional KokoroChat annotations and experimental source code at https://github.com/UEC-InabaLab/BeyondReflection.
Figures & tables
| Tag | Definition | Example utterance |
|---|---|---|
| OpenQuestion | A question intended to elicit a broad or elaborated response from the client. | “What has happened recently?” |
| ClosedQuestion | A yes/no question or one with a restricted answer set, including questions about quantities or time. | “Did that happen recently?” |
| Paraphrase | A restatement of the factual or propositional content of the client’s preceding utterance, without interpreting emotion. | “So, what you mean is that …” |
| Reflection | A verbalization of the client’s emotions or underlying personal meaning intended to clarify feelings or convey empathic understanding. | “That must have been painful for you.” |
| Affirmation | A positive evaluation of the client’s strengths, efforts, motivations, abilities, or actions; emotional validation alone is insufficient. | “It is wonderful that you have been able to keep trying that hard.” |
| Suggest | A proposal of an action, perspective, or solution intended to encourage the client to act or reconsider. | “How about trying …?” |
Table 1: Counselor strategy annotation scheme for KokoroChat, with definitions and example utterances.
Figure 1: Tag usage rates by client-rated score tier. Brackets and asterisks indicate significant pairwise differences between the connected tiers in Bonferroni-corrected chi-square tests at . The significance markers refer to tier differences, not correlations.
| Tag | Adj. | Partial | Adj. | |
|---|---|---|---|---|
| Affirmation | -0.201 | -0.200 | ||
| Suggest | -0.130 | -0.119 | ||
| Thanking | -0.040 | 0.025 | -0.050 | |
| Other | -0.027 | 0.523 | -0.035 | 0.080 |
| Paraphrase | -0.018 | 1.000 | -0.034 | 0.113 |
| Inform | -0.014 | 1.000 | -0.001 | 1.000 |
Table 2: Spearman and partial Spearman correlations between counselor strategy tag usage rates and distress. Partial correlations control for the initial distress level, defined as the mean distress level in the first half of the session. Adjusted -values (Adj. ) are Bonferroni-corrected.
| Feature | Coefficient |
|---|---|
| Affirmation | |
| Question | |
| Suggest | |
| Other | |
| Reflection |
Table 3: Regression coefficients in the KokoroChat-to-ESConv transfer model.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Prompt for counselor strategy tag annotation
Figure 3: English translation of the prompt for counselor strategy tag annotation
Figure 4: Prompt for distress level annotation
Figure 5: English translation of the prompt for distress level annotation
| Tag | Support | Precision | Recall | F1 |
|---|---|---|---|---|
| Other | 29 | 0.560 | 0.486 | 0.520 |
| ClosedQuestion | 46 | 0.714 | 0.650 | 0.666 |
| Suggest | 14 | 0.579 | 0.790 | 0.667 |
| Paraphrase | 36 | 0.750 | 0.675 | 0.710 |
| Greeting | 8 | 0.650 | 0.817 | 0.721 |
| Backchannel | 14 | 0.658 | 0.870 | 0.744 |
Table 4: Tag-level precision, recall, and F1 scores of LLM strategy annotations, averaged over Annotator 1 and Annotator 2 as references. Support denotes the average number of utterances assigned to each tag by the two human annotators.
| Tag | High (%) | Mid (%) | Low (%) | (H L) |
|---|---|---|---|---|
| Reflection ∗ | 20.4 (n=23,257) | 20.0 (n=20,207) | 18.4 (n=16,775) | +2.0 |
| OpenQuestion ∗ | 19.1 (n=21,825) | 21.1 (n=21,342) | 23.3 (n=21,319) | -4.2 |
| ClosedQuestion ∗ | 10.2 (n=11,663) | 12.0 (n=12,064) | 14.1 (n=12,833) | -3.8 |
| Affirmation ∗ | 9.9 (n=11,366) | 8.1 (n=8,209) | 6.5 (n=5,929) | +3.5 |
| Other ∗ | 7.7 (n=8,774) | 7.8 (n=7,914) | 8.5 (n=7,797) | -0.9 |
| Suggest ∗ | 7.6 (n=8,695) | 7.2 (n=7,256) | 6.3 (n=5,736) | +1.3 |
Table 5: Exact counselor strategy tag usage rates by client-rated score tier. Percentages are calculated over all counselor utterances within each tier; counts in parentheses indicate the number of utterances assigned to each tag. The total numbers of counselor utterances were 114,244 for the high tier, 100,926 for the mid tier, and 91,325 for the low tier. (H L) denotes the high-tier usage rate minus the low-tier usage rate. ∗ indicates a significant high-vs.-low difference after Bonferroni correction in chi-square tests ( ).
| Predictor | Outcome / Model | Effect | |
|---|---|---|---|
| Early-session Affirmation | distress | ||
| Early-session Affirmation | Client-rated score | ||
| Early distress | Later Affirmation usage | ||
| Current distress | Next-turn Affirmation (mixed logit) | OR = 1.068 | |
| Current distress | Next-turn Affirmation (GEE) | OR = 1.105 |
Table 6: Temporal robustness checks for the relationship between Affirmation, client distress, and session quality. Lower distress indicates greater distress reduction. The 95% CI for the mixed-effects logistic model was [1.050, 1.086]. GEE was clustered by session.
| Analysis | Outcome | Estimate | SE | 95% CI | |
|---|---|---|---|---|---|
| Null mixed-effects model | distress | ICC = 0.028 | – | – | – |
| Null mixed-effects model | Client-rated score | ICC = 0.110 | – | – | – |
| Random-intercept model | distress | – | – | ||
| Random-intercept model | Client-rated score | – | – | ||
| Mundlak within-counselor component | distress | – | – | ||
| Mundlak within-counselor component | Client-rated score | – | – |
Table 7: Additional analyses accounting for counselor-level clustering. Coefficients for Affirmation usage are based on rates ranging from 0 to 1 and therefore represent the expected outcome change associated with a full-unit increase in the Affirmation usage rate. Lower distress indicates greater distress reduction.
| KokoroChat tag | ESConv tag | Note |
|---|---|---|
| Affirmation | Affirmation and Reassurance | Same definition; shortened label |
| Reflection | Reflection of feelings | Same definition; shortened label |
| OpenQuestion + ClosedQuestion | Question | Merged for transfer |
| Suggest | Providing Suggestions | Same definition; shortened label |
| Other | Others | Residual category |
Table 8: Mapping between KokoroChat strategy tags and ESConv strategy tags used in the cross-dataset transfer experiment.