We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.
Figures & tables
Figure 1: Explanation set-up in the MUNDEX corpus (screenshot from one camera-perspective). The person on the left is the explainer who explains a board game; the person on the right is the explainee.
Label
N
%
Understanding
176
27.4
Partial Understanding
162
25.2
Non-Understanding
191
29.8
Misunderstanding
113
17.6
Table 1: Distribution of labels for different states of understanding extracted from annotations during the game explanation phase.
Figure 2: The general quantification pipeline for getting average information value, average gaze entropy, average syntactic complexity score, and average dependency length.
Predictor
H
p
η2
Information Value
9.029
0.0289
0.0094
Gaze Entropy
9.035
0.0288
0.0095
Syntactic Complexity
8.853
0.0313
0.0092
Dependency Length
1.344
0.7187
−0.0025
Table 2: Kruskal-Wallis test results showing associations between different understanding states and each predictor variable.
Figure 3: Variation of the quantified linguistic cues under different states of understanding (‘U’). Horizontal bars show statistically significant Dunn’s post-hoc tests (Bonferroni-corrected, α=0.05 ).
Figure 4: Understanding state classification by fusing linguistic cues to a fine-tuned BERT model. We first fine-tuned a (German) BERT model with the dialogue data from the MUNDEX corpus in order to learn potential textual features related to understanding states. We then focused on the last four hidden layers, fusing them with the three significant linguistic cues identified in Section 5.1 (average information value, average gaze entropy, syntactic complexity score) .
Random Forest
XGBoost
German BERT
Class
Precis.
Recall
F1
Precis.
Recall
F1
Precis.
Recall
F1
Textual only Xx
Understanding
0.74
0.70
0.72
0.59
0.80
0.68
0.76
0.80
0.78
Partial Understanding
0.75
0.67
0.71
0.71
0.59
0.65
0.80
0.67
0.73
Non-Understanding
0.55
0.77
0.64
0.79
0.65
0.71
0.71
0.91
0.80
Misunderstanding
0.67
0.33
0.44
0.54
0.54
0.54
0.88
0.58
0.70
10-CV Acc. (mean ± SD)
0.762 ± 0.050
0.695 ± 0.049
0.768 ± 0.139
Table 3: Comparison of the three classifiers with textual features only (upper half) and textual features and linguistic cues combined (lower half). Best scores in bold.
Figure 5: Confusion matrix for the German BERT model for classifying understanding states (‘U’) with textual features and linguistic cues.
Estimating cognitive load from speech has largely been studied in controlled laboratory settings, with limited understanding of its reliability in natural collaborative conversations. We investigate whether speech and interaction dynamics predict perceived cognitive load during dyadic conversations. We analyze audio from 53 dyads performing nine collaborative tasks and extract static acoustic, dynamic, and interaction features to train a two-head Gated Recurrent Unit encoder to predict cognitive load scores. Results show conversational interaction provides useful signals for predicting cognitive load related to time pressure, mental work, effort, and task performance. Temporal demand is associated with turn-taking dynamics such as overlap and speaker switch, while mental demand is linked to imbalanced participation between speakers. These findings highlight the importance of task structure and conversational interaction for modeling cognitive load in natural collaborative settings.
Tahiya Chowdhury
Department of Computer Science, Colby College, Waterville, Maine, USA
Measuring conversational states such as cognitive load and conversational power from multimodal behavior requires characteristic features that are not only predictive but also reliable across task contexts. We present a three-dimensional evaluation framework assessing predictive accuracy, cross-task generalizability, and test-retest reliability, applied to interactional, acoustic, and linguistic features extracted from dyadic conversations during collaborative tasks performed over a video-conferencing platform (AVCAffe dataset; 53 dyads, 9 tasks). Our results show that no single feature family dominates all three dimensions. Linguistic features show the highest predictive accuracy for cognitive load but collapse under cross-task evaluation, revealing sensitivity to task-specific vocabulary. Additionally, acoustic reliability, often reported as evidence of feature stability, degrades once speaker identity is controlled, confirming that standard prosodic features measure vocal characteristics rather than conversational state. Interaction features provide the only genuinely reliable signal, unchanged after speaker normalization. Interestingly, classifying power role remained near chance baseline across all conditions, indicating limitations of task-level aggregated behavior for predicting power role in conversation. Our findings reveal three insights: (1) linguistic features predict best but generalize poorly across task contexts; (2) acoustic reliability collapses to near-zero once speaker identity is controlled, challenging standard evaluation practice; and (3) interaction features provide the only genuinely reliable signal, with floor dominance predicting within-dyad cognitive load asymmetry. These results argue for speaker normalization and multi-dimensional evaluation as prerequisites for context-aware, robust multimodal feature selection in conversational systems.
Speech-based cognitive impairment detection offers a noninvasive, accessible alternative to costly biomarker assays, yet transformer-based models remain clinically uninterpretable. We propose a multi-stage explainability framework that translates black-box transformer predictions into clinically grounded narratives by integrating SHapley Additive exPlanations (SHAP)-based token attribution, theory-informed linguistic features, and a four-stage LLM reasoning pipeline using LLaMA-3.1-70B-Instruct. Built on the SpeechCARE-Adaptive Gating Network multimodal screening model (F1 = 72.11% on the NIA PREPARE benchmark), the framework maps model outputs to four cognitive-linguistic dimensions, including lexical richness, syntactic complexity, and semantic coherence. Physician evaluation on 70 stratified English samples demonstrated strong alignment with patient-level cognitive profiles, and a System Usability Scale score of 82/100 indicated high potential for clinical workflow integration.
Yasaman Haghbin, Sina Rashidi, Ali Zolnour +6
Independent Researcher · Columbia University, United States · Chalmers University of Technology, Sweden