Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues

Yu Wang; Olcay Türk; Angela Grimminger; Hendrik Buschmeier

cs.CLMar 20, 2026

Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues

Authors: Yu Wang, Olcay Türk, Angela Grimminger, Hendrik Buschmeier

Organizations: Faculty of Linguistics and Literary Studies, Bielefeld University, Bielefeld, Germany · SFB/Transregio 318 ‘Constructing Explainability’, Bielefeld & Paderborn, Germany · Faculty of Arts and Humanities, Paderborn University, Paderborn, Germany

Abstract

We investigate how verbal and nonverbal linguistic features, exhibited by speakers and listeners in dialogue, can contribute to predicting the listener's state of understanding in explanatory interactions on a moment-by-moment basis. Specifically, we examine three linguistic cues related to cognitive load and hypothesised to correlate with listener understanding: the information value (operationalised with surprisal) and syntactic complexity of the speaker's utterances, and the variation in the listener's interactive gaze behaviour. Based on statistical analyses of the MUNDEX corpus of face-to-face dialogic board game explanations, we find that individual cues vary with the listener's level of understanding. Listener states ('Understanding', 'Partial Understanding', 'Non-Understanding' and 'Misunderstanding') were self-annotated by the listeners using a retrospective video-recall method. The results of a subsequent classification experiment, involving two off-the-shelf classifiers and a fine-tuned German BERT-based multimodal classifier, demonstrate that prediction of these four states of understanding is generally possible and improves when the three linguistic cues are considered alongside textual features.

Figures & tables

Figure 1: Explanation set-up in the MUNDEX corpus (screenshot from one camera-perspective). The person on the left is the explainer who explains a board game; the person on the right is the explainee.

Label	N	%
Understanding	176	27.4
Partial Understanding	162	25.2
Non-Understanding	191	29.8
Misunderstanding	113	17.6

Table 1: Distribution of labels for different states of understanding extracted from annotations during the game explanation phase.

Figure 2: The general quantification pipeline for getting average information value, average gaze entropy, average syntactic complexity score, and average dependency length.

Predictor	H	p	$\eta^{2}$
Information Value	$9.029$	$0.0289$	$0.0094$
Gaze Entropy	$9.035$	$0.0288$	$0.0095$
Syntactic Complexity	$8.853$	$0.0313$	$0.0092$
Dependency Length	$1.344$	$0.7187$	$-0.0025$

Table 2: Kruskal-Wallis test results showing associations between different understanding states and each predictor variable.

Figure 3: Variation of the quantified linguistic cues under different states of understanding (‘U’). Horizontal bars show statistically significant Dunn’s post-hoc tests (Bonferroni-corrected,

\alpha=0.05

Figure 4: Understanding state classification by fusing linguistic cues to a fine-tuned BERT model. We first fine-tuned a (German) BERT model with the dialogue data from the MUNDEX corpus in order to learn potential textual features related to understanding states. We then focused on the last four hidden layers, fusing them with the three significant linguistic cues identified in Section 5.1 (average information value, average gaze entropy, syntactic complexity score) .

		Random Forest			XGBoost			German BERT
	Class	Precis.	Recall	F1	Precis.	Recall	F1	Precis.	Recall	F1
Textual only Xx	Understanding	0.74	0.70	0.72	0.59	0.80	0.68	0.76	0.80	0.78
	Partial Understanding	0.75	0.67	0.71	0.71	0.59	0.65	0.80	0.67	0.73
	Non-Understanding	0.55	0.77	0.64	0.79	0.65	0.71	0.71	0.91	0.80
	Misunderstanding	0.67	0.33	0.44	0.54	0.54	0.54	0.88	0.58	0.70
	10-CV Acc. (mean ± SD)	0.762 ± 0.050			0.695 ± 0.049			0.768 ± 0.139

Table 3: Comparison of the three classifiers with textual features only (upper half) and textual features and linguistic cues combined (lower half). Best scores in bold.

Figure 5: Confusion matrix for the German BERT model for classifying understanding states (‘U’) with textual features and linguistic cues.

Multimodal Understanding

Multimodal understanding studies how AI systems interpret and reason across text, images, video, audio, and other signals.

See all 479 papers

Predicting Cognitive Load from Speech and Interaction Dynamics in Dyadic Conversations
Jun 11, 2026Tahiya ChowdhurySpeech Processing
Department of Computer Science, Colby College, Waterville, Maine, USA
How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks
Jul 20, 2026Tahiya ChowdhurySpoken Dialogue Systems Multimodal Model Evaluation
Colby College Waterville, Maine, United States
From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection
Jun 26, 2026Yasaman Haghbin, Sina Rashidi, Ali Zolnour +6Transformer Interpretability Explainable Artificial Intelligence
Independent Researcher · Columbia University, United States · Chalmers University of Technology, Sweden

Jun 11, 2026J/K move · Enter open · S save

Predicting Cognitive Load from Speech and Interaction Dynamics in Dyadic Conversations

Tahiya Chowdhury

Department of Computer Science, Colby College, Waterville, Maine, USA

Speech Processing

Open paper AskPDF arXiv

Estimating cognitive load from speech has largely been studied in controlled laboratory settings, with limited understanding of its reliability in natural collaborative conversations. We investigate whether speech and interaction dynamics predict perceived cognitive load during dyadic conversations. We analyze audio from 53 dyads performing nine collaborative tasks and extract static acoustic, dynamic, and interaction features to train a two-head Gated Recurrent Unit encoder to predict cognitive load scores. Results show conversational interaction provides useful signals for predicting cognitive load related to time pressure, mental work, effort, and task performance. Temporal demand is associated with turn-taking dynamics such as overlap and speaker switch, while mental demand is linked to imbalanced participation between speakers. These findings highlight the importance of task structure and conversational interaction for modeling cognitive load in natural collaborative settings.

Abstract

Multimodal Understanding

Explore similar work

Predicting Cognitive Load from Speech and Interaction Dynamics in Dyadic Conversations

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks

From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection