CLAP-Based Automatic Word Naming Recognition in Post-Stroke Aphasia

Yacouba Kaloga; Marina Laganaro; Ina Kodrasi

eess.ASFeb 16, 2026

CLAP-Based Automatic Word Naming Recognition in Post-Stroke Aphasia

Authors: Yacouba Kaloga, Marina Laganaro, Ina Kodrasi

Organizations: Signal Processing for Communication Group, Idiap Research Institute, Martigny, Switzerland · Faculty of Psychology and Educational Science, University of Geneva, Geneva, Switzerland

Abstract

Conventional automatic word-naming recognition systems struggle to recognize words from post-stroke patients with aphasia because of disfluencies and mispronunciations, limiting reliable automated assessment in this population. In this paper, we propose a Contrastive Language-Audio Pretraining (CLAP) based approach for automatic word-naming recognition to address this challenge by leveraging text-audio alignment. Our approach treats word-naming recognition as an audio-text matching problem, projecting speech signals and textual prompts into a shared embedding space to identify intended words even in challenging recordings. Evaluated on two speech datasets of French post-stroke patients with aphasia, our approach achieves up to 90% accuracy, outperforming existing classification-based and automatic speech recognition-based baselines.

Figures & tables

Metric	Proposed	Classification	ASR
Accuracy	0.90 $\pm$ 0.07	0.85 $\pm$ 0.08	0.86 $\pm$ 0.04
Precision	0.90 $\pm$ 0.07	0.85 $\pm$ 0.09	0.85 $\pm$ 0.05
Recall	0.93 $\pm$ 0.06	0.89 $\pm$ 0.09	0.90 $\pm$ 0.05
F1 score	0.90 $\pm$ 0.07	0.85 $\pm$ 0.09	0.86 $\pm$ 0.05

TABLE I: Mean and standard deviation of accuracy, precision, recall, and F1 score across all folds for the proposed CLAP-based approach and the baseline approaches on the first dataset.

Metric	Proposed	Classification	ASR
Accuracy	0.85 $\pm$ 0.06	0.79 $\pm$ 0.08	0.80 $\pm$ 0.06
Precision	0.72 $\pm$ 0.21	0.65 $\pm$ 0.22	0.67 $\pm$ 0.16
Recall	0.79 $\pm$ 0.23	0.69 $\pm$ 0.23	0.68 $\pm$ 0.19
F1 score	0.74 $\pm$ 0.22	0.65 $\pm$ 0.23	0.66 $\pm$ 0.17

TABLE II: Mean and standard deviation of accuracy, precision, recall, and F1 score across all folds for the proposed CLAP-based approach and the baseline approaches on the second dataset.

Clinical Speech Processing

Clinical speech processing uses speech recognition and acoustic analysis to support health assessment, monitoring, and communication.

See all 57 papers

Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns
Jul 13, 2026Yong Yang, Xiang Guan, Sophie Arheix-Parras +7Multimodal Large Language Models Language Modeling
Department of Computer Science and Engineering, University of South Carolina · Department of Psychology, University of South Carolina · Department of Communication Sciences and Disorders, University of South Carolina +2
AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation
Oct 10, 2026Al Varrel Putra Kusuma, Fernando Bello, Joshua BrownTTS Synthesis Clinical Speech Processing
Imperial College London, United Kingdom
Cross-lingual Retrieval-Augmented Classification for Dysarthria Severity Assessment
Jun 22, 2026Taeyoung Jeong, Insung Lee, Du-Seong Chang +1Retrieval-Augmented Classification Speech Quality Assessment
Department of Artificial Intelligence, Sogang University, South Korea

Jul 13, 2026J/K move · Enter open · S save

Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns

Yong Yang, Xiang Guan, Sophie Arheix-Parras, Saeed Ahmadi+6

Department of Computer Science and Engineering, University of South Carolina · Department of Psychology, University of South Carolina · Department of Communication Sciences and Disorders, University of South Carolina+2

Multimodal Large Language Models Language Modeling Clinical Language Model Evaluation

Open paper AskPDF arXiv

Aphasia following stroke commonly produces systematic naming errors with characteristic profiles, but whether general-purpose language models not designed for clinical simulation can reproduce these patterns remains untested. We investigated (1) whether lesions or controlled perturbations to a multimodal language model can reproduce different types of errors in picture naming, and (2) whether the framework can reproduce the complete error profile of individual persons with aphasia (PWAs). Using LLaVA 1.6, we evaluated perturbation configurations that varied the layer, proportion, and amount of noise applied to model units. We examined 278 PWAs on the Philadelphia Naming Test, classifying responses into seven categories using a validated neural classifier. Six of seven response categories (correct, semantic, mixed, unrelated, neologism, no response errors) emerged at clinically-comparable proportions across distinct parameter space regions, with formal paraphasia being the exception. Searching the perturbation space revealed configurations that reproduced the individual error profile in at least six of seven categories for 97.8% of PWAs and in all seven categories for 79.5% of PWAs. Monte Carlo baselines confirmed that this matching reflects joint inter-category structure rather than marginal overlap. These results establish a quantitative framework for reproducing individual aphasic error patterns in picture naming. They suggest the potential for language models to serve as digital twins of individuals with post-stroke aphasia.

Abstract

Clinical Speech Processing

Explore similar work

Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns

AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation

Cross-lingual Retrieval-Augmented Classification for Dysarthria Severity Assessment