Emotion prosody perception requires simultaneous processing of acoustic cues and speaker identity. While listeners effortlessly decode natural speech, AI synthetic voices introduce cognitive complexities due to subtle acoustic atypicalities. It remains unclear how these synthetic features interact with a listener's prior social knowledge and memory of a familiar speaker. This study investigated how speech sources (human vs. AI) and speaker familiarity affect emotion recognition accuracy and cognitive load. A within-subject task with Mandarin-speaking adults evaluated behavioral (accuracy, reaction time) and physiological data (heart rate variability). Results showed that human voices yielded significantly higher accuracy and faster processing times than AI voices, while HRV did not significantly differentiate between conditions. These findings show that decoding synthetic speech is gated by top-down social cognition, highlighting limitations in current AI synthesis technologies.
Wichita State University, Wichita, KS, USA. · Institut national de la recherche scientifique (INRS–EMT), QC, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Canada
Graduate School of Interdisciplinary Information Studies The University of Tokyo · Department of Psychology, Faculty of Health and Wellness Sciences Hiroshima International University · Interfaculty Initiative in Information Studies The University of Tokyo