Contextual Pictograms for Repairing Multilingual ASR Errors
Authors: Mariana Hernandez, Hector Luna, Carlos Bruno, Luis Ricardo Santos, Elena Simoes
Organizations: Department Signal Theory, Telematics and Communications, University of Granada · Brain, Mind, and Behavior Research Center (CIMCYC), University of Granada · Faculty of Psychology and Education Sciences, University of Coimbra · Institute of Intelligent Systems and Robotics, CNRS, Sorbonne Université
Speech transcription errors can be difficult to recognize when competing word hypotheses are unfamiliar or visually similar. We describe an interface that combines alternative transcripts, replay and contextually checked pictograms for selected uncertain concepts. The system will initially support French and Spanish functional utterances. A counterbalanced study will assess whether pictorial feedback helps listeners recover the speaker's intended action beyond text alternatives alone. We will also quantify semantic mismatch and additional interaction time, since an inappropriate image can reinforce the wrong interpretation rather than repair it.