cs.HCOct 2, 2025

EvolveCaptions: Empowering DHH Users Through Real-Time Collaborative Captioning

Authors: Liang-Yuan Wu, Dhruv Jain

Organizations: University of Michigan, Ann Arbor, MI, USA

Abstract

Automatic Speech Recognition (ASR) systems often struggle to accurately transcribe speech from Deaf and Hard of Hearing (DHH) speakers. Existing personalization approaches typically require substantial pre-recorded datasets and burden DHH users. We present EvolveCaptions, a collaborative, error-driven personalization system that connects caption correction with speaker-specific ASR adaptation. When transcribing a DHH speaker's speech, hearing partners repair transcription errors, and the system generates short, targeted clauses for the DHH speaker to record. These samples are used to incrementally fine-tune the model. In a study with 12 DHH and six hearing participants, EvolveCaptions reduced Word Error Rate (WER) across five scripted reading rounds, requiring an average of five minutes of additional speech recording. Participants found the interfaces usable but considered recording and correction effortful; acceptability depended on context and anticipated long-term benefit. These findings show the potential of collaborative, cumulative adaptation while exposing the labor and conditions needed to sustain it.

Figures & tables

Explore similar work

CardsList
  1. DIY Customized Speech Recognition for a Person with Speech Disabilities

    Nov 8, 2026Paulo A. Condado, Fernando G. LoboPersonalized Speech RecognitionDysarthric Speech Recognition

  2. Adapting Foundation ASR Models to Dysarthric Speech: A Case Study

    Jun 30, 2026Christian Huber, Laura Kernahan, Alexander WaibelPersonalized Speech RecognitionAutomatic Speech Recognition

  3. Demonstration of Adapt4Me: An Uncertainty-Aware Authoring Environment for Personalizing Automatic Speech Recognition to Non-normative Speech

    Mar 20, 2026Niclas Pokel, Yiming Zhao, Pehuén Moure +2Personalized Speech RecognitionAutomatic Speech Recognition