EvolveCaptions: Empowering DHH Users Through Real-Time Collaborative Captioning
Organizations: University of Michigan, Ann Arbor, MI, USA
Abstract
Automatic Speech Recognition (ASR) systems often struggle to accurately transcribe speech from Deaf and Hard of Hearing (DHH) speakers. Existing personalization approaches typically require substantial pre-recorded datasets and burden DHH users. We present EvolveCaptions, a collaborative, error-driven personalization system that connects caption correction with speaker-specific ASR adaptation. When transcribing a DHH speaker's speech, hearing partners repair transcription errors, and the system generates short, targeted clauses for the DHH speaker to record. These samples are used to incrementally fine-tune the model. In a study with 12 DHH and six hearing participants, EvolveCaptions reduced Word Error Rate (WER) across five scripted reading rounds, requiring an average of five minutes of additional speech recording. Participants found the interfaces usable but considered recording and correction effortful; acceptability depended on context and anticipated long-term benefit. These findings show the potential of collaborative, cumulative adaptation while exposing the labor and conditions needed to sustain it.
Figures & tables
get hub,'' and a hearing partner changes it to GitHub.'' The corrected term is sent to a large language model. In Stage 2, the model generates the clause ``I will push the code to GitHub today,'' which the DHH speaker records. In Stage 3, the recording and transcript fine-tune the speaker's automatic speech recognition model. The updated model loops back to caption the speaker's future speech.| ID | Age | Gender | Identity | Self-Reported Hearing Loss | Age of Onset | Paired Participant |
|---|---|---|---|---|---|---|
| P1 | 38 | Female | HoH | Severe | 18 months | H1 |
| P2 | 25 | Male | Deaf | Profound | 5 years | H2 |
| P3 | 68 | Male | HoH | Severe | Birth | H2 |
| P4 | 30 | Male | HoH | Moderately Severe | 25 years | H3 |
| P5 | 85 | Male | HoH | Profound | Birth | H2 |
| P6 | 32 | Female | HoH | Severe | Birth | H3 |
| ID | Age | Gender | English Proficiency |
|---|---|---|---|
| H1 | 21 | Female | Native Speaker |
| H2 | 25 | Female | Bilingual |
| H3 | 24 | Male | Native Speaker |
| H4 | 29 | Male | Proficient |
| H5 | 28 | Female | Proficient |
| H6 | 26 | Male | Proficient |