Audio-Visual Speech Recognition

Audio-visual speech recognition studies how to combine acoustic speech with lip movements and other visual context to produce more accurate transcripts. Common approaches fuse audio and video features using CTC or attention-based models, self-supervised learning, and increasingly large language models with cross-modal alignment or iterative refinement. Researchers evaluate systems on benchmarks such as LRS3 as well as noisy, conversational, multi-view, and real-world recordings, where visual cues can help when audio is degraded. A central challenge is that gains on standard benchmarks do not ensure generalization: performance can collapse under domain shifts, and models may underuse or over-rely on one modality; limited labeled video data is another constraint.