Current ASR models perform well on general benchmarks but still struggle with acoustically ambiguous words. Visual context can aid recognition in screen-based scenarios, yet existing benchmarks provide limited insight into the benefits of different types of visual evidence. To address this gap, we introduce ScreenASR-Bench , a tiered benchmark with three progressively challenging levels: Explicit Textual Evidence (ETE), based on direct on-screen text; Explicit Visual Evidence (EVE), using visible content rather than on-screen text; and Implicit Semantic Evidence (ISE), requiring scene-level inference. Extensive experiments on ASR models, Large Audio-Language Models (LALMs) and Omni models show that incorporating visual context substantially improves recall over the audio-only setting across all three model families. The gains are largest for ETE and smallest for ISE, indicating that models leverage direct visual evidence more effectively than indirect semantic cues. To facilitate future research on visually grounded speech recognition, we publicly release ScreenASR-Bench.
Figures & tables
Figure 1: Representative examples of the three visual-evidence levels in ScreenASR-Bench . From ETE to ISE, the semantic distance between the target keyword and its visual evidence increases.
Figure 2: Overview of the ScreenASR-Bench construction pipeline, consisting of three stages: (1) candidate collection, (2) automatic screening, and (3) expert verification. Candidate keywords are mined from multi-ASR results and the VLM refers to Qwen3.5-35B-A3B.
Model
Vision
ETE
EVE
ISE
Overall
WER
Recall
WER
Recall
WER
Recall
WER
Recall
ASR Models
FireRedASR2-LLM
-
7.02
35.11
5.34
43.07
6.15
34.62
6.22
37.87
MiMo-V2.5-ASR
-
6.97
47.07
4.80
59.28
5.22
54.99
5.77
53.43
Qwen3-ASR-1.7B
-
7.88
34.10
5.99
42.52
6.47
39.10
6.87
38.37
Caption
6.08
77.86
5.34
52.77
6.03
44.40
5.81
60.58
Table 1: Model-level WER and Recall. Vision denotes the visual input: – for audio-only, Caption for a VLM-generated keyframe caption, and Image for direct keyframe input. ETE, EVE, and ISE are the three evidence levels. Best and second-best results are bold and underlined.
Figure 3: Statistics of the ScreenASR-Bench .
Family
Level
Recall (%)
Δ Recall
NVG(%)
Audio-only
Audio + Vision
ASR
ETE
42.64
69.17
26.53
45.16
EVE
54.40
61.35
6.95
15.13
ISE
47.90
52.29
4.39
8.33
LALM
ETE
43.21
78.24
35.03
61.34
EVE
51.85
58.31
6.46
13.37
Table 2: Level-wise visual-context utilization. Recall, Δ Recall, and NVG are averaged within each family model; Δ Recall denotes the absolute recall gain.
Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich context present in the video such as speaking scene and on-screen text. To tackle such CAVSR (AVSR including rich visual Context), we propose VASR designed to "see" and reason the visual context to improve speech recognition. Specifically, we construct an Audio-Visual Chain-of-Thought (AV-CoT) that explicitly enforces intermediate cross-modal grounding between acoustic signals and visual evidence. This evidence-driven reasoning mitigates the "single-modality dominance" problem, where models either over-rely on visual context or fail to utilize it. Besides, to address the data scarcity, we construct and release a corresponding data pipeline and test set. Experiments show that AV-CoT effectively mitigates the single-modality dominance, achieving state-of-the-art performance in CAVSR. The project is open-sourced.
Wenjie Tian, Mingchen Shao, Bingshen Mu +8
Northwestern Polytechnical University, Xi’an, China
We introduce LRS-VoxMM, an in-the-wild benchmark for audio-visual speech recognition (AVSR). The benchmark is derived from VoxMM, a dataset of diverse real-world spoken conversations with human-annotated transcriptions. We select AVSR-suitable samples and preprocess them in an LRS-style format for direct use in existing AVSR pipelines. Compared with commonly used benchmarks, LRS-VoxMM covers a more diverse range of scenarios and acoustic conditions. We also release distorted evaluation sets with additive noise, reverberation, and bandwidth limitation to support evaluation under severe acoustic degradation. Experimental results show that LRS-VoxMM is considerably harder than LRS3 and that the contribution of visual information becomes more evident as the audio signal degrades. LRS-VoxMM supports more realistic AVSR benchmarking and encourages further research on the role of visual information in challenging real-world conditions.
Doyeop Kwak, Jeongsoo Choi, Suyeon Lee +1
Korea Advanced Institute of Science and Technology, South Korea
Contextual ASR can benefit from semantic cues or from target words explicitly provided in the context. We introduce HearInContext, a Mandarin-English benchmark that pairs shared synthetic speech with assistant replies supporting different interpretations. The benchmark comprises 3,764 semantic test cases built around homophones. Implicit contexts exclude candidate words; explicit contexts name the target. No-context and unrelated-context controls measure the benefit of relevant history and sensitivity to irrelevant history. Context-capable models benefit from implicit cues but achieve higher target recall with explicit hints. Fine-tuning Qwen3-ASR-1.7B improves implicit-context target recall by 11.4 percentage points in both Mandarin and English, while absolute CER/WER changes on AISHELL-1 and LibriSpeech remain below 0.1 percentage points. Gains extend to explicit conditions excluded from fine-tuning and to Mandarin hotword recognition on real recordings. Code and data are available at https://github.com/OPPO-Mente-Lab/HearInContext