We present the Spoken Wikipedia Presentation Corpus, an extension of the Spoken Wikipedia Corpora featuring LLM-generated slide decks for multimodal ASR. Slides are created from LLM-segmented sections using a hybrid pipeline that combines LLM-based content planning with rule-based design decisions. For each section, an LLM generates a slide title, bullet points, a takeaway message, and a visual description that is used to create an illustration. Rule-based matching then selects layouts, themes, and styles to produce the final slides. A vision LLM extracts slide text as Markdown. We evaluate multiple ASR and spoken language models (SLMs). The best model achieves an average micro-WER of 10.23% and an average micro-CER of 6.48% on audio-only inputs. English yields the lowest error rates, followed by German and Dutch, while performance declines across lower-resource languages. Although audio-only baselines are strong, multimodal zero-shot prompting of omni models remains challenging. The aligned slide, text, and audio data show a strong potential to improve recognition through cross-modal context.
Figures & tables
Fig. 1: Example of a rendered SWPC slide with title, bullet points, takeaway message, and generated illustrative image on the right.
Fig. 2: High-level overview of the SWPC data creation workflow. Spoken Wikipedia audio and article text are processed into aligned utterance–section mappings, which are used to generate slide images, extract slide text, apply quality control, and assemble the final multimodal dataset.
Category
Theme
Layout
Literary analysis
Literature
Editorial cards
Historical context
History
Timeline visual
Person biography
Biography
Portrait focus
Place geography
Geography
Map focus
Science technical
Science
Concept system
Arts culture
Arts
Gallery focus
TABLE I: Rule-based mapping from category to slide theme and layout.
Split
Lang.
Utterances
Articles
Text Seg.
Slides
Speakers
Duration
Train
EN
69,672
4,950
13,636
14,138
363
149:09:36
Train
DE
86,354
6,067
14,823
15,684
281
213:38:15
Train
NL
36,842
3,890
7,094
7,322
57
76:12:42
Train
Total
192,868
14,907
35,553
37,144
701
439:00:33
Dev
EN
7,531
597
1,460
1,526
46
19:01:12
Dev
DE
9,959
712
1,730
1,823
35
27:15:15
TABLE II: SWPC dataset statistics for train, development, test, and total splits. We report utterance, article, text-segment, unique slide, speaker counts, and aligned audio duration.
Model
Lang.
WER μ
WER M
CER μ
CER M
whisper-large-v3
EN
8.14
9.78
5.90
7.04
DE
10.60
12.12
6.25
7.58
NL
17.21
23.36
10.27
13.57
Avg.
10.77
13.81
6.81
8.73
canary-1b-v2
EN
9.85
12.82
6.83
8.86
DE
11.09
13.61
6.38
8.26
TABLE III: SWPC results for baseline ASR and multimodal speech models, sorted by parameter count. Micro/macro WER and CER (%) are reported for English, German, Dutch, and averaged.
Automatic speech recognition (ASR) performs well for high-resource languages with abundant paired audio-transcript data, but its accuracy degrades sharply for most languages due to limited publicly available aligned data. To this end, we introduce WorldSpeech, a 24 kHz multilingual speech corpus comprising 65k hours of aligned audio-transcript data across 76 languages, collected from diverse public sources including parliamentary proceedings, international broadcasts, and public-domain audiobooks. For 37 languages, WorldSpeech provides more than 200 hours of aligned speech, with 28 exceeding 500 hours and 24 surpassing 1k hours. Fine-tuning existing ASR models on WorldSpeech results in an average relative Word-Error-Rate reduction of 63.5% across 11 typologically diverse languages.
Antonis Asonitis, Luca A. Lanzendörfer, Frédéric Berdoz +1
We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.
Large Language Models (LLMs) voice assistants are commonly built as cascaded Automatic Speech recognition (ASR) to LLM systems, where recognition errors can distort user intent. Dislikes may also arise from ambiguous, out-of-domain, or non-request turns, making it hard to isolate ASR effects. We release WASIL (it denotes connection or linking in Arabic): in-the-wild Arabic spoken interaction prompts with audio, ASR hypotheses, assistant responses, and explicit like/dislike feedback (8,529 turns; 14.2% dislikes), plus a 2,000-turn test set covering Modern Standard Arabic (MSA) and four major dialects with their labels. We provide low-cost gold transcripts via multi-ASR agreement-guided post-editing and annotate answerability (answerable, ambiguous/needs-clarification, unsupported, not-a-request/noise) to separate intrinsic unanswerability from ASR-induced degradation. Finally, we describe scalable reference-free evaluation of responses from ASR vs. gold transcripts using multi-judge LLM scoring.