CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech
Organizations: ELYADATA, Paris, France
Abstract
End-to-end speech Named Entity Recognition (NER) aims to directly extract entities from speech. Prior work has shown that end-to-end (E2E) approaches can outperform cascaded pipelines for English, French, and Chinese, but Arabic remains under-explored due to its morphological complexity, the absence of short vowels, and limited annotated resources. We introduce CV-18 NER, the first publicly available dataset for NER from Arabic speech, created by augmenting the Arabic Common Voice 18 corpus with manual NER annotations following the fine-grained Wojood schema (21 entity types). We benchmark both pipeline systems (ASR + text NER) and E2E models based on Whisper and AraBEST-RQ. E2E systems substantially outperform the best pipeline configuration on the test set, reaching 37.0% CoER (AraBEST-RQ 300M) and 38.0% CVER (Whisper-medium). Further analysis shows that Arabic-specific self-supervised pretraining yields strong ASR performance, while multilingual weak supervision transfers more effectively to joint speech-to-entity learning, and that larger models may be harder to adapt in this low-resource setting. Our dataset and models are publicly released, providing the first open benchmark for end-to-end named entity recognition from Arabic speech https://huggingface.co/datasets/Elyadata/CV18-NER.
Figures & tables
| Split | Train | Dev | Test |
|---|---|---|---|
| # Utterances | 7119 | 2263 | 2325 |
| Total Duration (h:mm) | 8:15 | 2:54 | 2:59 |
| Entity Class | Train | Dev | Test |
|---|---|---|---|
| CARDINAL | 311 | 139 | 138 |
| CURR | 2 | 0 | 0 |
| DATE | 1176 | 516 | 597 |
| EVENT | 112 | 26 | 50 |
| FAC | 112 | 46 | 58 |
| GPE | 541 | 327 | 288 |
| Model | Valid. | Test |
|---|---|---|
| Whisper-medium (ZS) | 107.4 | 112.3 |
| Whisper-medium (FT) | 14.9 | 22.0 |
| Whisper-large-v3 (ZS) | 92.7 | 95.1 |
| Whisper-large-v3 (FT) | 31.8 | 32.3 |
| AraBEST-RQ 300M 6k (FT) | 11.2 | 15.1 |
| AraBEST-RQ 600M 6k (FT) | 12.2 | 17.4 |
| Model | Validation | Test |
|---|---|---|
| CamelBERT-Mix | 79.66 | 77.24 |
| CamelBERT-MSA | 79.76 | 76.13 |
| AraBERTv2 | 81.06 | 76.58 |
| AraBERTv0.2 | 81.10 | 77.34 |
| NER Model | ASR Model | CoER | CVER |
|---|---|---|---|
| CamelBERT-Mix | AraBEST-RQ 600M 14k | 66.4 | 65.8 |
| AraBEST-RQ 300M 6k | 66.1 | 65.4 | |
| AraBEST-RQ 600M 6k | 67.7 | 67.0 | |
| Whisper-large-v3 (FT) | 57.7 | 56.7 | |
| Whisper-medium (FT) | 57.2 | 55.9 | |
| CamelBERT-MSA | AraBEST-RQ 600M 14k | 66.3 | 65.7 |
| Model | WER | CoER | CVER |
|---|---|---|---|
| AraBEST-RQ 300M 6k | 16.0 | 37.0 | 43.5 |
| AraBEST-RQ 600M 6k | 99.4 | 87.1 | 94.0 |
| AraBEST-RQ 600M 14k | 99.6 | 99.8 | 100 |
| Whisper-medium | 20.7 | 38.8 | 38.0 |
| Whisper-large-v3 | 41.9 | 59.6 | 59.1 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Tag | INS | DEL | SUB | CoER (%) |
|---|---|---|---|---|
| B-CARDINAL | 2 | 10 | 31 | 36.8 |
| I-CARDINAL | 1 | 4 | 9 | 93.3 |
| B-DATE | 29 | 43 | 42 | 31.2 |
| I-DATE | 11 | 59 | 46 | 50.4 |
| B-EVENT | 4 | 14 | 10 | 112.0 |
| I-EVENT | 4 | 8 | 14 | 108.3 |
| Tag | INS | DEL | SUB | CoER (%) |
|---|---|---|---|---|
| B-CARDINAL | 4 | 3 | 26 | 28.2 |
| I-CARDINAL | 0 | 5 | 3 | 53.3 |
| B-DATE | 34 | 31 | 41 | 29.0 |
| I-DATE | 17 | 43 | 38 | 42.6 |
| B-EVENT | 8 | 6 | 7 | 84.0 |
| I-EVENT | 5 | 9 | 7 | 84.0 |
Named Entity Recognition
Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.
See all 110 papers