cs.SDMay 9, 2026

WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

Authors: Zien Sheikh AliHamdy MubarakSoon-Gyo JungHunzalah Hassan BhattiFiroj AlamShammur Absar Chowdhury

Organizations: Qatar Computing Research Institute, Qatar

Abstract

Large Language Models (LLMs) voice assistants are commonly built as cascaded Automatic Speech recognition (ASR) to LLM systems, where recognition errors can distort user intent. Dislikes may also arise from ambiguous, out-of-domain, or non-request turns, making it hard to isolate ASR effects. We release WASIL (it denotes connection or linking in Arabic): in-the-wild Arabic spoken interaction prompts with audio, ASR hypotheses, assistant responses, and explicit like/dislike feedback (8,529 turns; 14.2% dislikes), plus a 2,000-turn test set covering Modern Standard Arabic (MSA) and four major dialects with their labels. We provide low-cost gold transcripts via multi-ASR agreement-guided post-editing and annotate answerability (answerable, ambiguous/needs-clarification, unsupported, not-a-request/noise) to separate intrinsic unanswerability from ASR-induced degradation. Finally, we describe scalable reference-free evaluation of responses from ASR vs. gold transcripts using multi-judge LLM scoring.

Explore similar work

CardsList
  1. Evaluation of Automatic Speech Recognition Using Generative Large Language Models

    Apr 23, 2026Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil +6