cs.CLNov 5, 2026

Beyond the Keyword: A Human-Annotated Benchmark for Arabic Anxiety-Related Discourse

Authors: Shimaa Ibrahim, Mabrouka Bessghaier, Md. Rafiul Biswas, Insaf Elouard, Wajdi Zaghouani

Organizations: Northwestern University in Qatar, Doha, Qatar · Hamad Bin Khalifa University, Doha, Qatar

Abstract

Keyword retrieval returns expressions of personal anxiety together with many other uses of anxiety-related language. We present a benchmark of 1,000 Arabic Twitter/X posts retrieved with the keyword qalaq ("anxiety/worry") and annotated for six discourse functions. Two native Arabic-speaking annotators labeled every post independently, reaching 51.1% raw agreement (Cohen's κ=0.378κ= 0.378). We then reviewed all 489 disagreements and all 511 agreements under the same guidelines. The review changed 120 of the agreed labels (23.5%), which shows that agreements need auditing as well as disagreements. The final labels form a consensus reference; on a random sample of 200 posts, an additional annotator agreed with it on 60.5% of posts (κ=0.517κ= 0.517). Against this reference, Fanar-C-2-27B reaches 79.8% accuracy and macro-F1 0.784 (κ=0.749κ= 0.749). Its errors concentrate in the residual category (41.1% of errors; precision 0.905, recall 0.447), whose definition is internally inconsistent in the executed prompt, so the low recall cannot be attributed to the model alone. Sociopolitical Anxiety (27.1%) outnumbers Personal Emotional Expression (22.0%); these are assignment proportions in the sample, not estimates of prevalence or retrieval precision. We release the guidelines, a residual-subtype scheme, a fixed data split, label provenance, and evaluation code.

Explore similar work

CardsList
  1. AraCrypto-ML: A Multi-Label Corpus of Communicative Functions in Arabic Facebook Cryptocurrency Discourse

    Nov 11, 2026Mabrouka Bessghaier, Mohamed Rashad Rouine, Md. Rafiul Biswas +2Social Media AnalysisArabic NLP

  2. Understanding the Sociocultural Dimensions of Mental Health Discourse in Arabic-Language X Communities

    Jun 6, 2026Amal Alqahtani, Rana Salama, Mona DiabSocial Media AnalysisMental Health