Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

Leen AlQadi; Ahmed Alzubaidi; Mohammed Alyafeai; Hamza Alobeidli; Maitha Alhammadi; Shaikha Alsuwaidi; Omar Alkaabi; Basma El Amel Boussaha

cs.CLApr 3, 2026

Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

Authors: Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai, Hamza Alobeidli, Maitha Alhammadi, Shaikha Alsuwaidi, Omar Alkaabi, Basma El Amel Boussaha,

Organizations: Technology Innovation Institute, Abu Dhabi, UAE

Abstract

We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,364 papers

AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic
Sep 28, 2026Ignacio Iacobacci, Faroq Altam, Zhaozhi Qian +1LLM Evaluation Arabic NLP
Elm Company
Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context
Sep 12, 2026Nawar S. Alseelawi, Mustafa S. AljumailyLLM Evaluation Arabic NLP
University of Misan, Amarah, Iraq · Missan Oil Company, Amarah, Iraq
Evaluation of Small Language Models for Arabic Language Processing
Jun 19, 2026Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi +5Arabic NLP Small Language Models
Naseej Innovation Lab, Naseej for Technology, Riyadh, Saudi Arabia.

Sep 28, 2026J/K move · Enter open · S save

AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

Ignacio Iacobacci, Faroq Altam, Zhaozhi Qian, Muhammad Alqurishi

Elm Company

LLM Evaluation Arabic NLP Benchmark Design Factual Knowledge in Language Models Cultural Knowledge in Language Models

Open paper AskPDF arXiv

As Large Language Models (LLMs) continue to scale both in size and capabilities, their proficiency in the Arabic Language has seen significant advancement. However, a critical gap remains: the extent of their factual knowledge and cultural sensitivity to the diverse Arabic-speaking world remains largely underexplored. Current evaluation metrics often focus on translation or generic reasoning, failing to capture the rich historical, social, and regional nuances inherent to Arabic culture. In addition, most benchmarks rely on heavy work, with human intervention in some steps, making the evaluation of knowledge coverage expensive and slow. To address this deficiency, we introduce AraDynFact, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs. Unlike static benchmarks, AraDynFact employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way. We apply AraDynFact to Arabic Wikipedia and audit the performance of several state-of-the-art models, ranging from Arabic-centric specialized LLMs to high-resource general purpose LLMs. In addition we found a high degree of correlation with existing, hand-crafted Arabic-centric benchmarks, confirming the potential of our dynamic approach.

Abstract

LLM Evaluation

Explore similar work

AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context

Evaluation of Small Language Models for Arabic Language Processing