Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models
Organizations: University of Misan, Amarah, Iraq · Missan Oil Company, Amarah, Iraq
Abstract
Fairness benchmarks for large language models (LLMs) are overwhelmingly English-born and American in their social categories; translated versions carry those categories into societies where they do not apply, and almost none fix their hypotheses before the models are run. We present Qistas (Arabic qistas, "the just balance"), a protocol for evaluating how LLMs treat the social, confessional, and regional constituencies of a society, portable across countries and instantiated first for Iraq. In its pilot form it measures only what needs no verdict on contested truth: procedural symmetry across matched counterfactual pairs, differential refusal, stereotype reliance under ambiguity, and terminological framing of documented events. Every measurement is paired by construction and analysed with paired machinery (sign-flip permutation tests, exact McNemar tests, template-clustered bootstrap intervals, Holm-corrected families), with a power analysis that states, before any data exist, the smallest effect each axis can detect after the design effect of template clustering. We release the authoring apparatus (a pruned 3x5 taxonomy with four cells closed by design, per-type checklists, a cumulative rulings log, a structural audit, and a human review notebook) and report its first field test: a semantic review of a 58-item seed bank in which 24 of 25 template groups failed at least one check. The defect patterns it exposed (evaluation criteria leaking into prompts, token-level swaps that break Arabic agreement, epistemically unbalanced pairs, single-theme refusal probes, unwritten dialect twins counted as coverage) are, we argue, general hazards of counterfactual fairness benchmarks. This is a protocol paper, not a results paper: no model has been evaluated, and the accompanying pre-registration commits the hypotheses, metrics, exclusion rules, and analysis plan that the pilot will follow.
Figures & tables
| Domain | F1 Asymmetry | F2 Differential refusal | F3 Stereotype reliance | F4 Framing / language | F5 Register gap (factorial) |
|---|---|---|---|---|---|
| Social-regional (D1) | active (T1) | active (T3) | active (T2) | active (T4) | active |
| Confessional-communal (D2) | active (T1) | active (T3) | closed | closed | active |
| Historical-political (D3) | active (T1, facts) | closed | closed | active (T4 sample) | active |
| Axis | Planned size | Statistic | Minimum detectable effect |
|---|---|---|---|
| T1 asymmetry | 100 pairs (200 items) | paired , two-sided | unclustered; with the design effect |
| T1, one domain | 40 pairs | paired , two-sided | ; with the design effect |
| T2 stereotype reliance | 60 ambiguous items | exact binomial, one-sided | share to among committed answers, depending on commitment rate (§ 5.3 ) |
| T3 differential refusal | 50 pairs (100 items) | exact McNemar, two-sided | split among 15 discordant pairs |
| T3 at the earlier plan | 30 pairs (60 items) | exact McNemar, two-sided | split among 9 discordant pairs |
| T4 framing | 40–50 items | descriptive | intervals and coder agreement only |
| Discordant pairs | Discordance at 50 pairs | Discordance at 30 pairs | Minimum split (power 0.80) |
|---|---|---|---|
| 6 | 12% | 20% | 96:4 (near-total one-sidedness) |
| 9 | 18% | 30% | 91:9 |
| 12 | 24% | 40% | 87:13 |
| 15 | 30% | 50% | 84:16 |
| 20 | 40% | - | 80:20 |
| 25 | 50% | - | 77:23 |
| R-001 | Framing items in the historical-political domain carry the verifiable fact reference category: the category describes the anchor event, which must be registry-documented, not the coding route. |
|---|---|
| R-002 | The manifestation-form field records the form the test machine measures, not the item’s theme: a counterfactual pair about a stereotype is still an asymmetry item. |
| R-003 | On the gender axis (and any axis with grammatical agreement), the swapped value is the whole agreeing phrase - verb, predicate, pronouns - never a single token. |
| R-004 | In T2 twins, the decisive sentence replaces the “no further information” sentence; it is scenario-specific with an evidence source that fits (a witness, exam results, a receipt log). |
| R-005 | A prompt may not name the criterion being measured (balance, symmetry, neutrality, equal depth, “present both sides”, respect): anything that reminds the model of the measurement contaminates it. |
| R-006 | Values of a knowledge request must be epistemically comparable: do not ask of one community what it keeps undisclosed by self-definition while asking the same of a community that publishes it; redirect to what is public for all. |
| R-007 | A governorate pair that crosses the Arab-Kurdish regional boundary bundles ethnicity with region and is declared as the ethno-regional axis; the pure governorate axis is built from pairs within the same regional space. |
| Defect family | Groups | Checks | Fix class |
|---|---|---|---|
| Dialect twins present as bracketed stubs (counted as coverage) | 5 | 5 | authoring |
| Two “alternative phrasings” of a factual question that are textually identical (one variable, generative form, and natural standing all fail) | 4 | 12 | authoring |
| Unbounded response length in a generative request | 3 | 3 | textual |
| Evaluation criterion named in the prompt (R-005) | 3 | 3 | textual |
| Refusal pairs with placeholder group values and a single theme | 3 | 3 | authoring |
| T2 decisive sentence left as a generic template; twin contradicts its own “no information” sentence (R-004) | 3 | 3 | textual |
Language Model Bias Evaluation
Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.
See all 528 papers