Organizations: Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Sankt Augustin, Germany · University of Bonn, Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
How does Jev compare with other systems on Arabic tasks, and how does its performance change between Modern Standard Arabic (MSA) and dialects? We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. The Arabic-focused frozen suite includes English controls in three parallel resources, which we also remove in a separate descriptive comparison. Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. Removing the recorded English controls, repeated cross-task items or both leaves the full system order unchanged. Relative performance and gains between Jev versions vary across task families. In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. Paired contrasts with Jev 1.13.0, Gemma-4-31B and Jais-2-8B show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.
Figures & tables
ID
Family
Tasks
Coverage
A
Varieties and script
4
Reading and topic decisions across language varieties and scripts
B
Native dialect text
3
Dialect identification, sentiment, sarcasm, and stance
C
Label ladders
5
Tasks grouped around coarse-to-fine labels, including sentiment and intent
D
Application tasks
12
Intent, inference, relevance, similarity, moderation, and safety
E
Knowledge
2
Multiple-choice questions across subject areas
F
News and claims
6
Subjectivity, propaganda, claims, authorship, and truthfulness
Table 1: Task families in the evaluation suite. Counts refer to tasks in our evaluation, not the number of labels, domains, or questions within each source dataset. Family membership organizes reporting; each research question uses its own specified tasks and conditions.
Figure 1: Evaluation flow from a shared task representation through distinct model interfaces to label-based scoring. The Arabic sentiment example and English gloss are self-authored illustrations; no benchmark item, model prediction or measured probability is shown. Actual requests use frozen task wording, and supported questions and coverage depend on the system and condition.
Group
Systems
Scope
Hosted decision models
4
Jev versions and the Microsoft and OpenAI decision services
Arabic decision models
5
Arabic-focused releases in the pinned comparison
Other general decision models
5
Previously selected open decision-model releases
Arabic-centric LLMs
9
Jais, Jais 2, ALLaM and Fanar
Qwen LLMs
6
Qwen3.5 sizes and the Qwen3.8 comparison model
Gemma LLMs
5
Gemma 4 releases at the pinned sizes
Table 2: Active systems grouped for presentation. Counts refer to evaluated versions or checkpoints, so related releases are counted separately. Arabic-centric and general-use labels follow the pinned study inventory. Group membership does not imply identical inference interfaces or coverage.
System
Score
Records/s
Arabic-only
Tasks
Coverage (%)
Gemma-4-31B
71.81
7.86
71.75
43/43
100.00
h2o-lightning-31b
71.12
7.11
71.07
43/43
100.00
jev-1.14.0
69.24
4.93
69.18
43/43
100.00
OpenAI-Decisions-gpt-6-luna
68.79
3.57
68.72
43/43
99.84
jev-1.13.0
68.77
5.22
68.71
43/43
100.00
SII-Decision-27B
68.39
2.06
68.31
43/43
100.00
Table 3: Fixed-group summary ordered by unrounded score. Scores are mean task-primary metrics multiplied by 100. Records/s uses recorded processing time under each backend’s execution settings ( section A.4 ). Arabic-only removes English controls; tasks and coverage describe the frozen evaluation population.
Figure 2: Task-family means multiplied by 100 for the fixed summary population, ordered by overall mean. Families: A, varieties/script (including English controls); B, native dialect; C, label ladders; D, applications; E, knowledge; F, news/claims; G, commonsense/culture; H, rating/grading; I, Arabic language. Colors use a fixed 0–100 scale. Family means use available task scores; tasks and coverage summarize the evaluation population. Numerical validation is described in section C.3 .
System
MSA (%)
Dialect (%)
Gap [95% interval]
Belebele : 4,500 pairs; 488 clusters
Jev 1.13.0
95.44
89.33
-6.11 [-7.24, -4.97]
Jev 1.14.0
95.56
89.87
-5.69 [-6.81, -4.57]
Gemma-4-31B
94.44
89.56
-4.89 [-6.13, -3.65]
Jais-2-8B
82.11
73.20
-8.91 [-10.90, -6.95]
SIB-200 : 8,032 pairs; 393 clusters
Table 4: Matched accuracy for the fixed RQ2 model panel. Gaps are dialect minus MSA in percentage points, with pointwise 95% whole-cluster percentile-bootstrap intervals. Pairs count source–variety comparisons; repeated MSA sources and paraphrases remain linked in resampling. ArBanking77 uses source-level means of paraphrase correctness. All four models use identical eligible items.
Figure 3: Jev 1.14.0’s matched dialect-minus-MSA accuracy gap for every retained dataset–dialect cell. Bars are pointwise 95% source/article-clustered intervals. The zero line denotes equal matched accuracy, not a tested equivalence margin. The main table pools these cells within each dataset; it does not pool datasets.
Dataset
vs. Jev 1.13
vs. Gemma
vs. Jais
Belebele
+0.42
-0.80
+3.22
[-1.02, +1.86]
[-2.82, +1.22]
[-0.22, +6.66]
SIB-200
+0.21
+0.45
+0.81
[-1.15, +1.57]
[-1.11, +2.00]
[-1.36, +2.98]
DialectalArabicMMLU
+0.95
-0.19
+0.21
[-0.33, +2.23]
[-1.87, +1.50]
[-1.97, +2.39]
Table 5: Pooled gap contrasts: Jev 1.14.0 minus each comparator, in percentage points. The second line gives approximate simultaneous 95% bands accounting for all 69 dialect-specific and pooled contrasts across four datasets. Positive values mean a less negative gap for Jev. Gemma is Gemma-4-31B; Jais is Jais-2-8B. Bands spanning zero do not establish equivalence.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Task
Types
Metric
Items
Source
A
Belebele
C
Acc.
7,200
[ 8 ]
A
SIB-200
C
Acc.
11,044
[ 16 ]
A
DialectalArabicMMLU
C
Acc.
21,945
[ 4 ]
A
TUNIZI
C
Acc.
3,000
[ 17 ]
B
NADI 2021
C2
Macro-F1
5,000
[ 7 ]
B
ArSarcasm-v2
N, C2
Macro-F1
3,000
[ 9 ]
Appendix
Table 6: Task inventory and source attribution. Types are Choice (C), Noul (N) and Score (S); trailing digits count questions. Items gives the frozen task size, not independent texts or completed answers. Full mappings and revisions are in the companion files.
System
Declared precision
Readout
Hosted decision models
jev-1.13.0
Not recorded
API
jev-1.14.0
Not recorded
API
Microsoft-Decision-1
Not recorded
API
OpenAI-Decisions-gpt-6-luna
Not recorded
API
Arabic decision models
Appendix
Table 7: Evaluated systems and declared primary readouts. API: hosted interface; TC: restricted token codes; NH: native heads; OI: own interface; LP: letter codes with its own prompt. Full repository identifiers,revisions and declared precision are in systems.csv.
Template
Frozen instruction and criterion descriptions
T-EN
Instruction: What is the overall sentiment of ‘tweet‘? negative: The text expresses a negative opinion. neutral: The text expresses neither a positive nor a negative opinion. positive: The text expresses a positive opinion.
Appendix
Table 8: Actual frozen ArSarcasm-v2 sentiment question under the primary T-EN template. Internal label keys are preserved; no tweet, predicted answer or full model-specific chat serialization is shown. Exact source strings for the complete main suite are retained in the accompanying local template bundle.
System
Score
Records/s
Tasks
Coverage (%)
Gemma-4-31B
71.81
7.86
43/43
100.00
h2o-lightning-31b
71.12
7.11
43/43
100.00
d3
70.78
4.05
43/43
100.00
GLiDE-1.2
70.36
4.23
43/43
100.00
pplx-decider-v1.2
69.90
4.88
43/43
100.00
jev-1.14.0
69.24
4.93
43/43
100.00
Appendix
Table 9: All systems ordered by the unrounded evaluation mean. Scores are mean task-primary metrics multiplied by 100. Records/s uses recorded processing time under each backend’s execution settings ( section A.4 ). Tasks and coverage describe the scored population. Masked means are in the companion sensitivities.csv.
Family
Jev 1.14
Jev 1.13
Gemma 31B
H2O 31B
A Varieties and script
83.00
82.89
84.24
84.13
B Native dialect text
48.14
48.08
53.22
52.56
C Label ladders
52.03
51.16
55.15
55.33
D Application tasks
74.08
73.41
77.87
78.74
E Knowledge
81.95
82.47
80.19
80.80
F News and claims
70.57
69.47
68.65
60.14
Appendix
Table 10: Task-family means multiplied by 100. Gemma 31B is Gemma-4-31B; H2O 31B is h2o-lightning-31b. All four systems have full primary-item coverage. Family composition and primary metrics differ.
System
Corrected mean
Corrected shared-task mean
Gemma-4-31B
71.81
72.92
h2o-lightning-31b
71.12
71.93
jev-1.14.0
69.24
70.69
OpenAI-Decisions-gpt-6-luna
68.79
69.93
jev-1.13.0
68.77
70.21
SII-Decision-27B
68.39
69.71
Appendix
Table 11: Available-task and common-task means after the completeness correction, multiplied by 100. The common column retains the same 40 tasks; item support can still differ within them.
Dialect
System
Pairs
MSA (%)
Dialect (%)
Gap [95% interval]
Belebele
Egyptian
Jev 1.13.0
900
95.44
93.11
-2.33 [-3.68, -1.01]
Egyptian
Jev 1.14.0
900
95.56
92.78
-2.78 [-4.30, -1.32]
Egyptian
Gemma-4-31B
900
94.44
92.00
-2.44 [-4.01, -0.89]
Egyptian
Jais-2-8B
900
82.11
75.11
-7.00 [-9.64, -4.43]
Najdi
Jev 1.13.0
900
95.44
90.89
-4.56 [-6.19, -2.99]
Appendix
Table 12: Per-dialect matched accuracy and pointwise 95% intervals. Gaps are dialect minus MSA in percentage points; all four models use identical items.
Dialect
Comparator
Contrast
Simultaneous band
Belebele
Egyptian
Jev 1.13.0
-0.44
[-2.22, +1.33]
Egyptian
Gemma-4-31B
-0.33
[-3.13, +2.47]
Egyptian
Jais-2-8B
+4.22
[-0.32, +8.76]
Najdi
Jev 1.13.0
+0.33
[-1.61, +2.28]
Najdi
Gemma-4-31B
-1.11
[-4.05, +1.82]
Appendix
Table 13: Jev 1.14.0 minus comparator dialect–MSA gap, by dialect. Bands account for the complete 69-contrast grid. Positive values mean a less negative gap for Jev.
Dataset
Removed
Retained
Full gap
Nonidentical gap [95% interval]
Belebele
0
4,500
-5.69
-5.69 [-6.78, -4.58]
SIB-200
795
7,237
-0.35
-0.39 [-1.14, +0.40]
DialectalArabicMMLU
120
15,555
-3.63
-3.66 [-4.35, -2.96]
ArBanking77 (Palestinian)
0
3,073
-7.94
-7.94 [-9.26, -6.59]
Appendix
Table 14: Jev 1.14.0 after removal of identical full-input pairs. Gaps and pointwise 95% intervals are in percentage points; remaining pairs still retain source/article clustering. This is a selected nonidentical-input population, not an estimate of performance on all dialect text.
Reporting
AraTox items
Cov. (%)
AraTox mean
Overall mean
Original saved export
300
100.00
61.65
68.78
Complete primary items
298
99.33
62.00
68.79
Appendix
Table 15: Original and corrected OpenAI Decisions reporting. Scores are multiplied by 100. Original coverage is the saved export’s value; the corrected row requires all primary heads.
Dataset
Original saved interval
Recomputed interval
Belebele
[-6.73, -4.56]
[-6.82, -4.63]
SIB-200
[-1.02, +0.35]
[-1.02, +0.36]
DialectalArabicMMLU
[-4.29, -2.93]
[-4.35, -2.93]
ArBanking77 (Palestinian)
[-9.27, -6.61]
[-9.27, -6.61]
Appendix
Table 16: Historical original and first corrected Jev 1.14.0 interval endpoints, in percentage points. The expanded main-text panel uses more draws and different fixed seeds; matched populations and point estimates agree.