Typed Decision Models such as Jev, Clef and pplx-decider return a probability for each answer an application provides. Applications use them in many languages, but are compared mostly in English. We introduce XDecideBench, a multilingual benchmark of nine suites in 122 languages. On our twelve-language suite, three open models fine-tuned for typed decisions beat their Qwen base models in every language. On new test data with local base models, their accuracy gain is larger in languages with less web text than in those with more. They fix a similar share of base-model errors in both, so the larger gain may reflect weaker base models. The best model on the twelve-language suite is best on only two other suites. The best English model on each suite, chosen in hindsight, is significantly beaten in about a quarter of suite and language combinations. On Belebele, Decision-1 is best in English but falls far behind Jev in Burmese. A confidence threshold fitted on English to allow 5% errors gives a higher error rate on XNLI, even in German and French. XDecideBench shows that results from one suite or from English are a poor guide to how a model and its threshold behave in other languages.
Figures & tables
Figure 1: The best model in English is confidently wrong on this Burmese item. A real Belebele item, selected after the runs, with the Burmese options glossed in English. Bars show the probabilities of Microsoft’s Decision-1, the best model on English Belebele. Two options that both Decision-1 and Jev put near zero are omitted. Over all Burmese items its Brier skill is .336, against .814 for Jev (§ 6.1 ).
Study
Langs
DMs
Types
Same
P/N
H–B
Transf.
Deußer et al.
200 +
1
CNS
–
–
–
–
yanng981
8
5
C
✓
–
–
–
Convai Innovations
51
1
CNS
–
–
–
–
Tateno
1
23
CNS
✓
–
–
–
Rafe and Das
1
8
CN
✓
–
✓
–
Lu and Wu
1
2
C
✓
–
–
task
Table 1: No prior study covers more than two of the last four columns.
Figure 2: Overview of XDecideBench. (a) What we score (§ 5 ). (b) Headline suites by label regime, and the held-out cohort. A question is one typed decision about a text. Multilingual NLI reuses the 6,000 Core12 XNLI texts, so the suites hold 88,425 distinct texts and 153,875 questions. (c) One tick per covered language, ordered by share of Common Crawl pages (log scale). Languages missing from the crawl use their macrolanguage’s share, and languages with neither are at the right. (d) Research questions (§ 6 ).
Typed Decision Model
Access
Backbone
Precision
Clef
local, API
Qwen3.8-27B †
fp32/4 dp
Clef-flash
local, API
Qwen3.5-9B †
fp32/4 dp
pplx-decider -v1.1
API
Qwen3.8-27B ‡
full
Microsoft- Decision-1
API
Qwen3.5-9B ‡
full
Kev-4B
API
Qwen3.5-4B ‡
4 dp
Laya -multilingual
local
mmBERT-base
full
Table 2: The eleven Typed Decision Models. Three are paired with their own Qwen backbone, and a fourth pair, Decision-1, is in Appendix D.4 .
Figure 3: The best model in English is often beaten in other languages. Belebele Brier skill of Decision-1, which is best in English (line), and of the best model per language, chosen in hindsight (markers). The five languages without a web share are left out. Dark stems: significant paired losses (37 of 105 languages, 35 shown).
Figure 4: Accuracy gains grow as web share falls. Heads against their own backbones per Core12 language, by decreasing web share. (a) Accuracy gain. (b) The gain as a share of the backbone’s errors, which barely changes under the local readout. Solid: local readout. Pale, dotted: hosted readout. Bars: 95% paired intervals. Segments: group means. Shaded: low-web-share group.
Figure 5: English-fitted thresholds let through more errors on XNLI and fewer decisions on MASSIVE. Error rate among admitted decisions (top) and admitted share (bottom) per Core12 language under an English-fitted 5% threshold, for the hosted models we could assess. Luna admits no English XNLI decisions, so it has no XNLI risk. Dotted box: English. Shaded: low-web-share group. Bars: 95% paired source-cluster intervals. Appendix E lists the omitted cells.
Machine translation of MMLU-Pro with expert review
SIB-200
38977a6
CC-BY-SA-4.0
English topic labels propagated through FLORES-200
INCLUDE
d2e1f60
Apache-2.0 (card only)
Native regional exams, not translated
Appendix
Table 4: Several sources have noncommercial or ShareAlike terms. Sources of the evaluation suites and Global PIQA, with pinned revisions (abbreviated), licences as declared upstream and as XDecideBench treats them, and provenance. MultiEURLEX, PAWS-X, XCOPA and MIRACL enter only diagnostic suites. Arabic in Core12 is Modern Standard Arabic and Chinese is simplified script.
Rubric (levels)
Agree
Locale − en
Differ ≥ 2
toxicity (5)
0.457
−0.269
0.167
insult (3)
0.655
+0.076
0.149
bias (3)
0.729
+0.074
0.130
identity attack (3)
0.740
+0.087
0.120
violence (3)
0.864
+0.090
0.058
microaggression (3)
0.896
+0.152
0.048
Appendix
Table 5: Locale labels often differ from the English gold. They differ most on toxicity. Each row pools the 7,400 aligned pairs of a non-English locale and the English item with the same source index (37 locales × 200).
Lang
Test rows
Eligible
Frozen
ar
552
544
500
de
139
138
138
es
550
549
500
fr
419
419
419
hi
547
545
500
ru
552
542
500
Appendix
Table 6: The INCLUDE sample. It caps the items frozen per language and keeps every eligible item where fewer exist. This nine-language subset of the INCLUDE suite serves the native-versus-translated comparison (Appendix C.2 ).
Rule
∣∑p−1∣≤
Models
two decimals
0.01
Jev, Luna
four decimals
0.005
Kev-4B; Clef, Clef-flash (API)
six decimals
10−4
Solar
strict
10−5
pplx-decider, d1, Mercury, Decision-1
fp32
strict
Clef, Clef-flash (local)
Appendix
Table 7: Acceptance tolerances follow each interface’s precision. Accepted vectors are renormalised, and the native choice may trail the rounded maximum by one rounding unit.
Model
Brier skill
Rank
Worst
Gap to en
Cov.
Hosted Typed Decision Models, by Core12 skill
pplx-decider
0.712
[.693, .731]
1–3
sw
0.62
−0.075
1.000
Jev
0.688
[.667, .710]
1–7
sw
0.58
−0.096
1.000
Clef, API
0.674
[.655, .695]
2–5
sw
0.56
−0.103
1.000
d1
0.663
[.642, .682]
2–6
sw
0.55
−0.102
1.000
Decision-1
0.651
[.633, .669]
1–7
sw
0.52
−0.131
1.000
Appendix
Table 8: pplx-decider leads Core12. Every model is weakest in Swahili or Hindi. Per-model Core12 Brier skill against uniform chance, rank interval, worst language, gap to English and coverage. Solar’s 12 MASSIVE cells score as uniform (26-option cap), and without them it scores .497. Laya runs a separate Core12 manifest, and mmBERT has no Belebele or MMLU-ProX head, so neither has a suite score (§ 4 ).
NAT: INCLUDE −
DESC: described − bare
Arm
MMLU-ProX
MASSIVE
SIB-200
Typed Decision Models
pplx-decider
0.052
[ − .005, .109]
0.009
[ − .001, .019]
−0.011
[ − .049, .023]
Jev
−0.082
[ − .136, − .028]
0.043
[.020, .066]
0.000
[ − .028, .025]
Clef, API
0.095
[.045, .144]
−0.021
[ − .037, − .006]
−0.008
[ − .051, .032]
Clef, local
0.094
[.045, .143]
−0.021
[ − .037, − .005]
−0.008
[ − .051, .031]
Appendix
Table 9: The native-exam gap changes sign across arms. Described labels move the skill of Typed Decision Models much less. Change in Brier skill against chance for native against translated exams (NAT) and described against bare labels (DESC).
Native instr.
Fully native
Model
XNLI
MASSIVE
XNLI
MASSIVE
Hosted Typed Decision Models
Jev
+0.009 ∗
−0.001
+0.025 ∗
−0.027 ∗
pplx-decider
−0.002
+0.001
+0.006
−0.024 ∗
d1
−0.001
−0.016 ∗
−0.001
−0.016 ∗
Luna
+0.101 ∗
−0.000
+0.065 ∗
−0.171 ∗
Appendix
Table 10: Native instructions cost the Qwen-based heads most on XNLI. Neutral option IDs cost most on MASSIVE. Change in Brier skill against chance relative to the English-schema Core12 run on the same items, mean over eight languages.
Figure 6: Each head beats its backbone in all Core12 languages and metrics. Head minus backbone for accuracy and backbone minus head for Brier score and ECE under the local readout, so higher is better and 0 is the backbone. Bars show 95% paired source-cluster bootstrap intervals, shading marks the low-web-share group, and open squares mark pplx-decider, the vendor-stated pair.
Th
Tb
Δ ECE raw
Δ ECE scaled
Pair
MASSIVE
XNLI
MASSIVE
XNLI
lowest to
highest
lowest to
highest
Clef
0.97
0.86
0.65
1.24
−0.218 to
−0.118
−0.121 to
−0.019
Clef-flash
0.97
0.78
0.68
0.22
−0.236 to
+0.005
−0.077 to
−0.003
pplx-decider
1.20
0.92
0.65
1.24
−0.214 to
−0.121
−0.130 to
−0.050
Appendix
Table 11: English temperatures shrink the heads’ ECE advantage. Clef-flash loses the most. One temperature per arm and dataset, MASSIVE/XNLI, fitted by negative log-likelihood on the English development half used for temperatures, for each head ( Th ) and its backbone’s local readout ( Tb ). ECE differences are head minus backbone per language, as the mean over MASSIVE and XNLI, ranged over the twelve Core12 languages. On these two datasets raw Clef-flash ECE is higher than its backbone’s in Swahili ( + .005).
Figure 7: Heads keep part of their ECE advantage after English temperature scaling. Clef-flash keeps the least. ECE reduction, backbone minus head, on Core12 MASSIVE and XNLI under the local readout, (a) raw and (b) after applying one temperature per dataset, fitted on English development data, to both the head and its backbone. Bars show 95% paired source-cluster intervals, and shading marks the low-web-share group.
Head
Gain >0
CI >0
Median gain
Median share
ρ with web share ( p )
Slope
Clef
95
28
4.0
0.26
−0.41
( p<.001 )
−1.0
Clef-flash
98
44
7.0
0.31
+0.02
( p=.824 )
+0.3
pplx-decider
103
50
5.5
0.40
−0.54
( p<.001 )
−2.4
Appendix
Table 12: Every head beats its backbone in most Wide-Belebele languages. Only Clef’s and pplx-decider’s gains fall with web share. Wide-Belebele breadth check (exploratory, local readout): each head against its own backbone in 106 languages.
Risk at 5% (admitted %)
90% set: coverage/size
Model
En
High
Low
En
High
Low
XNLI
Jev
0.035 (
63 )
0.098 (
55 )
0.099 (
37 )
0.92 /
1.06
0.81 /
1.07
0.73 /
1.03
pplx-decider
0.045 (
93 )
0.104 (
89 )
0.133 (
82 )
0.91 /
0.96
0.83 /
0.95
0.75 /
0.90
d1
0.047 (
89 )
0.100 (
81 )
0.127 (
67 )
0.92 /
0.99
0.82 /
0.97
0.74 /
0.93
Mercury
0.049 (
85 )
0.096 (
74 )
0.105 (
58 )
0.91 /
1.02
0.82 /
1.03
0.76 /
1.04
Appendix
Table 13: English-fitted policies miss their targets on XNLI outside English. Policies fitted on English development data and applied unchanged on Core12, for every hosted model with English development runs, with a floor of 10−6 . Solar cannot be fitted on MASSIVE, where its option cap leaves no valid English development decision.
Time (UTC, 2026)
Event
10-04 00:56
Statistical protocol frozen, with accuracy as the primary endpoint
10-04 01:23–03:58
Core12 and English development manifests
10-06 20:25
Held-out cohort manifest frozen
10-07 14:42
Described-label manifest
10-07 14:43–10-08 20:41
Model registrations (Laya’s on 10-04), each before that model’s first benchmark request
10-07 14:44–10-08 20:38
Core12 inference
Appendix
Table 14: Protocol, manifests and registrations were fixed before the requests they cover. Chronology from git commits, manifest creation times and run logs. Laya’s Core12 manifest (10-08 08:20) has the same case identifiers and input hashes as the 10-04 manifest, and Laya’s breadth manifests date from 10-08 19:27. Some registrations followed synthetic probe and smoke requests.
Spend (US$)
Tokens (M)
Hosted arm
Bench.
Held-out cohort
Explor.
Pilot
Total
In
Out
Qwen3.8-27B, hosted
40.52
1.43
0.00
0.00
41.95
42.1
0.2
Luna
5.75
0.19
0.72
0.00
6.66
66.6
0.0
Clef
3.34
0.14
2.90
0.00
6.38
85.1
0.0
Jev
3.97
0.16
0.58
0.35
5.06
120.5
26.5
Solar
4.73
0.10
0.09
0.00
4.92
98.5
0.2
Appendix
Table 15: Hosted API spend, tokens and local compute hours. Spend in US dollars by run group, and the input and output tokens the providers reported, in millions (top); local compute hours (bottom).
(a)
Aligned, shared labels
Locale labels
Core12
Belebele
MASSIVE
NLI
RTP-LX
Model
Fail.
Zero
Fail.
Zero
Fail.
Zero
Fail.
Zero
Fail.
Zero
Hosted Typed Decision Models
pplx-decider
0
0.0
0
0.0
0
0.0
0
0.0
0
0.0
Jev
0
86.5
0
75.6
0
100.0
0
72.7
0
93.1
Clef
0
0.0
0
0.0
0
0.0
0
0.0
8
0.0
Appendix
Table 16: Failures and exact zeros per model and suite. Failed items (scored as uniform) and the share of valid answers whose probability vector contains an exact zero: (a) aligned suites; (b) native suites.
Aligned, shared labels
Locale labels
Native-authored
Model
Core12
Belebele
MASSIVE
NLI
RTP-LX
INCLUDE
Injongo
Moder.
BRIGHTER
Hosted Typed Decision Models
pplx-decider
.712
0.756 .756
0.718
.397
0.051
0.536 .536
0.578 .578
.673
0.144 .144
Jev
0.688 .688
.785
0.619
0.128
−0.101
0.531
0.499
0.561 .561
−0.428
Clef
0.674
0.701
0.714
0.254
0.045
0.459
0.492
0.274
.167
d1
0.663
0.711
0.634
0.315 .315
0.070 .070
0.488
0.481
0.449
0.118
Appendix
Table 17: Suite Brier skill under both refusal views. Each suite’s skill against its declared reference with failures scored as uniform, and the constant-predictor baseline.
(a)
Core12
Model
MASSIVE
XNLI
Belebele
MMLU-ProX
SIB-200
Hosted Typed Decision Models
pplx-decider
0.766
[.729, .803]
.663
[.619, .700]
0.887 .887
[.861, .910]
0.450 .450
[.402, .498]
0.794
[.735, .844]
Jev
0.651
[.608, .693]
0.482
[.424, .536]
0.883
[.851, .911]
.584
[.535, .634]
0.839 .839
[.783, .888]
Clef
.776
[.739, .811]
0.572
[.525, .615]
0.853
[.819, .880]
0.363
[.318, .404]
0.808
[.749, .868]
d1
0.712
[.676, .748]
0.601 .601
[.550, .645]
0.856
[.824, .883]
0.358
[.313, .406]
0.786
[.733, .834]
Appendix
Table 18: Per-dataset Brier skill in the multi-dataset suites. (a) The five Core12 datasets; (b) the datasets of the NLI and native moderation (Moder.) suites, each against its own reference.