Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized into contrast groups with computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35 public test sets, preserving their original evaluation metrics. We evaluate 40 decision models and 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage. Our results reveal substantial differences obscured by aggregate scores: two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%. Across 2,400 binary contrast pairs, Perplexity Decider correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%, exposing a gap between probability ordering and decision correctness. Models with similar item accuracy also differ in the reliability of their high-confidence predictions. On a separate 1,296-item subset, GLM-5.3 achieves 93.5% contrast-group accuracy at a higher reasoning setting, compared with 81.5% at its lowest effort. By linking errors to pivotal details and difficulty levels, DecisionPivot provides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
system
parameters
type
inference protocol
release status
exposure flags
Perplexity Decider v1.1
27B
decision model
open weights, revision 3b45dea, using the model-card inference protocol
released
16 (2 unknown)
Kev 27B
27B
decision model
open weights, revision af0e6d5, the authors’ code at commit 5e42a7a
released
9
JEV-27B
27B
decision model; model card identifies it as a student of the hosted reference
open weights, revision 51740a8, backbone, adapter and decision head
released
no list
Clef 27B
27B
decision model
open weights, revision 2f3de3d, using the authors’ serving implementation
released
no list
Clef flash
9.5B
decision model
open weights, revision 17f0b0a, using the authors’ serving implementation
released
no list
Decision 2.0 Vega 27B [Decision 2.0 Vega]
27B
decision model, an adapter and decision head on Qwen3.8-27B
open weights, revision 7aec49a with the base at revision 1d4bf0f, using the authors’ decision2 inference package
released
no list
Appendix
Table B1: Evaluated models, inference protocols, and source-exposure flags. The table covers 40 decision models and 14 general models; additional reasoning settings are reported in Table D5 . Open models from other groups use the listed revisions and bf16 precision on H200 GPUs, except for Blink’s FP8 build and the recommended float16 setting for d1-omni-600M. Exposure flags count named reference sources or related data families; for our models, they count tasks with at least 3% detected training overlap before filtering. Jebadiah refers to Kev’s sources without listing them, leaving nine tasks with unknown exposure status. Bracketed names indicate table abbreviations.
task
decision
items
labels
metric (chance)
rule
StartLux
Perplexity
Kev
release
A ambigqa
does the question have several readings
990
human
accuracy (0.500)
0.554
– (0.8%)
–
–
SA
A cap_boundary_card
will the named chat model answer correctly
911
programmatic
skill (0.000)
–
exposed (8.0%)
family
family
R
A cap_boundary_record
the same, with the model’s record on related questions
774
programmatic
skill (0.000)
–
exposed (21.8%)
family
family
R
A freshqa_premise
does the question rest on a false premise
495
human
AUROC (0.500)
–
– (1.0%)
–
–
R
A known_unknown
does the question have a known answer
972
human
accuracy (0.500)
0.580
– (1.7%)
–
family
I
A selfaware
can the question be answered at all
970
human
accuracy (0.500)
0.571
– (1.6%)
–
family
SA
Appendix
Table C2: Reference tasks retain their source labels and evaluation metrics. AUROC denotes area under the receiver operating characteristic curve; nDCG@10 denotes normalized discounted cumulative gain at rank 10.
task
source
license as stated
use
A ambigqa
AmbigQA (AmbigNQ)
CC BY-SA 3.0
SA
A cap_boundary_card
questions of NQ, PopQA, WebQuestions, TriviaQA, BBH, StrategyQA, FOLIO, ProofWriter, DROP, SVAMP, ASDiv, GSM-Hard, GSM-Plus and MAWPS
labels ours, question texts under the licenses of their sources
R, labels from our sampled answers of Gemma 3 4B and 12B, Phi-4-mini and Phi-4
A cap_boundary_record
as above, with the model’s record on related questions
as above
R, as above
A freshqa_premise
FreshQA, sheet of 21 April 2026
Apache-2.0 (repository), no statement on the question sheet
R
A known_unknown
Known-Unknown Questions
MIT
I
A selfaware
SelfAware
CC BY-SA 4.0
SA
Appendix
Table C4: Recorded source terms for the 35 reference tasks. Licence statements follow source cards or license files, with unresolved or source-specific qualifications shown explicitly. I and SA denote the stated item-release routes; R denotes reconstruction-script availability, not an independent statement of redistribution status. Sampling targets at most 1,000 items per task, or 100 queries with 50 candidates for reranking. Final counts follow the filtering described in Section 3.4 . Core items are generated without third-party text and use CC BY 4.0.
task
metric
SL-27B Instant
Perplexity
hosted
JEV-27B
Kev
Clef 27B
A ambigqa
accuracy
35.2
4.0
29.3
26.5
5.7
5.5
A cap_boundary_card
skill
12.3
9.3
4.8
23.2
21.3
− 2.5
A cap_boundary_record
skill
20.8
17.4
23.8
29.7
27.8
6.2
A freshqa_premise
auroc
91.3
91.1
88.2
85.0
84.1
84.4
A known_unknown
accuracy
71.8
72.4
54.9
57.6
70.2
71.2
A selfaware
accuracy
12.4
19.2
17.9
28.2
25.8
31.3
Appendix
Table D4: Chance-corrected score per reference task for six systems (0 = chance, 100 = perfect, negative values kept). Skill: the skill score against success-rate labels. Hosted: the hosted reference.
question
answer
Motivation
Why was the benchmark created?
To diagnose decision correctness, sensitivity to decisive changes, and confidence reliability across seven decision roles in LLM systems. Public evaluations provide useful comparisons but do not alone establish consistent performance across controlled contrasts (Section 1 ).
Who created it?
StartLux Labs, which also develops the StartLux-Decision models (Section 6 ).
Composition
What does an item represent?
A state, a typed question, an option set, and an answer label. Core items have a gold option and record their contrast group, difficulty level, and, where available, variation type. Reference tasks may also use success-rate labels or ranking judgments (Section 3.1 ).
How many items are there?
40,033 items across 48 tasks: 7,800 items from 13 generated core families (600 per family) and 32,233 items from 35 public reference test sets.
Appendix
Table F1: Datasheet for the evaluated DecisionPivot release. Questions follow the categories of Gebru et al. [4] , with answers and references to supporting details in this paper.