System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
Figures & tables
Figure 3: Overview of SO-Arena. (1) Tasks are drawn or derived from 40 established public sources; items with questionable labels are dropped after human and LLM review. (2) The 2,919 tasks span 11 domains in five capability families, nine of which have domain-relative difficulty tiers, and are posed as static or environment-based tasks. (3) The capability score C averages the hardest tier of each domain with equal domain weights, and the reliability score R combines the option-reordering and missing-correct-option tests.
Figure 4: Composition of the benchmark. (a) Natural tasks by capability family (inner ring) and domain (outer ring). (b) Share of each domain’s tasks in each difficulty tier, with task counts on the bars; side rules mark the families in (a). Tiers are defined within each domain and are not calibrated across domains.
Model
Developer
Size
Open weights
Reasoning
Model
Developer
Size
Open weights
Reasoning
GLiDE [ 28 ]
Fastino
27B
✓
No
Decider Flash [ 69 ]
Perplexity
4B
✓
No
KnowLine-31B [ 67 ]
PelaAI
31B
✓
No
JevAny-4B [ 86 ]
SimpleJev
4B
✓
No
Sieve-31B-v3 [ 76 ]
Sthanika AI
31B
✓
No
SII-4B [ 26 ]
SII
4B
✓
No
Decider-27B [ 68 ]
Perplexity
27B
✓
No
Kev-4B [ 65 ]
J. Palmer
4B
✓
No
Kev-27B [ 64 ]
J. Palmer
27B
✓
No
Jeva-4B [ 51 ]
NUS
4B
✓
No
JevAny-27B [ 86 ]
SimpleJev
27B
✓
No
Laya [ 19 ]
ConvAI Innovations
0.4B
✓
No
Table 1: The 21 evaluated models. Open weights: ✓ available, ✗ closed API. Reasoning: whether the model reasons before answering; auto lets the model decide for each request. Releases and the settings of our latency and reliability runs are listed in Table 4 .
Sent.
Intent
Math
Code
Science
Know.
Safety
Retrieval
Verification
Tools
Web
Model
Score
All
E
M
H
E
M
H
V
E
M
H
E
M
H
All
E
M
H
E
M
H
E
M
H
E
M
H
E
M
H
Closed-source APIs
GPT-6 Sol
82.3
88
100
78
82
100
100
100
100
98
96
93
98
96
91
94
84
82
75
99
91
62
100
99
100
99
88
66
98
76
53
GPT-6 Luna
79.7
85
99
77
81
100
100
100
97
98
91
86
98
96
88
90
82
80
70
98
91
62
100
98
100
96
86
68
99
77
48
Jev API 1.14
60.4
86
99
78
56
100
84
63
30
83
70
59
96
94
71
92
83
73
69
99
83
60
88
88
20
97
75
59
99
73
41
OpenAI Decisions
53.6
74
99
77
58
74
46
34
30
88
71
66
94
81
46
86
71
67
63
98
84
55
79
75
40
86
61
38
98
68
31
Table 2: Scores by domain and difficulty tier. E, M, H, and V denote the easy, medium, hard, and very hard tiers; All marks untiered domains. Score is the leaderboard score ( Equation 4 ). Rows are grouped by access and size and sorted by Score within each group; the two System Two references are shaded orange. Darker is higher; bold is the best in each column. Tiers are not calibrated across domains.
Figure 5: Performance against latency. Performance is given by the leaderboard score ( Equation 4 ). The Pareto frontier is formed by the System One models with optimal performance at each latency. Open-weight models run on one H100 GPU each, one task at a time. Closed APIs are timed end to end over the network ( Appendix C ).
Figure 6: Reliability and its two components for the 18 System One models. (a) Reliability R ( Equation 7 ), with each model’s performance (Perf.), its leaderboard score ( Equation 4 ), to the left of its bar. (b) N against S ; 0 is chance level for both, and dashed curves mark constant R .
Figure 7: The gap to a System Two model, and what narrows it, differ by domain. (a) Score difference of two System One models from GPT-6 Luna on the medium and hard tiers, with 95% confidence intervals ( Appendix D ). (b) Gain from the smaller to the 27B model of each family. (c) Gain of StartLux Auto from always thinking over never thinking. Rows are grouped by the three categories of Section 4.2.1 .
Figure 8: Order stability rises with performance; null discrimination follows it only loosely. In (a) and (c), each point is one of the 18 System One models, the line is a quadratic trend, and the band holds 90% of the fitted models ( Appendix D ). (a) Order stability S ( Equation 5 ) against performance. (b) Share of option reorderings that change the answer, for originally right and wrong answers: median and interquartile range over the models. (c) Null discrimination N ( Equation 6 ) against performance.
Table 3: Composition of SO-Arena. Counts are natural tasks. Tiers give easy/medium/hard counts; Math additionally has 30 very hard AIME problems, and “–” marks domains without tiers. Where tiers draw on different sources, sources are listed from easy to hard, separated by → . “Seq.” denotes a sequence of choice requests.
Model
Release
Settings
GLiDE
fastino/GLiDE @ d199b6d
In process ( Glide.decide ), BF16, SDPA attention, flash-linear-attention
KnowLine-31B
PelaAI/KnowLine-31B-Gen1 @ 8f052a8
SGLang 0.5.21, BF16 with an FP32 output layer, the documented read-twice front end
Sieve-31B-v3
sthanika-ai/Sieve-31B-v3 @ 44aa517
In process ( load_sieve ), BF16, eager; the released policy, which scores each request in two option orders in one pass
Decider-27B
perplexity-ai/pplx-decider-v1.2 @ cce0f3f
Documented fast path (layer compilation, CUDA graphs, length buckets) with FP8 matrix multiplication
Kev-27B
jaredpalmer/kev-27b @ af0e6d5
Documented serving defaults in process: BF16, fused kernels, CUDA graphs
JevAny-27B
SimpleJev/JevAny-Qwen3.8-27B-LoRA @ 09c9e91 on Qwen/Qwen3.8-27B @ 1d4bf0f
Adapter merged in BF16, SDPA attention, flash-linear-attention, CUDA graphs
Appendix
Table 4: Releases and settings. Open-weight models: the Hugging Face repository at the revision used, and the inference path of our latency and reliability runs, which follows each developer’s documented fast path. APIs: the model identifier and settings of the evaluation runs.
Model
Size
Mean
90th percentile
99th percentile
Laya
0.4B
6.0
10.2
37.2
Kev-4B
4B
25.2
61.0
157
JevAny-4B
4B
25.2
56.2
173
H2O Lightning
4B
26.5
48.4
155
StartLux Instant
4B
35.1
63.2
138
Decider Flash
4B
39.8
39.2
149
Appendix
Table 5: Latency per task (ms) on the 1,109 timed tasks: mean, 90th and 99th percentiles over tasks. Jeva-4B is timed with a warm and a cold option cache.
Difference from GPT-6 Luna
Backbone
StartLux-4B-AutoThink
Domain
Tasks
Luna
GLiDE
StartLux Instant
gain
Never
Always
Auto
Knowledge & Commons.
200
90.0
+5.0 [+1.5, +9.0]
− 1.5 [ − 6.0, +3.0]
+4.4
85.0
88.0
87.5
Sentiment & Emotion
200
84.8
+4.8 [+1.2, +8.5]
+3.0 [ − 2.0, +8.0]
+3.4
88.0
85.8
86.0
Retrieval & Reranking
200
76.5
− 5.8 [ − 11.2, − 0.2]
− 8.2 [ − 14.5, − 1.8]
+3.9
69.0
70.8
71.2
Intent & Routing
200
79.5
+0.1 [ − 2.4, +2.9]
− 8.9 [ − 12.0, − 5.8]
+12.0
68.8
70.8
71.1
Safety
323
75.2
+4.0 [ − 1.2, +9.0]
− 7.1 [ − 12.7, − 1.5]
+16.2
60.4
63.8
63.5
Appendix
Table 6: Per-domain values behind Figure 7 . Tasks: medium and hard tiers, or all tasks of an untiered domain. Luna: GPT-6 Luna’s score. Differences from Luna with 95% bootstrap intervals over tasks. Backbone gain: mean over the four families. StartLux-4B-AutoThink: its scores in the three thinking modes.
Medium and hard tiers
Hard tier only
Domain
Tasks
Backbone
Thinking
Tasks
Backbone
Thinking
Knowledge & Commons.
200
+4.4
+3.0
untiered
Sentiment & Emotion
200
+3.4
− 2.2
untiered
Retrieval & Reranking
200
+3.9
+1.8
100
+3.4
+4.0
Intent & Routing
200
+12.0
+2.0
100
+4.9
+3.6
Safety
323
+16.2
+3.4
158
+19.8
+4.4
Appendix
Table 7: Backbone and thinking gains on the medium and hard tiers and on the hard tier alone. Gains as in Figure 7 b and c (thinking: always minus never ). The untiered domains are identical in both selections.