SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
Abstract
System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
Figures & tables
| Model | Developer | Size | Open weights | Reasoning | Model | Developer | Size | Open weights | Reasoning |
| GLiDE [ 28 ] | Fastino | 27B | ✓ | No | Decider Flash [ 69 ] | Perplexity | 4B | ✓ | No |
| KnowLine-31B [ 67 ] | PelaAI | 31B | ✓ | No | JevAny-4B [ 86 ] | SimpleJev | 4B | ✓ | No |
| Sieve-31B-v3 [ 76 ] | Sthanika AI | 31B | ✓ | No | SII-4B [ 26 ] | SII | 4B | ✓ | No |
| Decider-27B [ 68 ] | Perplexity | 27B | ✓ | No | Kev-4B [ 65 ] | J. Palmer | 4B | ✓ | No |
| Kev-27B [ 64 ] | J. Palmer | 27B | ✓ | No | Jeva-4B [ 51 ] | NUS | 4B | ✓ | No |
| JevAny-27B [ 86 ] | SimpleJev | 27B | ✓ | No | Laya [ 19 ] | ConvAI Innovations | 0.4B | ✓ | No |
| Sent. | Intent | Math | Code | Science | Know. | Safety | Retrieval | Verification | Tools | Web | |||||||||||||||||||||
| Model | Score | All | E | M | H | E | M | H | V | E | M | H | E | M | H | All | E | M | H | E | M | H | E | M | H | E | M | H | E | M | H |
| Closed-source APIs | |||||||||||||||||||||||||||||||
| GPT-6 Sol | 82.3 | 88 | 100 | 78 | 82 | 100 | 100 | 100 | 100 | 98 | 96 | 93 | 98 | 96 | 91 | 94 | 84 | 82 | 75 | 99 | 91 | 62 | 100 | 99 | 100 | 99 | 88 | 66 | 98 | 76 | 53 |
| GPT-6 Luna | 79.7 | 85 | 99 | 77 | 81 | 100 | 100 | 100 | 97 | 98 | 91 | 86 | 98 | 96 | 88 | 90 | 82 | 80 | 70 | 98 | 91 | 62 | 100 | 98 | 100 | 96 | 86 | 68 | 99 | 77 | 48 |
| Jev API 1.14 | 60.4 | 86 | 99 | 78 | 56 | 100 | 84 | 63 | 30 | 83 | 70 | 59 | 96 | 94 | 71 | 92 | 83 | 73 | 69 | 99 | 83 | 60 | 88 | 88 | 20 | 97 | 75 | 59 | 99 | 73 | 41 |
| OpenAI Decisions | 53.6 | 74 | 99 | 77 | 58 | 74 | 46 | 34 | 30 | 88 | 71 | 66 | 94 | 81 | 46 | 86 | 71 | 67 | 63 | 98 | 84 | 55 | 79 | 75 | 40 | 86 | 61 | 38 | 98 | 68 | 31 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Tasks | Tiers | Format | Sources |
| Semantic understanding | ||||
| Sentiment & Emotion Understanding | 200 | – | choice , score | Amazon Reviews 2023 [ 42 ] , GoEmotions [ 24 ] , SST-5 [ 74 ] , DynaSent [ 70 ] , ARTS [ 88 ] |
| Intent & Routing | 300 | 100/100/100 | choice , noul | CLINC150 [ 48 ] , BANKING77 [ 8 ] , SGD [ 72 ] NatCS [ 35 ] NLU++ [ 9 ] |
| Knowledge and reasoning | ||||
| Math | 200 | 50/50/70 | choice | GSM8K-Platinum [ 18 , 84 ] , MATH500 [ 40 , 54 ] , AIME 2025–2026 [ 4 ] |
| Code Understanding | 200 | 60/70/70 | noul | CodeJudgeBench [ 44 ] , LiveCodeBench [ 43 ] , LeetCodeDataset [ 87 ] |
| Model | Release | Settings |
| GLiDE | fastino/GLiDE @ d199b6d | In process ( Glide.decide ), BF16, SDPA attention, flash-linear-attention |
| KnowLine-31B | PelaAI/KnowLine-31B-Gen1 @ 8f052a8 | SGLang 0.5.21, BF16 with an FP32 output layer, the documented read-twice front end |
| Sieve-31B-v3 | sthanika-ai/Sieve-31B-v3 @ 44aa517 | In process ( load_sieve ), BF16, eager; the released policy, which scores each request in two option orders in one pass |
| Decider-27B | perplexity-ai/pplx-decider-v1.2 @ cce0f3f | Documented fast path (layer compilation, CUDA graphs, length buckets) with FP8 matrix multiplication |
| Kev-27B | jaredpalmer/kev-27b @ af0e6d5 | Documented serving defaults in process: BF16, fused kernels, CUDA graphs |
| JevAny-27B | SimpleJev/JevAny-Qwen3.8-27B-LoRA @ 09c9e91 on Qwen/Qwen3.8-27B @ 1d4bf0f | Adapter merged in BF16, SDPA attention, flash-linear-attention, CUDA graphs |
| Model | Size | Mean | 90th percentile | 99th percentile |
| Laya | 0.4B | 6.0 | 10.2 | 37.2 |
| Kev-4B | 4B | 25.2 | 61.0 | 157 |
| JevAny-4B | 4B | 25.2 | 56.2 | 173 |
| H2O Lightning | 4B | 26.5 | 48.4 | 155 |
| StartLux Instant | 4B | 35.1 | 63.2 | 138 |
| Decider Flash | 4B | 39.8 | 39.2 | 149 |
| Difference from GPT-6 Luna | Backbone | StartLux-4B-AutoThink | ||||||
| Domain | Tasks | Luna | GLiDE | StartLux Instant | gain | Never | Always | Auto |
| Knowledge & Commons. | 200 | 90.0 | +5.0 [+1.5, +9.0] | 1.5 [ 6.0, +3.0] | +4.4 | 85.0 | 88.0 | 87.5 |
| Sentiment & Emotion | 200 | 84.8 | +4.8 [+1.2, +8.5] | +3.0 [ 2.0, +8.0] | +3.4 | 88.0 | 85.8 | 86.0 |
| Retrieval & Reranking | 200 | 76.5 | 5.8 [ 11.2, 0.2] | 8.2 [ 14.5, 1.8] | +3.9 | 69.0 | 70.8 | 71.2 |
| Intent & Routing | 200 | 79.5 | +0.1 [ 2.4, +2.9] | 8.9 [ 12.0, 5.8] | +12.0 | 68.8 | 70.8 | 71.1 |
| Safety | 323 | 75.2 | +4.0 [ 1.2, +9.0] | 7.1 [ 12.7, 1.5] | +16.2 | 60.4 | 63.8 | 63.5 |
| Medium and hard tiers | Hard tier only | |||||
| Domain | Tasks | Backbone | Thinking | Tasks | Backbone | Thinking |
| Knowledge & Commons. | 200 | +4.4 | +3.0 | untiered | ||
| Sentiment & Emotion | 200 | +3.4 | 2.2 | untiered | ||
| Retrieval & Reranking | 200 | +3.9 | +1.8 | 100 | +3.4 | +4.0 |
| Intent & Routing | 200 | +12.0 | +2.0 | 100 | +4.9 | +3.6 |
| Safety | 323 | +16.2 | +3.4 | 158 | +19.8 | +4.4 |