cs.CLOct 15, 2026
SaveSystem1-Arena: Which Decisions Can Be Left to System 1?
Organizations: The University of Texas at Austin
Abstract
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
Figures & tables
Figure 1: System-1 models combine fast, accurate decisions with well-calibrated probabilities. The strongest System-1 models match fast closed LLMs in accuracy, with better probability estimates and lower latency. Reasoning LLMs achieve the highest accuracy. See Section 4 for evaluation settings.
Figure 3: Task-domain gaps and agentic performance. Left: gaps by task domain in percentage points. Right: agentic task performance. Evaluation settings are given in Section 4 .
Figure 4: Thinking can improve accuracy while worsening Brier scores. Accuracy and Brier score on non-agentic tasks versus the mean number of reasoning tokens per question. The zero-token point disables thinking, as in Table 1 . Other points use reasoning token limits from 128 to 2,048.
Figure 5: Paired comparisons show mixed gains in non-agentic accuracy but consistently lower agentic step accuracy (left); higher cost does not always yield higher accuracy (right). † Base: Qwen3-0.6B-Base. ‡ Same LLM, different interface.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Source | Requests | Tier | License |
|---|---|---|---|---|
| Basic text classification | ||||
| entity_relation | Few-NERD supervised, fine-grained entity types ( Ding et al., 2021 ) | 167 | C | CC BY-SA 4.0 |
| entity_relation | FewRel 1.0, the 80 public Wikidata relations ( Han et al., 2018 ) | 167 | A | MIT |
| entity_relation | SemEval-2010 Task 8, relations between nominals ( Hendrickx et al., 2010 ) | 166 | B | CC BY 3.0 |
| intent | CLINC150, plus variant with out-of-scope queries ( Larson et al., 2019 ) | 250 | C | CC BY 3.0 |
| intent | MASSIVE, en-US ( FitzGerald et al., 2023 ) | 250 | C | CC BY 4.0 |
Table 4: All sources of the static tasks. Tier A keeps the source’s instruction verbatim, B adapts it, C is newly written from the source’s criteria. “(loader)” marks sources whose requests are rebuilt locally by a loader instead of being redistributed.
| Task | Source | Requests | Tier | License |
| logic_puzzles | Ordering puzzles from pairwise clues (generated) | 167 | C | CC BY 4.0 (generated by JevArena) |
| logic_puzzles | ZebraLogic, multiple-choice mode ( Lin et al., 2025 ) | 166 | C | Not stated |
| math_verification | ProcessBench model solutions ( Zheng et al., 2025 ) | 250 | C | Apache-2.0 |
| math_verification | VerifyBench, math problems ( Yan et al., 2026 ) | 250 | C | MIT |
| multihop | 2WikiMultihopQA, comparison and bridge-comparison questions ( Ho et al., 2020 ) | 167 | C | Apache-2.0 |
| multihop | HotpotQA, distractor setting, comparison questions ( Yang et al., 2018 ) | 167 | C | CC BY-SA 4.0 |
Table 4 (continued).
| Task | Source | Requests | Tier | License |
|---|---|---|---|---|
| Retrieval, ranking and matching | ||||
| context_sufficiency | DuoRC ParaphraseRC, test ( Saha et al., 2018 ) | 167 | C | MIT |
| context_sufficiency | IIRC, incomplete information reading comprehension, test ( Ferguson et al., 2020 ) | 166 | B | CC BY 4.0 |
| context_sufficiency | MS MARCO question answering v2.1, dev set ( Bajaj et al., 2018 ) | 166 | A | MS MARCO terms: non-commercial resear… |
| entity_matching | Abt-Buy ( Köpcke et al., 2010 ), DeepMatcher test split ( Mudgal et al., 2018 ) | 167 | B | Not stated (Leipzig entity-resolution… |
| entity_matching | DBLP-Scholar ( Köpcke et al., 2010 ), DeepMatcher test split ( Mudgal et al., 2018 ) | 167 | B | Not stated (Leipzig entity-resolution… |
Table 4 (continued).
| Task | Source | Requests | Tier | License |
| patch_correctness | SWE-bench-extra tasks with SWE-agent final patches from Nebius’s open-weight agent runs and their test results (Golubev et al., 2024) | 250 | C | CC BY 4.0 |
| patch_correctness | SWE-rebench tasks with OpenHands + Qwen3-Coder-480B final patches and their test results ( Badertdinov et al., 2025 ) | 250 | C | CC BY 4.0 |
| static_analysis_triage | D2A leaderboard, Trace + Code task, dev split: Infer warnings on C projects labelled by differential analysis ( Zheng et al., 2021 ) | 250 | B | Apache-2.0 (as listed by the Hugging… |
| static_analysis_triage | OWASP Benchmark for Java v1.2: 2,740 servlets, each a true or a false positive for one vulnerability category (OWASP Foundation) | 250 | C | GPL-2.0 |
| test_outcome | HumanEvalFix from HumanEvalPack, correct and buggy solutions with their unit tests, six languages ( Muennighoff et al., 2024 ) | 250 | C | MIT |
| test_outcome | LiveCodeBench code generation, recorded per-test results of Gemini-Pro-1.5 programs ( Jain et al., 2025 ) | 136 | C | MIT |
Table 4 (continued).
| Task | Source | Requests | Tier | License |
| semantic_ranking | TREC 2022 Clinical Trials track: synthetic patient cases and ClinicalTrials.gov records with NIST judgments ( Roberts et al., 2022 ) | 250 | A | CC BY-SA 4.0 (Hugging Face copy) |
| semantic_ranking | TripAdvisor hotel reviews with the reviewers’ aspect ratings, English reviews of 2012 ( Li et al., 2013 ; processed by Niimi, 2024) | 250 | B | Apache-2.0 (Hugging Face release) |
| LLM-as-judge and evaluators | ||||
| agent_trajectory | AgentRewardBench, expert-annotated web agent trajectories, test split ( Lù et al., 2025 ) | 250 | A | AgentRewardBench terms of use: resear… |
| agent_trajectory | tau-bench historical trajectories of GPT-4o and Claude 3.5 Sonnet (new) in the airline and retail domains ( Yao et al., 2024 ) | 250 | C | MIT |
| atomic_factuality | FActScore human-annotated biographies by InstructGPT, ChatGPT and PerplexityAI ( Min et al., 2023 ) | 250 | B | MIT |
Table 4 (continued).
| Task | Source | Requests | Tier | License |
| injection | AgentDojo tool-use trajectories of GPT-4o, clean and with injections in tool outputs ( Debenedetti et al., 2024 ) | 166 | C | MIT |
| injection | BIPIA indirect prompt injection benchmark: email, table and code contexts with test attacks ( Yi et al., 2025 ) | 166 | C | MIT (BIPIA) |
| injection | ToxicChat 0124, test, human-annotated user prompts with the jailbreaking label ( Lin et al., 2023 ) | 166 | B | CC BY-NC 4.0 |
| pii | Text Anonymization Benchmark (TAB), test split: ECHR court judgments annotated for anonymization ( Pilán et al., 2022 ) | 250 | B | MIT |
| pii | Customer messages, forms and log lines with inserted personal information (generated) | 250 | C | CC BY 4.0 (generated by JevArena) |
| refusal | CoCoNot, human-verified noncompliance test set and contrast set ( Brahman et al., 2024 ) | 167 | B | ODC-BY (repository LICENSE.md) |
Table 4 (continued).
| Task | Source | Requests | Tier | License |
| perspective_labeling | DICES-350: adversarial chatbot conversations rated for safety by all 104 raters of a pool balanced by gender and race ( Aroyo et al., 2023 ) | 167 | B | CC BY 4.0 |
| perspective_labeling | DICES-990: adversarial chatbot conversations, each rated for safety by 60-70 raters balanced by gender and locale (US, India) ( Aroyo et al., 2023 ) | 167 | B | CC BY 4.0 |
| perspective_labeling | Measuring Hate Speech corpus, the 70 reference-set comments rated by 240-815 U.S. crowd workers each ( Kennedy et al., 2020 ; Sachdeva et al., 2022 ) | 166 | A | CC BY 4.0 |
| persuasion_humor | Persuasion dataset: claims with human- and Claude-written arguments, each read by three participants (Durmus et al., 2024) | 125 | B | CC BY-NC-SA 4.0 |
| persuasion_humor | ExPUN: puns and non-puns from SemEval-2017 Task 7, each rated by five annotators ( Sun et al., 2022 ) | 125 | B | CC BY-NC 4.0 (annotations) |
| persuasion_humor | New Yorker Cartoon Caption Contest crowd ratings (Jain et al., 2020; Zhang et al., 2024 ), cartoons described by the human-written annotations of Hessel et al. (2023), ranking test split | 125 | B | CC BY-NC 4.0 (ratings) |
Table 4 (continued).
| Task | Source | Requests | Tier | License |
| position_eval | ChessBench test positions with Stockfish 16 state values ( Ruoss et al., 2024 ) | 167 | B | CC BY 4.0 (ChessBench annotations) |
| position_eval | Connect-4 opening database: all 8-ply positions with their game-theoretic value (John Tromp, UCI repository) | 167 | C | CC BY 4.0 |
| position_eval | Lichess rated standard games of 1 June 2024, first shard of the June 2024 export (lichess.org) | 166 | C | CC0 1.0 |
| puzzles | Minesweeper Beginner positions (9x9, 10 mines) with an exact deduction per square (generated) | 167 | C | CC BY 4.0 (generated by JevArena) |
| puzzles | 8-puzzle positions with a unique best first move (generated) | 167 | C | CC BY 4.0 (generated by JevArena) |
| puzzles | Sudoku grids: rule check and completability (generated) | 166 | C | CC BY 4.0 (generated by JevArena) |
Table 4 (continued).
| Environment | Source | Mode | Episodes | Max dec. | Score | Jev |
| embodied/household_loop | ||||||
| alfworld | ALFWorld ( Shridhar et al., 2020 ), TextWorld version, out-of-distribution evaluation games (valid_unseen) | paused | 24 | 50 | won (1) or not (0) | 0.46 / 0.5 |
| embodied/mujoco_control | ||||||
| half_cheetah | Gymnasium HalfCheetah-v5 (MuJoCo) | paused | 10 | 300 | return | – / -44.9 |
| hopper | Gymnasium Hopper-v5 (MuJoCo) | paused | 10 | 250 | return | – / 306.6 |
| inverted_double_pendulum | Gymnasium InvertedDoublePendulum-v5 (MuJoCo) | paused | 10 | 500 | return | 0.00 / 75.2 |
Table 5: All closed-loop environments. Seeds are 0 to episodes 1; “Jev” is success rate / mean score over all seeds (“–” where success is undefined or runs are pending).
| Environment | Source | Mode | Episodes | Max dec. | Score | Jev |
|---|---|---|---|---|---|---|
| game/rule_puzzles | ||||||
| baba_is_ai/goto_win | Baba Is AI env/goto_win ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
| baba_is_ai/goto_win-distr_obj | Baba Is AI env/goto_win-distr_obj ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.33 / 33.3 |
| baba_is_ai/goto_win-distr_obj-irrelevant_rule | Baba Is AI env/goto_win-distr_obj-irrelevant_rule ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
| baba_is_ai/goto_win-distr_obj_rule | Baba Is AI env/goto_win-distr_obj_rule ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.33 / 33.3 |
| baba_is_ai/goto_win-distr_rule | Baba Is AI env/goto_win-distr_rule ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
Table 5 (continued).
| Environment | Source | Mode | Episodes | Max dec. | Score | Jev |
| baba_is_ai/two_room-goto_win-distr_rule | Baba Is AI env/two_room-goto_win-distr_rule ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.67 / 66.7 |
| baba_is_ai/two_room-goto_win-distr_win_rule | Baba Is AI env/two_room-goto_win-distr_win_rule ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.33 / 33.3 |
| baba_is_ai/two_room-make_wall_win | Baba Is AI env/two_room-make_wall_win ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
| baba_is_ai/two_room-make_win | Baba Is AI env/two_room-make_win ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
| baba_is_ai/two_room-make_win-distr_obj | Baba Is AI env/two_room-make_win-distr_obj ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
| baba_is_ai/two_room-make_win-distr_obj-irrelevant_rule | Baba Is AI env/two_room-make_win-distr_obj-irrelevant_rule ( Cloos et al., 2024 ; baba-is-ai at 40ae1b2), a BALROG task | paused | 3 | 100 | progression (%: 100 if solved) | 0.00 / 0.0 |
Table 5 (continued).
| Environment | Source | Mode | Episodes | Max dec. | Score | Jev |
| jericho_detective | Jericho ( Hausknecht et al., 2019 ) game suite: Detective (Matt Barringer, ported by Stuart Moore, 2000) | paused | 1 | 100 | points | 0.00 / 30.0 |
| jericho_dragon | Jericho ( Hausknecht et al., 2019 ) game suite: Dragon Adventure (William Stott) | paused | 3 | 100 | points | 0.00 / -13.3 |
| jericho_library | Jericho ( Hausknecht et al., 2019 ) game suite: All Quiet on the Library Front (Michael S. Phillips, 1995) | paused | 1 | 100 | points | 0.00 / 0.0 |
| jericho_ludicorp | Jericho ( Hausknecht et al., 2019 ) game suite: The Ludicorp Mystery (Cal Henderson, 2003) | paused | 1 | 100 | points | 0.00 / 5.0 |
| jericho_omniquest | Jericho ( Hausknecht et al., 2019 ) game suite: OMNIQuest (Chris Barden and Chris Ethridge, c. 1988; Inform port) | paused | 1 | 100 | points | 0.00 / 0.0 |
| jericho_pentari | Jericho ( Hausknecht et al., 2019 ) game suite: Pentari (Howard A. Sherman, 1998) | paused | 3 | 100 | points | 0.00 / 0.0 |
Table 5 (continued).
| Environment | Source | Mode | Episodes | Max dec. | Score | Jev |
| miniwob/enter-date | MiniWoB++ enter-date ( Liu et al., 2018 ; miniwob 1.1) | paused | 5 | 6 | reward | 0.00 / -1.0 |
| miniwob/enter-password | MiniWoB++ enter-password ( Liu et al., 2018 ; miniwob 1.1) | paused | 5 | 8 | reward | 0.00 / -1.0 |
| miniwob/form-sequence-2 | MiniWoB++ form-sequence-2 ( Liu et al., 2018 ; miniwob 1.1) | paused | 5 | 8 | reward | 0.00 / -1.0 |
| miniwob/login-user-popup | MiniWoB++ login-user-popup ( Liu et al., 2018 ; miniwob 1.1) | paused | 5 | 12 | reward | 0.00 / -1.0 |
| miniwob/multi-layouts | MiniWoB++ multi-layouts ( Liu et al., 2018 ; miniwob 1.1) | paused | 5 | 10 | reward | 0.00 / -1.0 |
| miniwob/navigate-tree | MiniWoB++ navigate-tree ( Liu et al., 2018 ; miniwob 1.1) | paused | 5 | 10 | reward | 1.00 / 1.0 |
Table 5 (continued).
More in this topic
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 230 papers