S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments
Organizations: Tohoku University
Abstract
Decision-making in humans and AI involves multiple cognitive abilities. System One decision models produce fast, structured choices. Their cognitive strengths and limitations, however, remain poorly understood. We introduce S1CogBench, a benchmark of 24 established cognitive tasks and 27 evaluation modes, evaluating seven open models and the API model Jev. It combines human-response prediction, autonomous execution, and controlled task manipulations to reveal strengths and failures that aggregate scores miss. We find distinct behavioral profiles across the evaluated models. In these models, strong performance on some learning tasks coexists with failures to gather information or adapt to feedback, while high accuracy does not guarantee human-like responses to changing task demands. Shared readout controls show that model rankings depend partly on how choices are extracted. Separate latency comparisons with generative language models show that long responses are much slower than direct decisions, whereas matched one-token generation adds little overhead. To our knowledge, S1CogBench is the first benchmark of System One decision models on established cognitive experiments. It provides a common empirical basis for identifying their capabilities and limitations before integrating them into automated decision systems.
Figures & tables
| Resource | Cognitive experiments | Predicts human responses | Closed-loop actions | System One models | Decision time or cost |
|---|---|---|---|---|---|
| CogBench ( Coda-Forno et al., 2024 ) | 7 | – | – | – | |
| CogGym, Sept. 2026 v1 ( Ying et al., 2026 ) | 258 | – | – | – | |
| Psych-101 / Centaur ( Binz et al., 2025 ) | 160 | ( ) | – | – | |
| CogBattery ( Hou et al., 2026b ) | 13 | – | – | – | – |
| BanditBench / EVOLvE ( Nie et al., 2025 ) | – | – | – | – | |
| System1-Arena ( Chen et al., 2026 ) | – | – |
| Domain | Tasks | What the experiments vary |
|---|---|---|
| Evidence | 1 Bayesian urn, 2 multiple-cue judgment, 3 multi-attribute choice, 4 probability coherence | Prior probability, likelihood, cue validity, coherence |
| Categories | 5 SHJ, 6 Medin, 7 weather prediction | Category structure, feedback, transfer and typicality |
| History | 8 RLWM, 9 recent probes | Set size, repetition interval, recent interference |
| Adaptation | 10 WCST, 11 restless bandit, 12 instrumental learning | Rule changes, changing rewards, outcome feedback |
| Exploration | 13 Horizon, 14 stationary bandits | Future opportunities, uncertainty |
| Information | 15 Hybrid sampling, 16 Mouselab | Reveal actions, sampling cost, stopping conditions |
| Open models | API models | ||||||||||
| Task | Uniform | Laya | Intern | Llama | Minitaur | Qwen | Nox | CLM | Jev | Haiku | Reference |
| Human-response NLL | |||||||||||
| Multiple-cue judgment | 2.197 | 2.342 | 1.570 | 2.687 | 1.984 | 3.012 | 1.491 | 2.675 | – | 2.029 | |
| Multi-attribute choice | 0.693 | 0.730 | 0.546 | 0.786 | 0.722 | 0.613 | 0.366 | 1.039 | 0.329 | 0.206 | 0.192 |
| WCST | 1.386 | 1.462 | 1.378 | 1.431 | 1.344 | 1.968 | 1.552 | 2.368 | 1.543 | – | 0.840 |
| Horizon | 0.693 | 0.702 | 0.518 | 0.503 | 0.409 | 0.661 | 0.408 | 1.161 | 0.414 | 0.390 | 0.400 |
| Human | Mean reward | Optimal choice (%) | |||||
|---|---|---|---|---|---|---|---|
| Model | NLL | orig. | argmax | sample | orig. | argmax | sample |
| Uniform | 0.693 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 |
| Laya | 0.702 | 51.0 | 51.8 | 51.0 | 54.8 | 57.9 | 54.8 |
| Intern | 0.518 | 54.9 | 54.9 | 52.2 | 78.7 | 78.7 | 62.8 |
| Llama | 0.503 | 52.9 | 53.6 | 52.7 | 66.4 | 70.9 | 65.2 |
| Minitaur | 0.409 | 53.9 | 56.2 | 53.7 | 71.7 | 84.0 | 70.9 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Legal argmax | Legal sampling | |||||
|---|---|---|---|---|---|---|
| Model | Accuracy | Persev. | Set-loss | Accuracy | Persev. | Set-loss |
| Uniform | 25.00 | 17.13 | 17.13 | 25.00 | 17.13 | 17.13 |
| Laya | 25.47 | 15.24 | 18.16 | 25.18 | 16.61 | 17.19 |
| Intern | 27.90 | 21.25 | 15.84 | 26.32 | 17.00 | 16.90 |
| Llama | 32.37 | 35.49 | 10.71 | 31.01 | 23.25 | 15.64 |
| Minitaur | 34.28 | 48.46 | 3.71 | 27.07 | 16.91 | 16.83 |
| Overall | Size 6 minus 2 | Pres. 10 minus 1 | |
|---|---|---|---|
| Source | (%, ) | (pp, ) | size 6 (pp, ) |
| Human | 80.24 | 12.74 [ 14.54, 10.99] | +46.35 [41.13, 51.53] |
| Uniform | 33.33 | +0.00 | +0.00 |
| Laya | 34.85 | 7.17 [ 8.53, 5.79] | +0.21 [ 2.46, 2.69] |
| Intern | 77.83 | 11.37 [ 12.80, 9.96] | +61.49 [58.09, 64.84] |
| Llama | 41.78 | 16.73 [ 17.94, 15.46] | +10.49 [7.27, 13.62] |
| Target | Resampled units | Laya | Intern | Llama | Minitaur | Qwen | Nox | CLM | Jev |
|---|---|---|---|---|---|---|---|---|---|
| Bayesian urn | 200 generated runs | ||||||||
| Multiple-cue judgment | 28 participants | ||||||||
| Multi-attribute choice | 6 participants | ||||||||
| Probability coherence | 15 participants | ||||||||
| SHJ categories | 10 participants | ||||||||
| Medin categories | 42 participants |
| Prediction target | Support and primary score | Aggregation |
|---|---|---|
| Multiple-cue judgment | Original nine response levels, , no epsilon | Response-weighted mean |
| Multi-attribute choice | Two legal product keys, , no epsilon | Response-weighted mean |
| WCST | Four card keys, | Mean of participant means |
| Horizon | H/I keys, | Response-weighted mean |
| Hybrid sampling | Four source action keys, , no epsilon | Pooled held-out actions |
| Mouselab | Currently legal reveals and STOP, , no epsilon | Mean over scored decisions |
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 275 papers