cs.CLOct 31, 2026

S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments

Authors: Dengzhe Hou, Kazunori D Yamada

Organizations: Tohoku University

Abstract

Decision-making in humans and AI involves multiple cognitive abilities. System One decision models produce fast, structured choices. Their cognitive strengths and limitations, however, remain poorly understood. We introduce S1CogBench, a benchmark of 24 established cognitive tasks and 27 evaluation modes, evaluating seven open models and the API model Jev. It combines human-response prediction, autonomous execution, and controlled task manipulations to reveal strengths and failures that aggregate scores miss. We find distinct behavioral profiles across the evaluated models. In these models, strong performance on some learning tasks coexists with failures to gather information or adapt to feedback, while high accuracy does not guarantee human-like responses to changing task demands. Shared readout controls show that model rankings depend partly on how choices are extracted. Separate latency comparisons with generative language models show that long responses are much slower than direct decisions, whereas matched one-token generation adds little overhead. To our knowledge, S1CogBench is the first benchmark of System One decision models on established cognitive experiments. It provides a common empirical basis for identifying their capabilities and limitations before integrating them into automated decision systems.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. System1-Arena: Which Decisions Can Be Left to System 1?

    Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-MakingLLM Evaluation

  2. Benchmarking System One decision models against trained classifiers and language models for automated decision gates

    Sep 29, 2026Amir Rafe, Subasish Dassystem one decision modelsLLM Decision-Making

  3. TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models

    Oct 8, 2026Rahul Sharma, Andrew B. Duncan, Gaétan Marceau Caron +1Typed Decision ModelsLLM Decision-Making