Do AI models designed for fast, structured decisions exhibit the cognitive and social biases observed in human judgment? We evaluate Jev, a model described as a System One decision-maker, using 41,500 responses across cognitive reflection tasks, five classical cognitive-bias paradigms, and a pricing experiment testing ethnic and gender bias. At the level of selected answers, the results are largely encouraging. We confirm previous findings that Jev performs extremely well on cognitive reflection problems. We also observe that it generally avoids the characteristic choice patterns associated with several well-documented human biases, and exhibits only small differences among demographic groups in the tested pricing scenario. However, this apparent robustness conceals important vulnerabilities. Its judgments are sensitive to irrelevant numerical information and the ordering of response options. Moreover, several cognitive-bias manipulations substantially affect returned probabilities even when selected answers remain unchanged. Most strikingly, Jev answers a well-known probability problem correctly in every response but violates the same underlying probability rule in every response to an adapted version. We investigate this discrepancy using 15,000 additional responses across 30 conjunction tasks. Jev violates the rule in 54% of responses, with accuracy varying dramatically across formulations and answer orders. These findings suggest that evaluating decision models solely on familiar benchmarks and selected answers may substantially overstate their reliability in practical applications. They also identify vulnerabilities for developers to address and practical considerations for users integrating such models into automated decision-making systems.
Figures & tables
Task
What the task tests
Cognitive Reflection (4,500)
Whether the correct answer is selected in short numerical and verbal problems that have an intuitive but incorrect response.
Anchoring (5,500)
Whether an irrelevant numerical value influences a judgment that should be based on unrelated information.
Conjunction (1,000)
Whether the conjunction of two events is judged as more likely than one of its constituent events.
Framing (2,000)
Whether equivalent outcomes produce different choices when they are framed as gains rather than losses.
Sunk Cost (2,000)
Whether prior expenditure increases willingness to continue investing in an otherwise unattractive project.
Zero risk (1,500)
Whether complete elimination of one risk affects preferences when alternatives provide equal or greater total harm reduction.
Table 1: Tasks and experimental paradigms included in the evaluation. Counts in parentheses indicate the number of Jev’s responses collected for each task, summed across all versions, conditions, and answer-list orders tested ( N=41,500 overall).
Field
Definition
Primitive
probabilities
Probability distribution over the response options (Choice) or ordered levels (Score)
Both
score
Probability-weighted mean of the level indices: Eq. ( 1 )
Score
choice
The option with the highest returned probability: Eq. ( 3 )
Choice
confidence
Value in [0,1] derived from probabilities : Eq. ( 2 ) (Score), Eq. ( 4 ) (Choice)
Both
Table 2: Main output fields returned by the TypeSafe API for the Score and Choice primitives used in this study TypeSafe (2026) .
Item
Version
Correct Option
Correct
Probability Correct Option (%)
Confidence (%)
Bat and ball
Original
5 pence
500
95.89
94.54
Machines
Original
5 minutes
500
100.00
99.98
Lily pads
Original
47 days
500
100.00
100.00
Race
Original
Second place
500
97.78
97.01
Sheep
Original
8
500
97.39
96.56
Daughters
Original
Emily
500
99.02
98.44
Table 3: Results for the nine cognitive reflection items. Each item has 500 responses, pooled across two response-option orders with 250 responses per answer order. “Correct” is the number of responses that choose the correct option. “Probability Correct Option” and “Confidence” are the mean probability assigned to the correct option and the mean returned confidence across responses, respectively.
Figure 1: Jev’s sentencing judgments by irrelevant numerical anchor value. The same shoplifting case was presented, preceded by information about the sum of two dice, ranging from 2 to 12, with 500 responses per sum. (A) Mean implied sentence in months by dice sum with 95% confidence intervals. (B) Mean probabilities assigned to the sentence answer options by dice sum. (C) Mean reported confidence by dice sum. Implied sentences are probability-weighted scores rather than discrete selections by Jev.
Figure 2: Conjunction judgments in the original and edited tasks, by answer order (250 responses per order condition). (A) Percentage of responses selecting the logically correct option with 95% Wilson confidence intervals. (B) Probability assigned to the correct option. (C) Reported confidence. In panels B and C, dots represent individual responses, boxes show the interquartile range, horizontal lines indicate medians, whiskers show the full range, and diamonds indicate means. Colors distinguish whether the correct or incorrect option was first in the answer list.
Figure 3: Jev’s output by gain/loss framing of the input and answer order in the original and edited versions of the tasks (250 responses per order condition). (A) Percentage of responses selecting the risky option, with 95% Wilson confidence intervals. (B) Probability assigned to the risky option. (C) Reported confidence. Panels B and C use the boxplot conventions of Figure 2 . Colors distinguish whether the certain or risky option was first in the answer list.
Figure 4: Jev’s output by sunk cost condition of the input and answer order in the original and edited versions of the tasks (250 responses per condition). (A) Percentage of responses rejecting investment, with 95% Wilson confidence intervals. (B) Probability assigned to rejecting investment. (C) Reported confidence. Panels B and C use the boxplot conventions of Figure 2 . Colors distinguish whether the option to invest or not to invest was first in the answer list.
Figure 5: Jev’s zero-risk judgements in the original control and treatment conditions and in the edited task, by answer order (250 responses answer order). (A) Percentage of responses ranking the zero-risk alternative last in the original task, or selecting the program producing the larger absolute risk reduction in the edited task, with 95% Wilson confidence intervals. (B) Probability assigned to the corresponding outcome. (C) Reported confidence. In the original version, panel B sums the returned probabilities of the two rankings placing the zero-risk alternative last. Panels B and C use the boxplot conventions of Figure 2 . Colors distinguish the forward and reversed answer orders.
Figure 6: Probability-weighted price summaries by ethnicity and gender in the base (name only) and explicit (name, race, and gender) conditions (125 names, 100 responses per name per condition). Small dots represent individual name means, larger dots show group means, and whiskers indicate 95% bootstrap confidence intervals across names. The dotted vertical line indicates the overall mean across both conditions. Prices are derived by weighting the listed dollar amounts by their returned probabilities and normalizing the totals.
General decision models, such as Jev, have recently emerged as efficient alternatives to LLMs for structured judgment and selection. But what kinds of decisions can these models reliably make, and how does their behavior change when individual decisions are composed into larger systems? To study this, we introduce JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, and evaluate 25 model configurations spanning general decision models and generative LLMs. Our results show that (1) general decision models are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation: they can often identify the most likely outcome while substantially overstating its probability. (2) In more dynamic and realistic systems involving long-horizon, multi-step interactions, the advantages of fast local decision making are offset by reliability failures at the system level. on τ-bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories. (3) In large-scale social simulation, decision models approach strong generative LLMs on individual response prediction at substantially lower inference cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias. Finally, we propose SII-Decision-4B and SII-Decision-27B, which internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation, with SII-Decision-27B performing on par with Jev on JEVal while answering a typical query in about 0.1 s.
Feiyu Duan, Jiayu Lin, Jia Wang +6
Shanghai Innovation Institute · Fudan University · Tongji University +2
An ordinary-looking background detail can turn a correct model decision into a confident mistake. We demonstrate this fragility in four decision systems, including Jev, across seven datasets covering knowledge, reasoning, and tool routing. Within 64 accepted target evaluations per decision, we uncover short context additions that redirect 61.4%-73.2% of each system's initially correct decisions toward a wrong option fixed in advance. The additions supply background or procedural information rather than explicit answer-selection instructions, leaving the original question and choices intact. We construct them through probability-guided context optimization, which uses shifts in the option distribution to refine surrounding text under naturalness and answer-preservation constraints. Redirection affects initially confident decisions, often produces high-confidence wrong choices, and transfers across models. In blinded human evaluation, 91.6% of 250 sampled successful contexts are judged natural, answer-preserving, and free of decisive answer-changing evidence by a majority of three independent annotators. These findings expose a weakness in current decision models: context that looks entirely compatible with an input can redirect the choices that agents, routers, and evaluators rely on.
Typed decision models such as Jev offer an efficient alternative to generative LLMs in decision-making workflows by selecting directly from predefined options. When candidate sets contain no valid answer, TypeSafe recommends including an "other" or "none-of-the-above" option to enable rejection. In this report, however, we identify an arithmetic-dependent rejection bottleneck: Jev reliably selects correct numerical answers when available but frequently accepts incorrect alternatives when they are absent despite an explicit rejection option. On paired arithmetic problems, answer-present accuracy reaches 99%, while correct rejection falls to 7%. Moreover, this gap persists across numerical magnitudes, operation depths, contextual formulations, and rejection labels, and extends to scenarios such as time calculation and capacity rounding. Yet native Boolean verification achieves 99% exact-match accuracy on the same answer-absent arithmetic cases, showing that categorical rejection can fail even when the model successfully verifies candidate correctness. Finally, we show that a simple decision threshold selected on separate development problems raises arithmetic rejection accuracy from 7% to 79% while retaining 97% answer-present accuracy, substantially mitigating the failure without retraining or additional inference.
Jike Zhong, Ming Li, Yuxiang Lai
University of Southern California · University of Florida · Emory University