Oct 12, 2026
LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a calibrated probability for each option in a single forward pass, without text generation. We show that open instruction-tuned LLMs can achieve performance comparable to Jev without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Next, we propose a simple supervised LoRA fine-tuning recipe that can improve their performance further. We evaluate four instruction-tuned LLMs: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench maintainers, 2026) and which computes a Capability score based on accuracy and calibration. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After fine-tuning, it scores 89.0, outperforming Jev by 8.9 Capability points and on par with the strongest open decision model we evaluated. The same recipe improves the three other models by 7.8 to 15.7 points. On a suite of public decision datasets, our models achieve accuracy and calibration comparable to Jev. Our results show that open LLMs already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, training and serving code, OpenDecisions, and DecisionBank, our training dataset consisting of 261,561 decisions.
Jonathan Geuter, Andrew Cheng, David Alvarez-Melis
Harvard University · Kempner Institute