Training Language Models To Be Coherent Decision-Makers

Khurram Yamin; Xavier Fernandes; Paul Koch; Bryan Wilder; Eric Horvitz

cs.AIOct 6, 2026

Training Language Models To Be Coherent Decision-Makers

Authors: Khurram Yamin, Xavier Fernandes, Paul Koch, Bryan Wilder, Eric Horvitz

Organizations: Carnegie Mellon University · Microsoft

Abstract

Reliable decision-making requires more than accurate prediction: a model must preserve its beliefs, apply the relevant utilities, and recognize when the information needed to justify an action is missing. We study whether language models can learn this decision procedure from supervised fine-tuning and generalize it across domains and differing natural-language expressions of the decision challenge. Across 20 datasets, we explore challenges of belief instability and decision-making errors by first eliciting probabilities of outcomes and then varying only the utilities and the framing of the decision problems, while holding the evidence fixed. We train models to preserve elicited beliefs while selecting the action that maximizes expected utility, and evaluate transfer to unseen application domains, held-out framings, and different classes of payoff structures. We further introduce incomplete-information settings in which required utilities are withheld and replaced with irrelevant text, testing whether models can distinguish missing decision-relevant information from merely additional context. We find that targeted fine-tuning substantially improves coherent decision-making and that in many situations, learning transfers across domains and framings to situations unobserved during training. Further, models trained for decidability learn to identify when action cannot be justified based on missing information. Finally, we show the value of a routed system that considers separately the recognition of decision completeness and utility-sensitive decision execution.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the benchmark and training pipeline. (a) We first fine-tune a complete-information solver to preserve its elicited belief and apply the utility-implied decision rule across eight equivalent framings of each case–utility pair. (b) We then study decision decidability under incomplete utility information. The decidability model identifies which information condition is present, reports a belief, withholds action for incomplete cases, and routes complete decision problems to the specialist solver.

Panel of Figure 3 — Figure 3: Complete-information results for Direct SFT (top row of each panel) and CoT SFT (bottom row), pooled over the eight framing holdouts: (a) belief-conditional decision error (Equation 6 ); (b) excess belief distortion error (Equation 5 ). The three left bars split test prompts by framing status (seen, held-out cross-class, held-out full-class) and the two right bars by domain status. Dashed lines are the same base model without SFT on the same prompts, reweighted to the test set’s domain mix. Error bars are 95% bootstrap intervals.

Figure 4: Decidability results by model size (solid: Gemma 4; dashed: Qwen3.5). Rows: (a) information-choice accuracy over all four conditions, (b) belief-conditional decision error on true-D items, (c) end-to-end joint accuracy, and (d) excess belief distortion error over all items that report a belief, against a floor measured on the decidability prompts. The routed system’s information accuracy is its gate’s (a), its decisions are the complete-information CoT specialist’s (b), and its floor is the route-weighted mixture of its components’ floors (d). Error bars are 95% bootstrap intervals, often smaller than the markers.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Domain	Role sentence opening each prompt	Event $E$
Training domains (15): train, validation, and in-domain test splits
Credit-card default risk	You are a credit-risk analyst at a bank reviewing a customer’s credit-card account to judge repayment risk.	the cardholder defaults on their next payment
Loan credit risk	You are a loan officer at a retail bank reviewing a new loan application.	the applicant turns out to be a bad credit risk
Corporate bankruptcy	You are a corporate credit analyst assessing whether a company is financially distressed from its accounting ratios.	the company goes bankrupt within the forecast horizon
Term-deposit marketing	You are a marketing analyst at a retail bank judging whether a client will subscribe to a term deposit if called.	the client subscribes to a term deposit if contacted
Income-support targeting	You are an analyst screening census records to identify lower-income individuals for a support program.	the person earns at most $50K a year (low income)

Appendix

Table 1: Decision domains, role sentences, and events. Ten cases are sampled per domain, giving 200 evidence contexts.

Explore similar work

Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow
Sep 30, 2026Polina Tsvilodub, Andreas Waldis, Linlu Qiu +2Large Language Model Fine-Tuning Model Fine-Tuning
University of Tübingen · Massachusetts Institute of Technology · New York University
Diagnosing and Improving Probabilistic Reasoning in Large Language Models
Sep 29, 2026Huaman Sun, Dingcheng Wang, Jason Hartline +1LLM Reasoning Strategies Probabilistic Model
Department of Computer Science Northwestern University Evanston, IL 60201, USA
Learning Transferable Latent User Preferences for Human-Aligned Decision Making
May 12, 2026Alina Hyk, Sandhya SaisubramanianLarge Language Model Decisions Preference Alignment Learning
Oregon State University

Sep 30, 2026, cs.CLJ/K move · Enter open · S save

Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow

Polina Tsvilodub, Andreas Waldis, Linlu Qiu, Tal Linzen+1

University of Tübingen · Massachusetts Institute of Technology · New York University

Large Language Model Fine-Tuning Model Fine-Tuning Bayesian Bayesian Inference Journey

Open paper AskPDF arXiv

Language models (LMs) are increasingly used for tasks that require reasoning about hidden variables from a few observations, for which Bayesian inference is the normatively correct solution. While supervised fine-tuning of an LM on the outputs of an optimal

\textit{Bayesian}

model leads to near-Bayesian behavior, standard supervised fine-tuning (SFT) on the true answers for the task falls short of it. But behavior alone does not tell us

\textit{why}

tuning on a

\textit{Bayesian}

or an

\textit{oracle}

(true answers) signal differs: whether the resulting LM represents Bayesian beliefs, acts on them, or turns them into a choice the way Bayes' rule does. To compare them, we formulate increasingly demanding requirements for an LM to count as a Bayesian decision maker, spanning its behavior, representations, and computations, and test them on a flight recommendation task. The Bayes-trained LM acts Bayesian, encodes quantities of Bayes' rule in its middle layers, and uses the encoded belief for the recommendation to a certain extent. The oracle-trained LM differs from it both in the beliefs it holds and whether it reads beliefs out into recommendations. Exchanging beliefs between the LMs transfers a part of the Bayesian advantage. Bayes fine-tuning thus installs usable Bayesian beliefs in an LM for reasoning under uncertainty in a way standard SFT on oracle answers cannot, highlighting the advantage of nuanced supervision.

Abstract

Explore similar work

Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

Learning Transferable Latent User Preferences for Human-Aligned Decision Making