Diagnosing and Improving Probabilistic Reasoning in Large Language Models

Huaman Sun; Dingcheng Wang; Jason Hartline; Jessica Hullman

cs.AISep 29, 2026

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

Authors: Huaman Sun, Dingcheng Wang, Jason Hartline, Jessica Hullman

Organizations: Department of Computer Science Northwestern University Evanston, IL 60201, USA

Abstract

Large language models (LLMs) are increasingly proposed as decision assistants who must reason probabilistically from available evidence under explicit decision costs. We propose a decision-theoretic framework that decomposes LLMs' decision loss into two components: forming accurate beliefs from provided evidence and translating those beliefs into actions that optimize a provided utility function. Using a synthetic benchmark with known ground truth, we apply the decomposition to characterize probabilistic reasoning in frontier and open-sourced models. We further evaluate whether RL interventions targeting beliefs, decisions, or both improve these components across three domains, whether improvements transfer across components and elicitation formats, and whether decision performance can improve without improvement in belief formation. We find that targeting one component of probabilistic reasoning redistributes decision loss, improving the target without necessarily transferring to others, and that jointly targeting belief formation and decision-making improves both but hinges on matched formats between training and evaluation.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Format	$\Delta$ R_total	$\Delta$ R_belief	$\Delta$ R_opt
B	independent	$-0.044\pm 0.014$	$-0.057\pm 0.014$	$0.013\pm 0.020$
D	independent	$-0.031\pm 0.012$	$-0.014\pm 0.015$	$-0.017\pm 0.019$
B/D	independent	$-0.053\pm 0.015$	$-0.057\pm 0.014$	$0.004\pm 0.020$
B+A	independent	$-0.059\pm 0.013$	$-0.057\pm 0.014$	$-0.002\pm 0.020$
B	two-turn	$-0.046\pm 0.011$	$-0.057\pm 0.014$	$0.011\pm 0.009$
D	two-turn	$-0.009\pm 0.011$	$-0.014\pm 0.015$	$0.005\pm 0.010$

Appendix

Table 1: Synthetic. Change in regret relative to Base (intervention

-

Base): mean difference with 95% confidence interval (

n=200

cases). Negative values indicate lower regret than Base.

Strategy	Format	$\Delta$ R_total	$\Delta$ R_belief	$\Delta$ R_opt
B	independent	$-0.024\pm 0.010$	$-0.125\pm 0.014$	$0.101\pm 0.015$
D	independent	$-0.184\pm 0.019$	$-0.020\pm 0.006$	$-0.165\pm 0.019$
B/D	independent	$-0.159\pm 0.019$	$-0.136\pm 0.017$	$-0.023\pm 0.018$
B+A	independent	$-0.008\pm 0.006$	$-0.136\pm 0.017$	$0.128\pm 0.018$
B	two-turn	$-0.111\pm 0.015$	$-0.125\pm 0.014$	$0.014\pm 0.006$
D	two-turn	$-0.032\pm 0.007$	$-0.020\pm 0.006$	$-0.013\pm 0.004$

Appendix

Table 2: HailFinder. Change in regret relative to Base (intervention

-

Base): mean difference with 95% confidence interval (

n=500

cases). Negative values indicate lower regret than Base.

Strategy	Format	$\Delta$ R_total	$\Delta$ R_belief	$\Delta$ R_opt
B	independent	$-0.025\pm 0.017$	$-0.079\pm 0.013$	$0.054\pm 0.022$
D	independent	$-0.180\pm 0.023$	$-0.035\pm 0.010$	$-0.145\pm 0.024$
B/D	independent	$-0.153\pm 0.023$	$-0.069\pm 0.013$	$-0.084\pm 0.025$
B+A	independent	$-0.042\pm 0.014$	$-0.076\pm 0.013$	$0.034\pm 0.019$
B	two-turn	$-0.037\pm 0.015$	$-0.079\pm 0.013$	$0.042\pm 0.016$
D	two-turn	$-0.073\pm 0.013$	$-0.035\pm 0.010$	$-0.038\pm 0.010$

Appendix

Table 3: SimSUM. Change in regret relative to Base (intervention

-

Base): mean difference with 95% confidence interval (

n=500

cases). Negative values indicate lower regret than Base.

Strategy	Format	Trained thresholds	Untrained thresholds	Gap (untrained $-$ trained)
B	independent	$-0.037^{*}\ [-0.053,\,-0.021]$	$-0.033^{*}\ [-0.049,\,-0.018]$	$0.004\ [-0.012,\,0.020]$
D	independent	$0.001\ [-0.013,\,0.015]$	$-0.010\ [-0.024,\,0.005]$	$-0.010\ [-0.028,\,0.007]$
B/D	independent	$-0.027^{*}\ [-0.046,\,-0.008]$	$-0.025^{*}\ [-0.045,\,-0.005]$	$0.002\ [-0.015,\,0.021]$
B+A	independent	$-0.067^{*}\ [-0.083,\,-0.052]$	$-0.070^{*}\ [-0.086,\,-0.054]$	$-0.003\ [-0.018,\,0.013]$
B	two-turn	$-0.040^{*}\ [-0.051,\,-0.030]$	$-0.048^{*}\ [-0.060,\,-0.037]$	$-0.008^{*}\ [-0.012,\,-0.004]$
D	two-turn	$-0.006\ [-0.016,\,0.003]$	$-0.008\ [-0.018,\,0.002]$	$-0.002\ [-0.006,\,0.003]$

Appendix

Table 4: Change in total regret relative to Base on the Synthetic task, evaluated separately at thresholds seen during training (

0.2,0.4,0.6,0.8

) and held-out thresholds (

0.1,0.3,0.5,0.7,0.9

). The gap is the held-out value minus the trained-threshold value. Entries are means over 200 cases with 95% paired bootstrap intervals; ∗ marks intervals excluding zero.

Explore similar work

How reliable are LLMs when it comes to playing dice?
Jun 5, 2026Luca Avena, Gianmarco Bet, Bernardo BusoniLLM Reasoning Strategies Probabilistic Model
Università degli Studi di Firenze Dipartimento di Matematica e Informatica ‘Ulisse Dini’
Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1LLM Reasoning Strategies Reasoning Skills
University of Southern California · 2DEVCOM ARL
LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs
May 7, 2026Chacha Chen, Matthew Jörke, Adam Goliński +4LLM Reasoning Strategies Bayesian
1Apple · 2Stanford University · †Work done while an intern at Apple. +1

Jun 5, 2026, cs.CLJ/K move · Enter open · S save

How reliable are LLMs when it comes to playing dice?

Luca Avena, Gianmarco Bet, Bernardo Busoni

Università degli Studi di Firenze Dipartimento di Matematica e Informatica ‘Ulisse Dini’

LLM Reasoning Strategies Probabilistic Model Chain-of-Thought Reasoning

Open paper AskPDF arXiv

We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems. We constructed two datasets, respectively a set of standard exercises and a set of counterintuitive exercises, designed to trigger heuristic reasoning, and evaluated 8 state-of-the-art models, each tested with and without Chain-of-Thought prompting. Models achieve an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones. We further provide empirical evidence of token bias: performance drops by over 20% when canonical formulations are replaced by disguised variants. Embedding misleading suggestions in the prompt reduces performance by up to 34%, with no model proving immune. Taken together, the reported findings suggest that current LLMs are not yet genuine probabilistic reasoners, despite their success in advanced mathematical problems.

Abstract

Explore similar work

How reliable are LLMs when it comes to playing dice?

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs