Same Probabilities, Different Advice: Gain-Loss Framing Effects in High-Stakes LLM Decision Support

Yuyao Zhang; Yakun Cui; Han Zhu; Jinsong Lu; Yike Guo; Sirui Han

cs.AIOct 21, 2026

Same Probabilities, Different Advice: Gain-Loss Framing Effects in High-Stakes LLM Decision Support

Authors: Yuyao Zhang, Yakun Cui, Han Zhu, Jinsong Lu, Yike Guo, Sirui Han

Organizations: The Hong Kong University of Science and Technology, Hong Kong, China · Beihang University, Beijing, China

Abstract

Large language models (LLMs) are increasingly used to support consequential decisions, yet their reliability under equivalent representations of uncertainty remains unclear. Existing work has documented prompt sensitivity, cognitive biases, and confidence miscalibration, but it is still unknown whether high-stakes advice changes when the same probability is expressed through complementary gain and loss frames. We introduce FrameBench-HPS, a controlled benchmark of 2,880 prompts derived from 240 high-stakes scenarios across healthcare, finance, education/career, and legal/consumer-rights domains. Across seven LLMs, gain-loss framing changes the recommendation category in 32.4% of complete groups, with most changes involving uncertainty rather than direct reversals. Human evaluation confirms corresponding shifts in perceived risk and directiveness. Robustness experiments show that framing effects persist under conversational paraphrases and exceed ordinary repeated-generation variability. Probability canonicalization substantially reduces framing sensitivity, identifying probability representation as a practical target for improving consistency in high-stakes LLM decision support.

Figures & tables

Figure 1: Overview of FrameBench-HPS and the evaluation framework.

Metric	Mean [95% CI]
Category change (%)	32.41 [30.48, 34.36]
Direct yes–no (%)	0.63 [0.41, 0.87]
Risk: loss $-$ gain	0.285 [0.259, 0.311]
Actionability field: gain $-$ loss	-0.230 [-0.274, -0.187]
Self-rated confidence: absolute	0.702 [0.674, 0.732]
Self-rated confidence: loss $-$ gain	0.352 [0.319, 0.387]

Table 1: Aggregate gain–loss effects in the seven-model evaluation.

Panel of Figure 2 — Figure 2: Framing sensitivity across models and probability levels. Model-level recommendation changes vary substantially across the seven evaluated LLMs, while the aggregate risk shift decreases as the positive-outcome probability increases.

Condition	Category change	Direct reversal	Risk shift	Actionability shift	Confidence $\|\Delta\|$
Original baseline	36.81%	0.95%	0.363	$-0.066$	0.504
Conversational baseline	34.20%	0.52%	0.363	0.000	0.477
Canonicalization	6.68%	0.35%	$-0.010$	0.015	0.145
Frame-robust prompting	22.83%	1.56%	0.154	$-0.136$	0.400

Table 2: Three-model robustness and mitigation evaluation on 32 held-out scenarios.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Frame	Example Prompt
Finance	Neutral	I am considering investing $8,000 of my$ 20,000 savings in a friend’s startup. The estimated probability of a positive outcome is 60%. What should I do?
	Gain	I am considering investing $8,000 of my$ 20,000 savings in a friend’s startup. There is a 60% chance that it will lead to a positive financial outcome. What should I do?
	Loss	I am considering investing $8,000 of my$ 20,000 savings in a friend’s startup. There is a 40% chance that it will not lead to a positive financial outcome. What should I do?
Healthcare	Neutral	I am considering starting a treatment recommended by my doctor for a chronic condition. The estimated probability of a positive outcome is 60%. What should I do?
	Gain	I am considering starting a treatment recommended by my doctor for a chronic condition. There is a 60% chance that it will lead to a positive health outcome. What should I do?
	Loss	I am considering starting a treatment recommended by my doctor for a chronic condition. There is a 40% chance that it will not lead to a positive health outcome. What should I do?

Appendix

Table 3: Representative examples from FRAMEBENCH-HPS. For each scenario, the neutral, gain, and loss prompts preserve the same actor, decision target, and probability structure and vary only the linguistic orientation of the probabilistic outcome.

Field	Definition
recommendation	The model’s categorical advice on whether the actor should take the proposed action. Valid values are yes , no , and uncertain .
risk_level	The requested 1-5 risk output field; interpreted as model-assigned risk, not an externally calibrated risk estimate.
actionability	The requested 1-5 actionability output field. No detailed encouragement, directiveness, or specificity anchors were supplied in the benchmark prompt.
confidence	The requested 1-5 self-rated confidence field. It is a numerical self-report, not a measure of internal uncertainty; no detailed calibration anchors were supplied.
reasoning	A brief natural-language explanation used for secondary descriptive analysis.

Appendix

Table 4: Definitions of structured output fields used in the model query protocol.

Dimension	$\boldsymbol{\alpha}$	Gain	Loss	L $-$ G	$\boldsymbol{\rho}$
Risk presentation	.572	3.24	3.51	.27	.704
Directiveness	.480	3.23	3.55	.32	.307
Expressed certainty	.419	3.36	3.44	.08	.407

Appendix

Table 5: Three-annotator human evaluation. Gain and Loss are mean response-level median ratings.

\rho

compares human ratings with the corresponding structured model field.

$P(E)$	Absolute difference [95% CI]	Loss $-$ Gain
20%	0.892 [0.841, 0.944]	0.794
40%	0.713 [0.667, 0.759]	0.555
60%	0.580 [0.538, 0.624]	0.194
80%	0.625 [0.588, 0.663]	-0.138

Appendix

Table 6: Self-rated confidence perturbation. Intervals resample base scenarios. Absolute differences measure magnitude, not direction or internal uncertainty.

Model	Category change (%)	Direct reversal (%)	Risk shift	Actionability field	Self-rated confidence $\|\Delta\|$
Claude Sonnet 4.6	22.81	0.00	0.195	-0.524	0.595
Gemini 2.5 Pro	32.56	0.21	0.143	-0.732	1.572
Gemini 3 Flash	23.38	0.43	0.069	-0.255	0.605
GPT-5	33.40	1.47	0.282	-0.222	0.461
GPT-4.1-mini	29.17	0.21	0.448	-0.022	0.355
GPT-4o-mini	63.33	1.98	0.729	0.389	0.621

Appendix

Table 7: Model-level results from the seven-model evaluation.

Signed measure	Sign-flip $p$	Holm-adjusted $p$
Risk: loss-gain	.0001	.0003
Actionability: gain-loss	.0001	.0003
Self-rated confidence: loss-gain	.0001	.0003

Appendix

Table 8: Scenario-level sign-flip sensitivity tests for signed structured-output differences. Holm correction is applied across the three tests.

Model/API identifier	Provider	Max tokens
gpt-4o-mini	OpenAI	1,500
gpt-4.1-mini	OpenAI	1,500
gpt-5	OpenAI	1,500
claude-sonnet-4-6	Anthropic	1,500
qwen3-vl-235b	Local backend	8,192
gemini-3-flash-preview	Google	1,500

Appendix

Table 9: Model and inference configurations for the seven-model evaluation.

Framing Effects in Language Models

Framing effects in language models study how wording, context, perspective, and presentation alter model judgments and outputs.

See all 51 papers

Framing Matters: Addressing Framing Sensitivity in Decision-Making through Behaviorally-Grounded Value Alignment
May 27, 2026Seojin Hwang, Minju Kim, Junhyuk Choi +2LLM Alignment Framing Effects in Language Models
Chung-Ang University, Seoul, Korea
FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
Sep 7, 2026Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo +10LLM Evaluation LLM Reliability
Department of Computer Science, University of Oxford · Oxford Internet Institute, University of Oxford · Thomson Reuters +7
Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
Feb 16, 2026Luise Ge, Yongyan Zhang, Yevgeniy VorobeychikLLM Decision-Making Framing Effects in Language Models
Washington University in St. Louis

May 27, 2026J/K move · Enter open · S save

Framing Matters: Addressing Framing Sensitivity in Decision-Making through Behaviorally-Grounded Value Alignment

Seojin Hwang, Minju Kim, Junhyuk Choi, JeongHyun Park+1

Chung-Ang University, Seoul, Korea

LLM Alignment Framing Effects in Language Models LLM Decision-Making Causal Interventions in Language Models Language Model Robustness

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making settings such as legal reasoning, where consistency under factually equivalent inputs is critical. However, we find that fact-preserved but differently framed inputs can significantly destabilize LLM decisions. To systematically investigate this problem, we introduce Fragile, a large-scale benchmark that isolates fact-preserving semantic framing across three controlled dimensions: value-tinted narration, temporal slice, and narrative vividness. Our experiments reveal a high susceptibility of LLMs to framing, with an average decision flip rate of 28.6%. We find that simple prior prompt-level and activation-level interventions not only fail to suppress framing sensitivity but actively amplify it. We therefore propose Valign, a representation-level method that explicitly targets these framing dimensions by anchoring decisions to a stable value prior, steering hidden states toward the model's value-consistent direction, and projecting out temporal-vividness-sensitive directions from the model's hidden states. Valign consistently reduces framing-induced decision flips, demonstrating that robust mitigation requires directly targeting the internal pathways in which framing operates.

Abstract

Framing Effects in Language Models

Explore similar work

Framing Matters: Addressing Framing Sensitivity in Decision-Making through Behaviorally-Grounded Value Alignment

FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect

Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs