Probability Elicitation from Language Models

Latest papers

10 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates
Oct 20, 2026An-Sheng JhangProbability Elicitation from Language Models Belief Updating in LLMs
Dongguan University of Technology
Verbalized and Internal Probabilities Are Coupled in Large Language Models
Sep 30, 2026Sinead Williamson, Jiaxuan Li, Nick Foti +2Confidence Estimation in Language Models LLM Uncertainty Estimation
Apple
Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
Sep 27, 2026Han Chen, Yingrui LiReasoning Consistency in Language Models LLM Reliability
Independent Researcher
PromptNCE: Conditional Probabilities and PMI Using Only LLMs and Contrastive Estimation Prompts
May 20, 2026Juliette Woodrow, Chris PiechLLM Prompting Conditional Distribution Estimation
Department of Computer Science Stanford University
Language-Induced Priors for Domain Adaptation
May 14, 2026Qiyuan Chen, Jiayu Zhou, Raed Al KontarMulti-Source Domain Adaptation Domain Adaptation
Industrial and Operations Engineering University of Michigan Ann Arbor, MI 48109 · School of Information University of Michigan Ann Arbor, MI 48109
Eliciting associations between clinical variables from LLMs via comparison questions across populations
May 7, 2026Fabian Kabus, Kian Kordtomeikel, Thomas Brox +3LLM Interpretability Causal Structure Learning
Institute of Medical Biometry and Statistics (IMBI), Medical Center, University of Freiburg · Department of Computer Science, Faculty of Engineering, University of Freiburg · Department of Neurology and Neurophysiology, Medical Center, University of Freiburg +1
Bayesian Elicitation with LLMs: Model Size Helps, Extra "Reasoning" Doesn't Always
Apr 2, 2026Luka Hobor, Mario Brcic, Mihael Kovac +1Probability Elicitation from Language Models LLM Uncertainty Estimation
University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia
Hidden Topics: Measuring Sensitive AI Beliefs with List Experiments
Feb 25, 2026Maxim ChupilkinProbability Elicitation from Language Models LLM Alignment
Large Language Model probabilities cannot reliably distinguish between possible and impossible language
Sep 18, 2025Evelina Leivada, Raquel Montero, Paolo Morosi +5LLM Evaluation Language Model Generation Evaluation
Departament de Filologia Catalana, Centre de Lingüística Teòrica, Universitat Autònoma de Barcelona, Barcelona 08193, Spain · Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, 08010, Spain · Similarweb, Spain +1
Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization
May 21, 2025Aliakbar Nafar, Kristen Brent Venable, Zijun Cui +1Bayesian Networks Probabilistic Graphical Models
Michigan State University · Florida Institute for Human and Machine Cognition (IHMC) University of West Florida

Oct 20, 2026J/K move · Enter open · S save

Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates

An-Sheng Jhang

Dongguan University of Technology

Probability Elicitation from Language Models Belief Updating in LLMs Framing Effects in Language Models LLM Evaluation LLM Uncertainty Estimation Language Model Probing

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as judgment assistants and as stand-in participants in psychology experiments, yet little is known about how much their probability estimates move in response to a single new piece of information. We pair premise-hypothesis items from

δ

-SNLI with one strengthener and one weakener each, elicit the prior and the posteriors in separate calls, and measure the directional movement s: the mean posterior change signed by the annotated direction, so that the prior cancels out. We vary whether the update is stated ("Here is something else you learned") or reported ("Someone said that"), and whether the model gives one probability or five weighted estimates. We test seven configurations: two local models (Qwen3-8B, OLMo-2-7B; 240 pairs) and five configurations run through agent command-line harnesses (120 pairs), with hypotheses and analysis rules fixed before the formal runs (not publicly registered). Under stated, direct elicitation all seven move in the annotated direction (s = 18-28 percentage points). Reporting lowers s in all seven point estimates (significantly in six, by 2-5 points) and raises the share of reverse movements in all seven: the effect is consistent but small. Five weighted estimates lower the share of responses at

\leq

5% or

\geq

95% in all seven, but in the two local models they also reduce s by 43% and 68%, whereas in the harnessed configurations the point estimates of the change lie between -3.7 and +0.5 points. An elicitation format that reduces extreme values thus need not preserve sensitivity to evidence; the two should be tested separately. Because

δ

-NLI annotates only direction, we make no claim about how large an update should be. Code, data and the frozen plan: https://github.com/asj252/llm-judgment-studies