Empirical Characterization of Inference-Time Elicited Probability Transformations in Large Language Models

Mike Farmer; Abhinav Kochar; Yugyung Lee

cs.CLFeb 26, 2026

Empirical Characterization of Inference-Time Elicited Probability Transformations in Large Language Models

Authors: Mike Farmer, Abhinav Kochar, Yugyung Lee

Organizations: Bloch School of Management, Regnier Institute for Entrepreneurship & Innovation, University of Missouri–Kansas City. · Department of Computer Science, School of Science and Engineering, University of Missouri–Kansas City.

Abstract

Large language models increasingly rely on inference-time procedures such as chain-of-thought reasoning, self-refinement, retrieval augmentation, and verifier-guided revision, yet the structure of elicited probability transformations under these procedures remains poorly understood. We study externally elicited probability assignments over candidate answers and observe recurring approximate log-ratio relationships:

\log \tilde q_t(i) = α_t \left( \log q_t(i) + \log b_t(i) \right) + c_t,

where $q_t$ and $\tilde q_t$ are pre- and post-elicitation probabilities, $b_t$ is an externally constructed evidence signal, and $α_t$ is an empirical descriptor of the prompting configuration. Across 4,975 reasoning problems from GPQA Diamond, TheoremQA, MMLU-Pro, and ARC-Challenge, evaluated on multiple instruction-tuned model families, we observe approximate log-ratio relationships with mean $R^2 \approx 0.76$ over about $1.3 \times 10^5$ candidate-level observations. Coefficients vary across elicitation settings, but qualitatively similar relationships persist across evaluated conditions. Robustness analyses using alternative statistical representations, prompting configurations, held-out evaluation, and token-level log-probabilities suggest that the observed structure is not tied to one prompting procedure or probability estimation method. The main contribution is not the algebraic form itself, which is related to generalized Bayesian updating and probability-transformation frameworks, but the empirical observation that diverse inference-time prompting pipelines repeatedly exhibit reproducible log-ratio structure under controlled conditions. The framework provides a protocol-sensitive perspective for analyzing calibration, evidence amplification, uncertainty propagation, and interaction sensitivity in inference-time LLM pipelines.

Figures & tables

Figure 1: Pooled regression results for the $\alpha$ -parameterized formulation across evaluated models and datasets. The fitted relationship remains approximately linear in log-ratio space, with the estimated slope moderately exceeding the unit-slope reference.

Figure 2: Residual analysis of the empirical log-ratio relationship. Residuals remain broadly centered around zero across fitted values without pronounced large-scale systematic structure.

Dataset	$\alpha$	$R^{2}$	Observations
GPQA Diamond	1.18	0.82	1,240
TheoremQA	1.12	0.79	1,050
MMLU-Pro	1.05	0.71	1,530
ARC-Challenge	1.26	0.88	1,180

Table 1: Cross-dataset regression estimates for the

\alpha

-parameterized formulation.

Model	Log-space MSE
Naive Bayes baseline	Baseline
Prior-only	-12%
Evidence-only	-9%
Fixed $\alpha=1$	-15%
Temperature scaling	-17%
Two-parameter model	-17.2%

Table 2: Held-out descriptive evaluation. Values represent relative log-space changes in MSE with respect to a naive Bayes baseline.

Transformation	$\alpha$ deviation	Correlation
ALR	¡2%	0.99
CLR	¡3%	0.99
ILR	¡4%	0.99

Table 3: Robustness of the

\alpha

-parameterized formulation under alternative compositional transformations.

Figure 3: Condition-specific estimates of $\alpha_{t}$ across independently evaluated prompting settings. Error bars denote bootstrap variability intervals. Although estimated coefficients vary across evidence-construction procedures, broadly similar log-ratio relationships remain observable across evaluated conditions.

Figure 4: Stage-specific $\alpha_{t}$ estimates across repeated elicitation stages under a fixed prompting and evidence-construction pipeline using GPT-5.2. Although substantial variability remains across individual problem instances, observable differences in independently estimated $\alpha_{t}$ values remain visible across prompting stages under the evaluated prompting conditions.

Figure 5: Sensitivity of estimated $\alpha$ values to evidence-encoding strength under alternative normalization procedures. Although coefficient magnitudes vary across evidence-scaling settings, qualitatively similar log-ratio relationships persist across evaluated configurations.

Prompt Configuration	$\alpha$	ECE $\downarrow$	Brier $\downarrow$	Log-loss $\downarrow$
Baseline prompting	1.03	0.11	0.18	0.62
Verifier-conditioned	1.11	0.14	0.21	0.67
Retrieval-augmented	1.18	0.17	0.25	0.73
Aggressive evidence scaling	1.26	0.21	0.30	0.86

Table 4: Representative calibration-oriented reliability metrics across prompting configurations with different estimated

\alpha

values. Larger estimated coefficients are generally associated with stronger evidence amplification and modest increases in calibration-oriented reliability error metrics under the evaluated elicitation conditions.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Residual diagnostics for the primary pooled regression model under the JSON-evidence encoding. Moderate heteroscedasticity and mild heavy-tail behavior remain visible, while residual patterns remain broadly consistent with the approximate log-ratio relationship.

Figure S2 : Comparison between OLS and ILR coefficient estimates across evaluated problem instances. Comparable empirical relationships remain visible across alternative compositional representations and statistical parameterizations.

Method	Correlation with OLS	Mean Relative Difference
ALR	$>0.99$	2.9%
CLR	$>0.99$	3.2%
ILR	$>0.99$	3.7%

Appendix

Table S1 : Comparison of coefficient estimates across compositional parameterizations.

Figure S3 : Bland–Altman comparison between OLS and ILR coefficient estimates. Differences remain small relative to overall variability across evaluated problem instances.

Figure S4 : Comparison between explicit self-reported probability elicitation (SR) and token-level logprob extraction (LP). Although token-level extraction exhibits greater variance and moderate systematic offsets, comparable empirical relationships remain visible across both elicitation paradigms.

Configuration	GPT-5.2	Claude Sonnet 4
Baseline	$0.84\;[0.79,\,0.88]$	$0.77\;[0.72,\,0.81]$
Alt Prompt A	$0.72\;[0.68,\,0.76]$	$0.71\;[0.67,\,0.75]$
Alt Prompt B	$0.61\;[0.57,\,0.65]$	$0.68\;[0.64,\,0.72]$
Alt Prompt C	$0.54\;[0.50,\,0.58]$	$0.66\;[0.62,\,0.70]$

Appendix

Table S2 : Condition-specific robustness analysis across related prompting configurations. Values report median

\alpha_{t}

estimates together with bootstrap variability intervals.

Figure S5 : Extended evidence-encoding sensitivity analysis across scaling regimes. Although the absolute magnitude of $\alpha$ varies across evidence constructions, a consistent empirical structure remains observable throughout the evaluated parameter range.

Figure S6 : Representative prompting patterns across reasoning problems under related prompting configurations. Although substantial variability remains across individual prompting conditions, broadly similar empirical structure persists across evaluated configurations.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 276 papers

Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates
Oct 20, 2026An-Sheng JhangProbability Elicitation from Language Models Belief Updating in LLMs
Dongguan University of Technology
Humans and LLMs Diverge on Probabilistic Inferences
Feb 26, 2026Gaurav Kamath, Sreenath Madathil, Sebastian Schuster +2LLM Reasoning
McGill University · Mila – Quebec AI Institute · University of Vienna +2
Verbalized and Internal Probabilities Are Coupled in Large Language Models
Sep 30, 2026Sinead Williamson, Jiaxuan Li, Nick Foti +2Confidence Estimation in Language Models LLM Uncertainty Estimation
Apple

Oct 20, 2026J/K move · Enter open · S save

Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates

An-Sheng Jhang

Dongguan University of Technology

Probability Elicitation from Language Models Belief Updating in LLMs Framing Effects in Language Models LLM Evaluation LLM Uncertainty Estimation Language Model Probing

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as judgment assistants and as stand-in participants in psychology experiments, yet little is known about how much their probability estimates move in response to a single new piece of information. We pair premise-hypothesis items from

δ

-SNLI with one strengthener and one weakener each, elicit the prior and the posteriors in separate calls, and measure the directional movement s: the mean posterior change signed by the annotated direction, so that the prior cancels out. We vary whether the update is stated ("Here is something else you learned") or reported ("Someone said that"), and whether the model gives one probability or five weighted estimates. We test seven configurations: two local models (Qwen3-8B, OLMo-2-7B; 240 pairs) and five configurations run through agent command-line harnesses (120 pairs), with hypotheses and analysis rules fixed before the formal runs (not publicly registered). Under stated, direct elicitation all seven move in the annotated direction (s = 18-28 percentage points). Reporting lowers s in all seven point estimates (significantly in six, by 2-5 points) and raises the share of reverse movements in all seven: the effect is consistent but small. Five weighted estimates lower the share of responses at

\leq

5% or

\geq

95% in all seven, but in the two local models they also reduce s by 43% and 68%, whereas in the harnessed configurations the point estimates of the change lie between -3.7 and +0.5 points. An elicitation format that reduces extreme values thus need not preserve sensitivity to evidence; the two should be tested separately. Because

δ

-NLI annotates only direction, we make no claim about how large an update should be. Code, data and the frozen plan: https://github.com/asj252/llm-judgment-studies

Abstract

LLM Inference

Explore similar work

Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates

Humans and LLMs Diverge on Probabilistic Inferences

Verbalized and Internal Probabilities Are Coupled in Large Language Models