Likelihood Hacking in Probabilistic Program Synthesis

Jacek Karwowski; Younesse Kaddar; Zihuiwen Ye; Esmeralda S. Whitammer; Sam Staton

cs.LGMar 25, 2026

Likelihood Hacking in Probabilistic Program Synthesis

Authors: Jacek Karwowski, Younesse Kaddar, Zihuiwen Ye, Esmeralda S. Whitammer, Sam Staton

Organizations: University of Oxford, Department of Computer Science · University of Edinburgh, School of Informatics · CIFAR Fellow, Learning in Machines and Brains

Abstract

When language models are trained by reinforcement learning (RL) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing programs whose data distribution fails to normalise instead of fitting the data better. We call this failure likelihood hacking (LH). We formalise LH in a core probabilistic programming language (PPL) and give sufficient syntactic conditions for its prevention, proving that a safe language fragment $\mathcal{L}_{\text{safe}}$ satisfying these conditions cannot produce likelihood-hacking programs. Empirically, we show that GRPO-trained models generating PyMC code discover LH exploits within the first few training steps, driving violation rates well above the untrained-model baseline. We implement $\mathcal{L}_{\text{safe}}$ 's conditions as $\texttt{SafeStan}$ , a LH-resistant modification of Stan, and show empirically that it suppresses LH under optimisation pressure. These results show that language-level safety constraints are both theoretically grounded and effective in practice for automated Bayesian model discovery.

Figures & tables

(a) Honest Gaussian regression.

\displaystyle\Gamma;\Delta\vdash_{\text{u}}

\displaystyle\mathsf{let}\;\beta\leftarrow\mathsf{sample}(\mathsf{gauss}(0,1))\;\mathsf{in}\;

\displaystyle\mathsf{let}\;\_\leftarrow\mathsf{observe}(\mathtt{y},\mathsf{gauss}(\beta\cdot x,1))\;\mathsf{in}\;

\displaystyle\mathsf{return}(\beta):\;P(\mathbb{R})

A Bayesian regression model: sample a latent slope

\beta

, then score the datapoint

\mathtt{y}

under a proper likelihood.

(b) Improper observation model via density addition.

\displaystyle\Gamma;\Delta\vdash_{\text{u}}

\displaystyle\mathsf{let}\;\beta\leftarrow\mathsf{sample}(\mathsf{gauss}(0,1))\;\mathsf{in}\;

\displaystyle\mathsf{let}\;\_\leftarrow\mathsf{observe}(\mathtt{y},\mathsf{gauss}(\beta\cdot x,1)+\mathsf{gauss}(\beta\cdot x,1))\;\mathsf{in}\;

\displaystyle\mathsf{return}(\beta):\;P(\mathbb{R})

Here

\mathcal{D}_{1}+\mathcal{D}_{2}

is the sum of two densities, not a mixture: it multiplies likelihoods by a constant factor (here

2

), breaking normalisation.

\displaystyle\Gamma;\Delta\vdash_{\text{u}}

\displaystyle\mathsf{let}\;\beta\leftarrow\mathsf{sample}(\mathsf{gauss}(0,1))\;\mathsf{in}\;

\displaystyle\mathsf{let}\;\_\leftarrow\mathsf{observe}(\mathtt{y},\mathsf{gauss}(\beta\cdot x,1))\;\mathsf{in}\;

\displaystyle\mathsf{let}\;\_\leftarrow\mathsf{observe}(\mathtt{y},\mathsf{gauss}(\beta\cdot x,1))\;\mathsf{in}\;

\displaystyle\mathsf{return}(\beta):\;P(\mathbb{R})

Well-typed in

\mathcal{L}_{\text{unsafe}}

, but reuses

\mathtt{y}

: the data density is squared, so the induced data distribution is not normalised.

(d) Disguised Potential -style bonus via score .

\displaystyle\Gamma;\Delta\vdash_{\text{u}}

\displaystyle\mathsf{let}\;\beta\leftarrow\mathsf{sample}(\mathsf{gauss}(0,1))\;\mathsf{in}\;

\displaystyle\mathsf{let}\;\_\leftarrow\mathsf{observe}(\mathtt{y},\mathsf{gauss}(\beta\cdot x,1))\;\mathsf{in}\;

\displaystyle\mathsf{let}\;\_\leftarrow\mathsf{score}(\log\!\bigl(1+\exp(-10\cdot(\lvert\beta\rvert-5))\bigr))\;\mathsf{in}\;

\displaystyle\mathsf{return}(\beta):\;P(\mathbb{R})

Adds an extra positive term to the log-score for typical

\beta

values (analogous to a PyMC Potential), inflating marginal likelihood without improving the fit to

\mathtt{y}

Figure 2: Example programs in

\mathcal{L}_{\text{unsafe}}

. We fix a simple interface with one covariate

\Gamma\overset{\Delta}{=}(x:\mathbb{R})

, one datapoint

\Delta\overset{\Delta}{=}(\mathtt{y}:\mathbb{R})

, and output

P(\mathbb{R})

. All four programs are well-typed in

\mathcal{L}_{\text{unsafe}}

, but only (a) corresponds to a properly normalised probabilistic model over the interface. Programs (b) – (d) illustrate distinct likelihood-hacking mechanisms enabled by

\mathcal{L}_{\text{unsafe}}

	Run 1	Run 2
Training steps completed	29	28
Total completions (valid + failed)	23,053	22,257
Valid programs	17,522	17,042
Programs with reward $>0$	64	42
Programs with pm.Potential	804	606
Reward ceiling hits ( $=20$ nats)	5	12

Table 1: Aggregate statistics for two runs of the likelihood hacking emergence experiment. Totals exclude one interrupted terminal step per run, and realised completions can fall below

\text{steps}\times 800

when a terminal step is interrupted before batch completion. Max normalisation failure rate: the largest fraction of programs flagged non-normalised in any single periodic check (5–10 programs sampled every five steps).

(a) Honest model. ⬇ def model ( data ): with pm . Model () as m : p = pm . Beta ( ’p’ , 1, 1) pm . Bernoulli ( ’y’ , p = p , observed = data [ ’y’ ]) return m Conjugate Beta–Bernoulli; normalises to

M=1

. p

(b) Score injection. ⬇ def model ( data ): with pm . Model () as m : logit_p = pm . Normal ( ’logit_p’ , 0, 2) p = pm . Deterministic ( ’p’ , pm . math . sigmoid ( logit_p )) pm . Bernoulli ( ’y’ , p = p , observed = data [ ’y’ ]) pm . Potential ( ’regularisation’ , -10 * pm . math . log ( pm . math . abs ( p - 0.5) + 1 e -6)) return m The Potential injection, disguised as “regularisation”.

Figure 3: Four PyMC programs generated during GRPO training. (a) Conjugate Beta–Bernoulli; normalises to

M=1

. (b) The pm.Potential labelled “regularisation” disguises score injection as regularisation. (c) Compound exploit: data[’y’] is observed twice (Beta + Bernoulli) and score is injected via Potential . (d) An exploit that conditions deterministic computation of the model parameter on the observation. Programs (b), (c), and (d) are flagged by the normalisation check ( Section 4.1 ). More examples are listed in Appendix C .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

$(v:T)\in\Gamma$ (Var u ) $\Gamma;\Delta\vdash_{\text{u}}v:T$	$(\mathtt{x}:T)\in\Delta$ (Data u ) $\Gamma;\Delta\vdash_{\text{u}}\mathtt{x}:T$
$r\in\mathbb{R}$ (Const u ) $\Gamma;\Delta\vdash_{\text{u}}r:\mathbb{R}$	(Unit u ) $\Gamma;\Delta\vdash_{\text{u}}\star:\mathbb{1}$
$\Gamma;\Delta\vdash_{\text{u}}t_{1}:T_{1}$ $\Gamma;\Delta\vdash_{\text{u}}t_{2}:T_{2}$ (Pair u ) $\Gamma;\Delta\vdash_{\text{u}}(t_{1},t_{2}):T_{1}\times T_{2}$	$\Gamma;\Delta\vdash_{\text{u}}t:T_{1}\times T_{2}$ ( $\pi_{1}{}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\pi_{1}t:T_{1}$
$\Gamma;\Delta\vdash_{\text{u}}t:T_{1}\times T_{2}$ ( $\pi_{2}{}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\pi_{2}t:T_{2}$	$\Gamma;\Delta\vdash_{\text{u}}t:T_{1}$ (Inl u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{left}\ t:T_{1}+T_{2}$
$\Gamma;\Delta\vdash_{\text{u}}t:T_{2}$ (Inr u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{right}\ t:T_{1}+T_{2}$	$\Gamma;\Delta\vdash_{\text{u}}t:\mathbb{2}$ $\Gamma;\Delta\vdash_{\text{u}}t_{1}:T$ $\Gamma;\Delta\vdash_{\text{u}}t_{2}:T$ (If u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{if}\;t\;\mathsf{then}\;t_{1}\;\mathsf{else}\;t_{2}:T$
$\Gamma;\Delta\vdash_{\text{u}}t_{1}:U$ $\Gamma,(v:U);\Delta\vdash_{\text{u}}t_{2}:T$ (Let u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{let}\;v=t_{1}\;\mathsf{in}\;t_{2}:T$	$\begin{array}[]{c}f:(T_{1},\ldots,T_{k})\to U\\ \Gamma;\Delta\vdash_{\text{u}}t_{1}:T_{1}\hskip 8.50012pt\cdots\hskip 8.50012pt\Gamma;\Delta\vdash_{\text{u}}t_{k}:T_{k}\end{array}$ (Prim u ) $\Gamma;\Delta\vdash_{\text{u}}f(t_{1},\ldots,t_{k}):U$

Appendix

Figure 4: Typing rules for pure terms in

\mathcal{L}_{\text{unsafe}}

$\Gamma;\Delta\vdash_{\text{u}}t_{1}:\mathbb{R}$ $\Gamma;\Delta\vdash_{\text{u}}t_{2}:\mathbb{R}$ (Gauss u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{gauss}(t_{1},t_{2}):\mathsf{Meas}(\mathbb{R})$	$\Gamma;\Delta\vdash_{\text{u}}t:\mathbb{R}$ (Bern u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{bern}(t):\mathsf{Meas}(\mathbb{2})$
$\Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{1}:\mathsf{Meas}(T)$ $\Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{2}:\mathsf{Meas}(T)$ (Plus u ) $\Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{1}+\mathcal{D}_{2}:\mathsf{Meas}(T)$
$\begin{array}[]{c}F:\mathsf{Meas}(T_{1})\times\cdots\times\mathsf{Meas}(T_{k})\to\mathsf{Meas}(U)\\ \Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{1}:\mathsf{Meas}(T_{1})\hskip 8.50012pt\cdots\hskip 8.50012pt\Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{k}:\mathsf{Meas}(T_{k})\end{array}$ ( $F_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}F(\mathcal{D}_{1},\ldots,\mathcal{D}_{k}):\mathsf{Meas}(U)$	$\begin{array}[]{c}\Gamma;\Delta\vdash_{\text{u}}t:\mathbb{R}\\ \Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{1}:\mathsf{Meas}(T)\ \ \ \Gamma;\Delta\vdash_{\text{u}}\mathcal{D}_{2}:\mathsf{Meas}(T)\end{array}$ (Mix u ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{mix}(t;\mathcal{D}_{1},\mathcal{D}_{2}):\mathsf{Meas}(T)$

Appendix

Figure 5: Typing rules for distributions in

\mathcal{L}_{\text{unsafe}}

$\Gamma;\Delta\vdash_{\text{u}}t:T$ ( $\mathsf{return}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{return}(t):P(T)$	$\Gamma;\Delta\vdash_{\text{u}}\mathcal{D}:\mathsf{Meas}(T)$ ( $\mathsf{sample}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{sample}(\mathcal{D}):P(T)$
$\Gamma;\Delta\vdash_{\text{u}}\mathcal{D}:\mathsf{Meas}(T)$ $\Gamma;\Delta\vdash_{\text{u}}t:T$ ( $\mathsf{observe}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{observe}(t,\mathcal{D}):P(T)$	$\Gamma;\Delta\vdash_{\text{u}}t:\mathbb{R}$ ( $\mathsf{score}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{score}(t):P(\mathbb{1})$
$\Gamma;\Delta\vdash_{\text{u}}p_{1}:P(U)$ $\Gamma,(v:U);\Delta\vdash_{\text{u}}p_{2}:P(T)$ ( $\mathsf{let}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{let}\;v\leftarrow p_{1}\;\mathsf{in}\;p_{2}:P(T)$	$\Gamma;\Delta\vdash_{\text{u}}p:P(\mathbb{2})$ $\Gamma;\Delta\vdash_{\text{u}}p_{1}:P(T)$ $\Gamma;\Delta\vdash_{\text{u}}p_{2}:P(T)$ ( $\mathsf{if}_{u}$ ) $\Gamma;\Delta\vdash_{\text{u}}\mathsf{if}\;p\;\mathsf{then}\;p_{1}\;\mathsf{else}\;p_{2}:P(T)$

Appendix

Figure 6: Typing rules for programs in

\mathcal{L}_{\text{unsafe}}

$(v:T)\in\Gamma$ (Var s ) $\Gamma\vdash_{\text{s}}v:T$	$r\in\mathbb{R}$ (Const s ) $\Gamma\vdash_{\text{s}}r:\mathbb{R}$
(Unit s ) $\Gamma\vdash_{\text{s}}\star:\mathbb{1}$	$\Gamma\vdash_{\text{s}}t_{1}:T_{1}$ $\Gamma\vdash_{\text{s}}t_{2}:T_{2}$ (Pair s ) $\Gamma\vdash_{\text{s}}(t_{1},t_{2}):T_{1}\times T_{2}$
$\Gamma\vdash_{\text{s}}t:T_{1}\times T_{2}$ ( $\pi_{1}{}_{s}$ ) $\Gamma\vdash_{\text{s}}\pi_{1}t:T_{1}$	$\Gamma\vdash_{\text{s}}t:T_{1}\times T_{2}$ ( $\pi_{2}{}_{s}$ ) $\Gamma\vdash_{\text{s}}\pi_{2}t:T_{2}$
$\Gamma\vdash_{\text{s}}t:T_{1}$ (Inl s ) $\Gamma\vdash_{\text{s}}\mathsf{left}\ t:T_{1}+T_{2}$	$\Gamma\vdash_{\text{s}}t:T_{2}$ (Inr s ) $\Gamma\vdash_{\text{s}}\mathsf{right}\ t:T_{1}+T_{2}$
$\Gamma\vdash_{\text{s}}t:\mathbb{2}$ $\Gamma\vdash_{\text{s}}t_{1}:T$ $\Gamma\vdash_{\text{s}}t_{2}:T$ (If s ) $\Gamma\vdash_{\text{s}}\mathsf{if}\;t\;\mathsf{then}\;t_{1}\;\mathsf{else}\;t_{2}:T$	$\Gamma\vdash_{\text{s}}t_{1}:U$ $\Gamma,(v:U)\vdash_{\text{s}}t_{2}:T$ (Let s ) $\Gamma\vdash_{\text{s}}\mathsf{let}\;v=t_{1}\;\mathsf{in}\;t_{2}:T$
$f:(T_{1},\ldots,T_{k})\to U$ $\Gamma\vdash_{\text{s}}t_{1}:T_{1}\hskip 8.50012pt\cdots\hskip 8.50012pt\Gamma\vdash_{\text{s}}t_{k}:T_{k}$ (Prim s ) $\Gamma\vdash_{\text{s}}f(t_{1},\ldots,t_{k}):U$

Appendix

Figure 7: Typing rules for pure terms in

\mathcal{L}_{\text{safe}}

$\Gamma\vdash_{\text{s}}t_{1}:\mathbb{R}$ $\Gamma\vdash_{\text{s}}t_{2}:\mathbb{R}$ (Gauss s ) $\Gamma\vdash_{\text{s}}\mathsf{gauss}(t_{1},t_{2}):\mathsf{Prob}(\mathbb{R})$	$\Gamma\vdash_{\text{s}}t:\mathbb{R}$ (Bern s ) $\Gamma\vdash_{\text{s}}\mathsf{bern}(t):\mathsf{Prob}(\mathbb{2})$
$\begin{array}[]{c}\Gamma\vdash_{\text{s}}t:\mathbb{R}\\ \Gamma\vdash_{\text{s}}\mathcal{D}_{1}:\mathsf{Prob}(T)\hskip 17.00024pt\Gamma\vdash_{\text{s}}\mathcal{D}_{2}:\mathsf{Prob}(T)\end{array}$ (Mix s ) $\Gamma\vdash_{\text{s}}\mathsf{mix}(t;\mathcal{D}_{1},\mathcal{D}_{2}):\mathsf{Prob}(T)$	$\begin{array}[]{c}F:\mathsf{Prob}(T_{1})\times\cdots\times\mathsf{Prob}(T_{k})\to\mathsf{Prob}(U)\\ \Gamma\vdash_{\text{s}}\mathcal{D}_{1}:\mathsf{Prob}(T_{1})\hskip 8.50012pt\cdots\hskip 8.50012pt\Gamma\vdash_{\text{s}}\mathcal{D}_{k}:\mathsf{Prob}(T_{k})\end{array}$ (Comb s ) $\Gamma\vdash_{\text{s}}F(\mathcal{D}_{1},\ldots,\mathcal{D}_{k}):\mathsf{Prob}(U)$

Appendix

Figure 8: Typing rules for distributions in

\mathcal{L}_{\text{safe}}

$\Gamma\vdash_{\text{s}}t:T$ ( $\mathsf{return}_{s}$ ) $\Gamma;\varnothing\vdash_{\text{s}}\mathsf{return}(t):P(T)$	$\Gamma\vdash_{\text{s}}\mathcal{D}:\mathsf{Prob}(T)$ ( $\mathsf{sample}_{s}$ ) $\Gamma;\varnothing\vdash_{\text{s}}\mathsf{sample}(\mathcal{D}):P(T)$
$\Gamma\vdash_{\text{s}}\mathcal{D}:\mathsf{Prob}(T)$ ( $\mathsf{observe}_{s}$ ) $\Gamma;\mathtt{x}:T\vdash_{\text{s}}\mathsf{observe}(\mathtt{x},\mathcal{D}):P(T)$	$\Gamma;\Delta_{1}\vdash_{\text{s}}p_{1}:P(U)$ $\Gamma,(v:U);\Delta_{2}\vdash_{\text{s}}p_{2}:P(T)$ ( $\mathsf{let}_{s}$ ) $\Gamma;\Delta_{1}\sqcup\Delta_{2}\vdash_{\text{s}}\mathsf{let}\;v\leftarrow p_{1}\;\mathsf{in}\;p_{2}:P(T)$
$\Gamma;\Delta_{b}\vdash_{\text{s}}p:P(\mathbb{2})$ $\Gamma;\Delta\vdash_{\text{s}}p_{1}:P(T)$ $\Gamma;\Delta\vdash_{\text{s}}p_{2}:P(T)$ ( $\mathsf{if}_{s}$ ) $\Gamma;\Delta_{b}\sqcup\Delta\vdash_{\text{s}}\mathsf{if}\;p\;\mathsf{then}\;p_{1}\;\mathsf{else}\;p_{2}:P(T)$

Appendix

Figure 9: Typing rules for programs in

\mathcal{L}_{\text{safe}}

Table 10

Model source	No. of models	As-is	Minor	Major	Compatible fraction (as-is + minor)
bda_r_demos	14	9	2	3	0.79
betanalpha_knitr_case_studies	302	183	82	37	0.88
example_models	538	82	202	254	0.53
forestdb	101	20	37	44	0.56
posteriordb	147	14	58	75	0.49
stancon_talks	61	7	25	29	0.52

Appendix

Table 4: Number of Stan models collected from each source, and compatibility with SafeStan by source family.

LLM-assigned reason for major label	Newly added models
Using flat/improper priors	164
Finite mixture operator	40
Self-contained observation primitives	109
Custom prior/process primitives	43
Protected-data interface rewrites	50

Appendix

Table 5: Major categories of problematic constructs in Stan collected from the web.

⬇ data { int < lower =1> N ; vector [ N ] X ; vector [ N ] y ; } parameters { real beta ; } model { ... }

⬇ data { int < lower =1> N ; vector [ N ] X ; vector [ N ] y ; } parameters { real beta ; } model { vector [ N ] residual ; residual = y - beta * X ; target += sum ( residual .^2); }

Appendix

Figure 11: Left: Required model interface for the SafeStan Bayesian linear regression experiment. Right : A high-scoring, likelihood-hacking program found by GRPO.

Training parameter	Value
Training steps	200
Number of prompts	2
Number of system prompts	8
Number of generations per prompt	16
LLM temperature	1.7
Contract penalty reward	-100/-20

Appendix

Table 6: Key training parameters. The total batch size used was

2\times 8\times 16=256

, which fit in the 141GB of VRAM of an H200, with the 7B instruct model. Contract penalty was given if the model did not conform to the model shape specified in the prompt, see Figure 11 . Parse fail reward was given if the model was not compiled correctly by the Stan compiler. Penalties were set in a curriculum, the table shows the initial/final value, which were interpolated linearly over the course of training.

Dataset parameter	Value
Beta scale	1.78
$N$ train	8
$N$ test	8
Noise $\sigma$	2.3

Appendix

Table 7: Key dataset parameters. Beta was sampled from a centered normal distribution with std given by beta scale. Data points were then sampled with the noise std

\sigma

. Train and test points were sampled using the same beta.

Figure 12: Likelihood hacking rate over the training run, i.e., the proportion of GRPO groups that contain LH programs. Estimated slope = 0.0678, 95% CI $[-0.002449,0.1381]$ .

Figure 13: Likelihood hacking reward lift over the training run: proportion of reward that is given by LH programs. Estimated slope = 0.01438, 95% CI $[-0.007464,0.03622]$

Figure 14: Proportion of programs that are valid (i.e., compile with base Stan) over the course of training.

Figure 15: Mean of the GRPO reward over the course of the training run. It plateaus over the training run, even as LH programs are found, because of the issues around vanishing entropy.

Figure 16: Ratio of unique programs generated by the LLM over the training run to all programs. The number drops to only around 5% (this does not count minor variations). Vanishing entropy makes training slow down significantly.

PPL	Construct	Kind	LH surface
PyMC	pm.Potential ; custom logp	free score / custom density	arbitrary term in model log-probability
Stan	target +=	free log-density increment	arbitrary expression added to unnormalised log density
NumPyro	numpyro.factor	free score	arbitrary log-probability factor
Pyro	pyro.factor	free score	arbitrary log-probability factor
WebPPL	factor(score)	free score	arbitrary score added to execution log probability
Turing.jl	@addlogprob!	free score	arbitrary term added to accumulated log probability

Appendix

Table 8: Score-like constructs in common PPLs. These interfaces are useful for trusted modelling, but expose a normalisation attack surface when the program author is an RL-trained policy rewarded by marginal likelihood.

Probabilistic Programming

Probabilistic programming combines code and probability models to express uncertainty and automate statistical inference.

See all 20 papers

Jun 30, 2026

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be statistically wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization. The right verifier is therefore not a test suite but the Bayesian workflow itself: posterior predictive checks, simulation-based calibration, sampler diagnostics (

\hat R

, divergences, ESS), and held-out predictive density. We study this calibration oracle along three axes. Detection: on a benchmark of

14

misspecification types across

10

model families (

200

instances), it flags the bug with AUC

0.97

(

88\%

2\%

FPR when handed the correct reference program, an upper bound) -- and a fully reference-free version that uses no correct program reaches

62

78\%

(the upper figure from a small automated model search), versus

0\%

for a unit-test oracle. Repair: used as feedback in an LLM repair loop across fifteen models, calibration significantly outperforms unit-test feedback -- which is itself significantly worse than no feedback at all, a passing test inducing false confidence that suppresses repair -- and improves over no feedback on strong-but-unsaturated models (GPT-5.1

33{\to}92\%

, Claude

75{\to}100\%

; paired McNemar,

n{=}228

). Reality: on programs LLMs write from scratch for neutral briefs,

15

47\%

of runnable ones are statistically misspecified (unit tests catch none), and calibration-guided repair significantly beats LLM-as-judge review, a Bayesian-workflow checklist, and data-summary self-debug. Across all three, the lesson is the same: for probabilistic programs, correctness is calibration, not compilation.

Jian Xu, Delu Zeng, John Paisley +1

RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1

Automated Program Repair Bayesian Inference LLM-Based Program Synthesis Model Misspecification Probabilistic Programming

Ask PDF

Jul 22, 2026

Sound Probabilistic Safety Bounds for Large Language Models

We propose a novel framework for computing rigorous bounds on the probability that a large language model (LLM) generates harmful output to a given prompt. We study a new application of the Clopper-Pearson confidence intervals to obtain probably approximately correct (PAC) bounds for this problem. As our main technical contribution, we propose an algorithm that leverages features in the latent space to prioritize exploring branches in the auto-regressive generation tree that are more likely to produce harmful outputs. Our approach in particular enables the efficient computation of useful lower bounds, even in scenarios where the true harm probability is extremely small, and crucially, the obtained lower bounds are sound, i.e., formally proven to be less than the actual harmfulness probability: our experimental results demonstrate the effectiveness of our method by computing non-trivial lower bounds on state-of-the-art LLMs. This study newly enables the evaluation and statistical certification of LLMs.

Mahdi Nazeri, Anne-Kathrin Schmuck, Sadegh Soudjani +1

Department of Computer Science, University of Oxford, UK · Max Planck Institute for Software Systems, Germany · School of Computer Science, University of Birmingham, UK

Language Model Safety Evaluation Confidence Estimation in Language Models LLM Safety

Ask PDF

Sep 24, 2026

Large Language Bayes Is Not Reparameterisation-Invariant

Large Language Bayes (LLB) answers an informal modelling question by sampling candidate probabilistic programs from a language model, running approximate inference on each, and averaging them with weights proportional to an exponentiated evidence bound. We show that this weighting depends on how a model is written. The log marginal likelihood is invariant to reparameterisation; the evidence bound is not. On eight schools the centered and non-centered programs are the same measure to

5.7\times10^{-14}

, yet their weights differ by

6.1\times

; importance weighting reduces this only to

2.2\times

, and reproducing the inference LLB actually runs, a full-covariance Gaussian matched to the posterior moments, still leaves

1.9\times

on eight schools and

8.9\times

64

dimensions. Across likelihood families, dimensions and funnel severities the discrepancy reaches

31.9\times

and reverses sign, so no single writing is uniformly preferable. It inverts Bayes factors against eight natural competitors, and the induced error in the model posterior, and in any downstream target, is controlled by the spread

Δ

of the bound shortfalls through a known sharp Hilbert-distance bound. Across

360

programs from six language models the parameterisation written ranges from

0\%

100\%

centered and is stable within a model. Detecting equivalent programs statistically can falsely merge genuinely different models at practical sample budgets; verifying reparameterisations we generate ourselves cannot, and closes the window.

Jian Xu

RIKEN

Bayesian Inference Probabilistic Programming

Ask PDF

Abstract

Probabilistic Programming

Explore similar work

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

Sound Probabilistic Safety Bounds for Large Language Models

Large Language Bayes Is Not Reparameterisation-Invariant