Theoretical Foundations of Latent Posterior Factors: Formal Guarantees for Multi-Evidence Reasoning

Aliyu Agboola Alege

cs.AIMar 13, 2026

Theoretical Foundations of Latent Posterior Factors: Formal Guarantees for Multi-Evidence Reasoning

Authors: Aliyu Agboola Alege

Organizations: Epalea

Abstract

We present a complete theoretical characterization of Latent Posterior Factors (LPF), a principled framework for aggregating multiple heterogeneous evidence items in probabilistic prediction tasks. Multi-evidence reasoning arises pervasively in high-stakes domains including healthcare diagnosis, financial risk assessment, legal case analysis, and regulatory compliance, yet existing approaches either lack formal guarantees or fail to handle multi-evidence scenarios architecturally. LPF encodes each evidence item into a Gaussian latent posterior via a variational autoencoder, converting posteriors to soft factors through Monte Carlo marginalization, and aggregating factors via exact Sum-Product Network inference (LPF-SPN) or a learned neural aggregator (LPF-Learned). We prove seven formal guarantees spanning the key desiderata for trustworthy AI: Calibration Preservation (ECE <= epsilon + C/sqrt(K_eff)); Monte Carlo Error decaying as O(1/sqrt(M)); a non-vacuous PAC-Bayes bound with train-test gap of 0.0085 at N=4200; operation within 1.12x of the information-theoretic lower bound; graceful degradation as O(epsilondeltasqrt(K)) under corruption, maintaining 88% performance with half of evidence adversarially replaced; O(1/sqrt(K)) calibration decay with R^2=0.849; and exact epistemic-aleatoric uncertainty decomposition with error below 0.002%. All theorems are empirically validated on controlled datasets spanning up to 4,200 training examples. Our theoretical framework establishes LPF as a foundation for trustworthy multi-evidence AI in safety-critical applications.

Figures & tables

Panel of Figure 1 — Figure 1: Dependency graph of LPF theoretical results. Assumptions (top) support lemmas and intermediate results, which enable the seven main theorems. Arrows indicate logical dependence. Note that different theorems use different subsets of assumptions: Theorem 3.3 (Generalization) is data-dependent and does not directly rely on Assumptions A1–A6, while Theorems 3.6 and 3.7 build on the results of Theorems 3.1 , 3.2 , and 3.4 rather than their assumptions alone.

Theorem	Key Implementation Details	Verification Experiment	Dataset	Key Metric	Code Variable
T1: Calibration	Does NOT use $\sigma_{\max}$ ; only A1, A3, A4	10-bin calibration	Synthetic ( $N=700$ )	ECE	epsilon , delta_theoretical
T2: MC Error	Uses A2 for bounded decoder inputs	$M$ -ablation study	20 posteriors	Max error	M , errors
T3: Generalization	Uses $d_{\mathrm{eff}}$ , NOT $\sigma_{\max}$	Train/test split	Dedicated ( $N=4200$ )	Gap vs bound	vc_dim , empirical_gap
T4: Info-Theoretic	Uniform weighting	MI computation	Synthetic ( $N=100$ )	ECE vs bound	I_E_Y , noise
T5: Robustness	Uses A1, A6	Corruption injection	Synthetic ( $N=100$ )	L1 distance	corruption_levels , l1_distances
T6: Sample Compl.	$K\in\{1,\ldots,20\}$ for scaling	$K$ -ablation	Synthetic ( $N=100$ )	ECE vs $K$	evidence_counts , lpf_ece

Table 1: Mapping from theoretical guarantees to implementation and empirical verification. All experiments use

K\leq 5

evidence items for main results (extended to

K=20

for Theorem 3.6 scaling studies), except Theorem 3.3 which uses a dedicated dataset with

N=4200

training examples to achieve non-vacuous generalization bounds.

Figure 2: Calibration verification (Theorem 1). Left: ECE for individual evidence (

0.140

), LPF-SPN (

0.185

), and LPF-Learned (

0.058

), with Hoeffding (

0.772

) and Bernstein (

0.459

) tight bounds annotated. Centre and right: reliability diagrams for LPF-SPN and LPF-Learned showing confidence vs. accuracy against the perfect-calibration diagonal.

$M$	Mean Error	Std Error	95th Percentile	Theoretical Bound
4	$0.019\pm 0.044$	0.044	0.080	0.774
8	$0.016\pm 0.030$	0.030	0.069	0.547
16	$0.013\pm 0.018$	0.018	0.053	0.387
32	$0.010\pm 0.012$	0.012	0.037	0.274
64	$0.008\pm 0.009$	0.009	0.025	0.193

Table 2: Monte Carlo error bounds: empirical results vs. theoretical guarantees (Theorem 2).

Figure 3: Monte Carlo error bounds (Theorem 2). Left: log-log plot of mean error, 95th-percentile error, and theoretical bound vs.

M\in\{4,8,16,32,64\}

; all empirical curves remain well below the bound. Right: normalised error scaling confirms the empirical rate closely tracks

O(1/\sqrt{M})

theory.

$N$	Train Loss	Test Loss	Gap	Bound
2002	0.0407	0.0496	0.0089	0.278
3003	0.0393	0.0455	0.0062	0.253
4200	0.0379	0.0463	0.0085	0.228

Table 3: Generalization bound verification across training sizes (Theorem 3).

Figure 4: Generalization bound verification (Theorem 3). Top-left: train and test loss learning curves with confidence intervals across

N\in\{2002,3003,4200\}

. Top-right: empirical gap (near zero) vs. VC bound (loose) and data-dependent PAC-Bayes bound (tight,

0.228

N=4200

). Bottom-left: bound-to-gap ratio on a log scale. Bottom-right: test loss vs.

N

with effective dimension

d_{\mathrm{eff}}=1335

marked.

Metric	Value	Interpretation
$\bar{H}(Y\|E)$ (uniform)	0.158 bits	Reported value
$H(Y)$	1.399 bits	Maximum possible
Reduction	88.7%	Evidence is highly informative
Evidence noise	0.317 bits	Moderate conflicts exist

Table 4: Theorem 4 approximation quality.

Figure 5: Information-theoretic lower bound (Theorem 4). Top-left: decomposition of total uncertainty

H(Y)=1.399

bits into evidence information

I(E;Y)=1.399

and residual

H(Y|E)\approx 0

. Top-right: ECE comparison — theoretical lower bound (

0.158

), achievable bound including MC term (

0.317

), and LPF-SPN empirical ECE (

0.178

). Bottom-left: evidence quality distribution (mean

\approx 1.0

). Bottom-right: scatter of calibration error vs. evidence conflict (KL divergence), with trend

y=0.248x+0.137

$\epsilon$	Mean L1	Std L1	Bound $C\cdot\epsilon\,\delta\,\sqrt{K}$	Actual / Bound
0.0	0.000	0.000	0.000	—
0.05	0.000	0.000	0.316	0%
0.1	0.000	0.000	0.632	0%
0.2	$0.115\pm 0.008$	0.008	1.265	9%
0.3	$0.115\pm 0.008$	0.008	1.897	6%
0.5	$0.122\pm 0.008$	0.008	3.162	4%

Table 5: Robustness verification: empirical degradation vs. theoretical bound (Theorem 5).

Figure 6: Robustness to evidence corruption (Theorem 5). Left: empirical L1 distance

\|p_{\text{clean}}-p_{\text{corrupted}}\|

(blue) remains near zero while the theoretical

O(\epsilon\sqrt{K})

bound (red dashed) grows linearly; the safe region is shaded. Right: bound-to-empirical ratio (up to

6\times 10^{7}

\epsilon=0.1

), confirming the bound is highly conservative in practice.

$K$	LPF-SPN ECE	Bound $C/\sqrt{K}+\epsilon_{0}$
1	$0.347\pm 0.004$	24.28
2	$0.334\pm 0.013$	17.17
3	$0.284\pm 0.008$	14.02
5	$0.186\pm 0.008$	10.86
7	$0.192\pm 0.010$	9.18
10	$0.192\pm 0.010$	7.68

Table 6: Sample complexity verification: LPF-SPN ECE vs. theoretical bounds (Theorem 6).

Figure 7: Sample complexity scaling (Theorem 6). Top-left: LPF-Learned ECE (blue) and baseline uniform ECE (green) both lie far below the theoretical

O(1/\sqrt{K})

bound (red dashed) for

K\in\{1,\ldots,20\}

. Top-right: bound-to-empirical ECE ratio. Bottom-left:

O(1/\sqrt{K})

fit (

0.25/\sqrt{K}+0.12

R^{2}=0.849

) with empirical ECE plateauing at

K\approx 7

. Bottom-right: LPF vs. uniform baseline at

K\in\{1,2,3,5\}

; baseline available only for

K\geq 5

$K$	Total Variance	Epistemic Variance	Aleatoric Variance	Decomp. Error
1	$0.0537\pm 0.053$	$0.0341\pm 0.039$	$0.0196\pm 0.016$	0.001%
2	$0.1302\pm 0.184$	$0.0920\pm 0.138$	$0.0383\pm 0.047$	0.002%
3	$0.1690\pm 0.212$	$0.1230\pm 0.163$	$0.0460\pm 0.050$	0.001%
5	$0.1532\pm 0.185$	$0.1107\pm 0.141$	$0.0425\pm 0.045$	0.001%

Table 7: Uncertainty decomposition results (Theorem 7).

Figure 8: Uncertainty decomposition (Theorem 7). Top-left: total, epistemic (reducible), and aleatoric (irreducible) variance vs.

K

, showing the non-monotonic epistemic trajectory (rises

K=1\to 3

, falls

K=3\to 5

) while aleatoric variance stabilises at

\approx 0.042

. Top-right: stacked area chart of variance components. Bottom-left: decomposition error remains

<0.002\%

, well below the

10\%

threshold (dashed). Bottom-right: epistemic variance isolated, confirming reduction with additional evidence against the constant aleatoric floor (

\approx 0.020

Theorem	Theory Prediction	Empirical Result	Status
T1: Calibration	$\mathrm{ECE}\leq\epsilon+C/\sqrt{K}$	$0.185\leq 1.034$	✓ 82% margin
T2: MC Error	$O(1/\sqrt{M})$ scaling	Strong fit ( $R^{2}=0.849$ )	✓ Verified
T3: Generalization	Non-vacuous bound	Gap $0.0085$ vs. bound $0.228$	✓ 96.3% margin
T4: Info-Theoretic	$\mathrm{ECE}\geq\mathrm{noise}+\bar{H}(Y\|E)/H(Y)$	$0.178$ vs. $0.317$ achievable	✓ $1.12\times$ optimal
T5: Robustness	$O(\epsilon\,\delta\,\sqrt{K})$ graceful	$0.122$ vs. $3.162$ bound	✓ 4% of worst-case
T6: Sample Complexity	$O(1/\sqrt{K})$ scaling	ECE plateau at $K\approx 7$	✓ Strong fit

Table 8: Theoretical predictions vs. empirical results ( Alege, 2026 ) .

Property	Baseline (Uniform Avg)	LPF-SPN	LPF-Learned
Valid probability distribution	✓	✓ (Lemma 3)	✓ (Lemma 3)
Order invariance	✓	✓ (by design)	✓ (symmetric arch.)
Calibration preservation	$\times$	✓ $\ECE\leq\epsilon+C/\sqrt{K}$ (T1)	Empirical only (0.058)
MC error control	N/A	✓ $O(1/\sqrt{M})$ (T2)	✓ $O(1/\sqrt{M})$ (T2)
Generalization bound	Vacuous	N/A (non-parametric)	✓ Non-vacuous at $N=4200$ (T3)
Info-theoretic optimality	$\times$	✓ $1.12\times$ achievable (T4)	Empirical

Table 9: Theoretical property comparison. LPF offers provably better robustness (

\sqrt{K}

vs.

K

scaling), near-optimal calibration (

1.12\times

information-theoretic bound), and exact uncertainty decomposition. Note: LPF-SPN has numerically worse empirical ECE (0.185) than LPF-Learned (0.058) and Baseline (0.036) at

K=5

, but uniquely provides formal calibration guarantees (Theorem 3.1 ) and exact uncertainty decomposition (Theorem 3.7 ).

Metric	Baseline	LPF-SPN	LPF-Learned	Note
Calibration (ECE, $K=5$ )	0.036	0.186	0.058	Baseline best empirically
Test accuracy	$\sim$ 85%	$\sim$ 92%	95.4%	$+$ 10.4 pp vs baseline
Train-test gap	Unknown	N/A	0.0085	96.3% below bound
Epistemic decomp. error	N/A	$<$ 0.002%	$<$ 0.002%	Exact
Robustness ( $\epsilon=0.5$ )	$\sim$ 50%	12% L1	12% L1	$4\times$ more robust
MC error ( $M=16$ )	N/A	$0.013\pm 0.018$	$0.013\pm 0.018$	Within $O(1/\sqrt{M})$

Table 10: Empirical performance comparison

Explore similar work

Mar 13, 2026

I Know What I Don't Know: Latent Posterior Factor Models for Multi-Evidence Probabilistic Reasoning

Real-world decision-making, from tax compliance assessment to medical diagnosis, requires aggregating multiple noisy and potentially contradictory evidence sources. Existing approaches either lack explicit uncertainty quantification (neural aggregation methods) or rely on manually engineered discrete predicates (probabilistic logic frameworks), limiting scalability to unstructured data. We introduce Latent Posterior Factors (LPF), a framework that transforms Variational Autoencoder (VAE) latent posteriors into soft likelihood factors for Sum-Product Network (SPN) inference, enabling tractable probabilistic reasoning over unstructured evidence while preserving calibrated uncertainty estimates. We instantiate LPF as LPF-SPN (structured factor-based inference) and LPF-Learned (end-to-end learned aggregation), enabling a principled comparison between explicit probabilistic reasoning and learned aggregation under a shared uncertainty representation. Across eight domains (seven synthetic and the FEVER benchmark), LPF-SPN achieves high accuracy (up to 97.8%), low calibration error (ECE 1.4%), and strong probabilistic fit, substantially outperforming evidential deep learning, LLMs and graph-based baselines over 15 random seeds. Contributions: (1) A framework bridging latent uncertainty representations with structured probabilistic reasoning. (2) Dual architectures enabling controlled comparison of reasoning paradigms. (3) Reproducible training methodology with seed selection. (4) Evaluation against EDL, BERT, R-GCN, and large language model baselines. (5) Cross-domain validation. (6) Formal guarantees in a companion paper.

Aliyu Agboola Alege

Epalea

Latent Variable Models Uncertainty Calibration

Ask PDF

Jun 29, 2026

BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation

Large language models (LLMs) are typically deployed in multi-turn conversations, where each turn provides new evidence that should reduce epistemic uncertainty about their environment. Acting rationally then requires inferring the unobserved quantities that govern it and updating beliefs about them as evidence accumulates. Yet most evaluations only score the model's final-turn answer in a single-turn format, leaving this process unexamined. We ask how closely LLMs' belief updates match those of a rational Bayesian reasoner in multi-turn settings, and introduce BayesBench, a suite of simulation environments that probe this across three progressively complex tasks: (i) Bayesian estimation, where the model infers an unknown parameter from sequential evidence; (ii) Bayesian prediction, where the model turns inferred beliefs about a latent variable into outcome forecasts; and (iii) latent-framed Bayesian prediction, where observations are filtered through a user-persona framing, requiring joint inference over the latent state and the persona. Across seven LLMs (3B--70B), scaling improves latent inference and evidence accumulation, with updates occasionally matching the Bayesian posterior. However, these gains do not reliably carry over to downstream prediction, exposing a gap between inferring latent structure and using it to rationally update beliefs about the target outcome.

Ankur Samanta, Akshayaa Magesh, Tal Lancewicki +7

Meta AI · Columbia University · Work done at Meta +2

LLM Evaluation Bayesian Inference Belief Updating in LLMs LLM Reasoning

Ask PDF

Mar 20, 2026

EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models

Large Language Models (LLMs) are fluent but prone to hallucinations, producing answers that appear plausible yet are unsupported by available evidence. This failure is especially problematic in high-stakes domains where decisions must be justified by verifiable information. We introduce EvidenceRL, a reinforcement learning framework that enforces evidence adherence during training. EvidenceRL scores candidate responses for grounding (entailment with retrieved evidence and context) and correctness (agreement with reference answers) and optimizes the generator using Group Relative Policy Optimization (GRPO). We evaluate across two high-stakes domains, cardiac diagnosis and legal reasoning, where EvidenceRL consistently improves evidence grounding and faithfulness without sacrificing task accuracy. On cardiac diagnosis, F1@3 increases from 37.0 to 54.5 on Llama-3.2-3B while grounding (

G_{\max}@3

) rises from 47.6 to 78.2; hallucinations drop nearly 5

\times

and evidence-supported diagnoses increase from 31.8% to 61.6%. On legal reasoning, EvidenceRL raises Faithfulness from 32.8% to 67.6% on Llama-3.1-8B, demonstrating consistent behavioral change across domains. Our code is open-sourced at https://github.com/Wizaaard/EvidenceRL.git.

J. Ben Tamo, Yuxing Lu, Benoit L. Marteau +2

Georgia Institute of Technology · Pekin University

LLM Hallucination Mitigation LLM Grounding Faithfulness of Language Model Explanations RL for Language Model Reasoning

Ask PDF