General Bayesian Policy Learning

Masahiro Kato

stat.MLFeb 27, 2026

General Bayesian Policy Learning

Authors: Masahiro Kato

Organizations: 1Mizuho-DL Financial Technology Co., Ltd., Tokyo, Japan

Abstract

This study proposes a General Bayes framework for policy learning. We consider decision problems in which a decision-maker chooses an action from a given set to maximize expected welfare. Typical examples include treatment choice and portfolio optimization. In such problems, the statistical target is a decision rule, and predicting each potential outcome is not necessarily of primary interest. We formulate this policy-learning problem through loss-based Bayesian updating. Our main technical device is a squared-loss surrogate for welfare maximization. We show that maximizing empirical welfare over a policy class with a quadratic penalty controlled by a tuning parameter $ζ>0$ is equivalent to minimizing a scaled squared error in the outcome difference. The resulting General Bayes posterior over decision rules admits two equivalent characterizations: a Gaussian pseudo-likelihood representation and a decision-theoretic loss-based characterization. As one implementation, we introduce GBPLNet, a neural network implementation with a tanh-squashed output. Finally, we establish PAC-Bayes-type guarantees for the surrogate risk and the corresponding penalized welfare.

Figures & tables

DGP	method	$\zeta$	welfare_mean	welfare_var	welfare_se	regret_mean	regret_se
DGP1	DiffReg	–	1.0272	0.0081	0.0090	0.2291	0.0033
	GBPLNet	0.01	1.0240	0.0084	0.0091	0.2323	0.0034
		0.1	1.0253	0.0081	0.0090	0.2310	0.0033
		1.0	1.0298	0.0083	0.0091	0.2265	0.0034
		CV	1.0295	0.0083	0.0091	0.2268	0.0033
	PluginReg	–	1.0265	0.0085	0.0092	0.2298	0.0034

Table 1: Synthetic binary results,

100

trials. Columns report welfare mean, welfare variance, welfare standard error, regret mean, and regret standard error across trials.

Panel of Figure 1 — Figure 1: Synthetic binary welfare boxplots across $100$ trials for DGP1, DGP2, and DGP3.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: GBPLNet posterior visualization in the one-dimensional binary example. The plot shows posterior draws of the score function $f_{w}\left(x\right)$ , the posterior mean, and a pointwise $95\%$ credible band, along with the population target $\max\left(-1,\min\left(1,\tau\left(x\right)/\zeta\right)\right)$ .

Figure 3: Posterior distribution of test welfare in the one-dimensional binary example. The histogram is computed from SGLD draws of $w$ and welfare evaluation under the induced deterministic policy.

Figure 4: Posterior distributions of $f_{w}\left(x_{0}\right)$ at two fixed covariate values $x_{0}\in\left\{1,2\right\}$ in the one-dimensional binary example. These histograms show posterior variation in the score at the selected covariate values.

Figure 5: Full-feedback learning curves for binary DGP2 over

100

replications. The left panel reports welfare, and the right panel reports regret. Error bars show Monte Carlo standard errors.

$n$	Method	$\zeta$	Welfare	Regret	Agreement
100	GBPLNet	CV	0.1157 (0.0072)	0.1566 (0.0072)	0.7730 (0.0065)
	DirectWelfare	–	0.1182 (0.0070)	0.1541 (0.0070)	0.7752 (0.0063)
	DiffReg	–	0.0611 (0.0100)	0.2112 (0.0100)	0.7238 (0.0088)
	PluginReg	–	0.0965 (0.0070)	0.1757 (0.0070)	0.7551 (0.0063)
	WeightedLogistic	–	0.1162 (0.0075)	0.1561 (0.0075)	0.7729 (0.0069)
500	GBPLNet	CV	0.2233 (0.0015)	0.0490 (0.0013)	0.8860 (0.0018)

Appendix

Table 2: Full-feedback DGP2 learning-curve summary over

100

replications. Entries are means followed by Monte Carlo standard errors in parentheses.

Figure 6: Logged-feedback learning curves for binary DGP2 over

50

replications. The left panel reports welfare, and the right panel reports regret. Error bars show Monte Carlo standard errors.

$n$	Method	$\zeta$	Welfare	Regret	Agreement
100	GBPLNet-DR	CV	-0.0356 (0.0146)	0.3068 (0.0147)	0.6405 (0.0126)
	GBPLNet-IPW	CV	-0.0480 (0.0140)	0.3192 (0.0140)	0.6299 (0.0119)
	DirectDRWelfare	–	-0.0397 (0.0140)	0.3109 (0.0141)	0.6367 (0.0120)
	DirectIPWWelfare	–	-0.0611 (0.0151)	0.3323 (0.0150)	0.6185 (0.0129)
	PluginReg	–	-0.0347 (0.0147)	0.3060 (0.0146)	0.6411 (0.0124)
500	GBPLNet-DR	CV	0.1708 (0.0056)	0.1004 (0.0057)	0.8267 (0.0057)

Appendix

Table 3: Logged-feedback DGP2 learning-curve summary over

50

replications. Entries are means followed by Monte Carlo standard errors in parentheses.

Figure 7: Weak-overlap diagnostics over

50

replications. The left panel reports welfare by clipping level. The right panel reports the

0.99

quantile of the absolute pseudo-difference. Error bars show Monte Carlo standard errors.

Clip	Method	$\zeta$	Welfare	Regret	Clipped fraction	ESS	$q_{0.99}\left(\left\|\widetilde{U}\right\|\right)$	$\max_{i}\left\|\widetilde{U}_{i}\right\|$
0.01	GBPLNet-DR	CV	0.2341 (0.0015)	0.0371 (0.0010)	0.0684	1090.1	13.6938	179.0290
	GBPLNet-IPW	CV	0.2125 (0.0046)	0.0587 (0.0042)			18.1773	196.5811
	PluginReg	–	0.2390 (0.0015)	0.0323 (0.0008)			–	–
0.05	GBPLNet-DR	CV	0.2363 (0.0017)	0.0349 (0.0011)	0.2433	2071.9	12.9393	57.6668
	GBPLNet-IPW	CV	0.2022 (0.0055)	0.0690 (0.0052)			17.2457	66.3725
	PluginReg	–	0.2390 (0.0015)	0.0323 (0.0008)			–	–

Appendix

Table 4: Weak-overlap diagnostics over

50

replications. Welfare and regret are means followed by Monte Carlo standard errors in parentheses. The clipped fraction, effective sample size (ESS),

q_{0.99}

, and maximum are computed from the training portion of the logged sample.

$\eta$	$50\%$	$80\%$	$90\%$	$95\%$
0.25	0.93 (0.0479)	1.00 (0.1114)	1.00 (0.1670)	1.00 (0.2219)
0.5	0.78 (0.0376)	0.99 (0.0822)	0.99 (0.1195)	1.00 (0.1566)
1	0.69 (0.0274)	0.95 (0.0604)	0.99 (0.0869)	0.99 (0.1152)
2	0.63 (0.0195)	0.88 (0.0426)	0.97 (0.0615)	0.99 (0.0830)
4	0.54 (0.0135)	0.80 (0.0289)	0.90 (0.0413)	0.93 (0.0545)
8	0.47 (0.0093)	0.72 (0.0192)	0.80 (0.0265)	0.84 (0.0342)

Appendix

Table 5: Welfare-interval calibration over

100

replications. Each entry is empirical coverage followed by mean interval width in parentheses.

Figure 8: Empirical welfare-interval coverage on the

100

calibration replications.

$\eta$	$50\%$	$80\%$	$90\%$	$95\%$
1	0.750 (0.0283)	0.945 (0.0624)	0.980 (0.0913)	0.990 (0.1231)
4	0.635 (0.0139)	0.855 (0.0300)	0.910 (0.0433)	0.950 (0.0575)

Appendix

Table 6: Independent welfare-interval evaluation over

200

replications. Each entry is empirical coverage followed by mean interval width in parentheses.

Figure 9: Independent welfare-interval evaluation over

200

replications. The dashed line is nominal coverage.

Group	Points	Disagreement	Agreement	Margin	Regret
1	391154	0.0023 (0.0001)	0.9955 (0.0005)	0.8822 (0.0002)	0.0002 (0.0000)
2	3026	0.2923 (0.0005)	0.7176 (0.0173)	0.0863 (0.0032)	0.0165 (0.0017)
3	2210	0.4958 (0.0006)	0.6568 (0.0153)	0.0802 (0.0037)	0.0219 (0.0018)
4	1859	0.6972 (0.0007)	0.5799 (0.0122)	0.0762 (0.0039)	0.0278 (0.0019)
5	1751	0.8992 (0.0006)	0.5225 (0.0103)	0.0748 (0.0040)	0.0348 (0.0023)

Appendix

Table 7: Posterior action disagreement over

200

replications. Points is the pooled number of evaluation points in each group. The remaining entries are averages of replication-level group means followed by Monte Carlo standard errors in parentheses.

Figure 10: Posterior action disagreement over

200

replications. The left panel reports agreement with the Bayes-optimal action. The right panel reports the regret contribution. Error bars show Monte Carlo standard errors.

$\alpha$	Welfare	5th pct.	MAP	Difference	Uncertain	Change
0.05	0.4309 (0.0001)	0.4278	0.4315	-0.0006 (0.0001)	0.0285	0.0144
0.10	0.4311 (0.0001)	0.4287	0.4315	-0.0004 (0.0001)	0.0222	0.0112
0.20	0.4313 (0.0001)	0.4293	0.4315	-0.0002 (0.0001)	0.0146	0.0074
0.30	0.4314 (0.0001)	0.4293	0.4315	-0.0001 (0.0000)	0.0090	0.0046

Appendix

Table 8: Posterior-threshold rules over

200

replications. Welfare and Difference are means followed by Monte Carlo standard errors in parentheses. The fifth percentile is computed across replications.

Figure 11: Posterior-threshold rules over

200

replications. The vertical axis reports the welfare difference from the MAP rule.

dataset	method	$\zeta$	welfare_mean	welfare_var	welfare_se	regret_mean	regret_se
DGP1	DirectWelfare	–	1.3366	0.0117	0.0108	0.4972	0.0066
	GBPLNet	0.01	1.3404	0.0115	0.0107	0.4934	0.0063
		0.1	1.3432	0.0105	0.0102	0.4906	0.0056
		1.0	1.3417	0.0095	0.0097	0.4921	0.0050
		CV	1.3588	0.0098	0.0099	0.4751	0.0051
DGP2	DirectWelfare	–	1.4239	0.0096	0.0098	0.4850	0.0052

Appendix

Table 9: Synthetic

K=5

results,

100

trials. Columns report welfare mean, welfare variance, welfare standard error, regret mean, and regret standard error across trials.

Figure 12: Synthetic

K=5

welfare means with Monte Carlo standard errors across

100

trials for DGP1, DGP2, and DGP3.

dataset	method	$\zeta$	welfare_mean	welfare_var	welfare_se	regret_mean	regret_se
energy_efficiency	DiffReg	–	0.3569	0.0295	0.0172	0.0001	0.0000
	GBPLNet	0.01	0.3565	0.0295	0.0172	0.0004	0.0001
		0.1	0.3568	0.0295	0.0172	0.0002	0.0001
		1.0	0.3568	0.0295	0.0172	0.0002	0.0000
		CV	0.3568	0.0295	0.0172	0.0002	0.0000
	PluginReg	–	0.3567	0.0296	0.0172	0.0002	0.0000

Appendix

Table 10: UCI and OpenML results, binary actions,

100

trials. Columns report welfare mean, welfare variance, welfare standard error, regret mean, and regret standard error across trials.

Panel of Figure 13 — Figure 13: UCI and OpenML welfare boxplots across $100$ trials for yacht and energy_efficiency.

dgp	method	$\zeta$	welfare_mean	welfare_var	welfare_se	regret_mean	regret_se
DGP1	GBPLNet-DR	0.01	1.0184	0.0086	0.0131	0.2336	0.0055
		0.1	1.0161	0.0082	0.0128	0.2359	0.0051
		1.0	1.0212	0.0088	0.0133	0.2309	0.0054
		CV	1.0211	0.0081	0.0127	0.2309	0.0050
	GBPLNet-IPW	0.01	1.0041	0.0085	0.0130	0.2479	0.0054
		0.1	1.0002	0.0082	0.0128	0.2518	0.0054

Appendix

Table 11: Counterfactual synthetic binary results,

50

trials. Columns report welfare mean, welfare variance, welfare standard error, regret mean, and regret standard error across trials. GBPLNet-DR and GBPLNet-IPW use DR and IPW pseudo-outcome differences, respectively. For PluginReg, welfare standard errors are computed as the square root of the reported welfare variance divided by the number of trials; a dash indicates that a regret standard error is not reported.

DGP	method	$\zeta$	welfare_mean	welfare_var	welfare_se	regret_mean	regret_se
DGP1	GBPLNet-DR	0.01	1.3312	0.0112	0.0150	0.5122	0.0081
		0.1	1.3467	0.0127	0.0159	0.4967	0.0084
		1.0	1.3313	0.0124	0.0157	0.5122	0.0079
		CV	1.3325	0.0125	0.0158	0.5109	0.0082
	GBPLNet-IPW	0.01	1.3081	0.0130	0.0161	0.5353	0.0096
		0.1	1.3139	0.0125	0.0158	0.5296	0.0084

Appendix

Table 12: Counterfactual synthetic

K=5

results,

50

trials. Columns report welfare mean, welfare variance, welfare standard error, regret mean, and regret standard error across trials. GBPLNet-DR and GBPLNet-IPW use DR and IPW pseudo-outcome vectors, respectively. For PluginRegK, welfare standard errors are computed as the square root of the reported welfare variance divided by the number of trials; a dash indicates that a regret standard error is not reported.

dataset	method	$\zeta$	welfare_mean	welfare_var	welfare_se	regret_mean	regret_se
energy_efficiency	GBPLNet-DR	0.01	0.3687	0.0422	0.0375	0.0020	0.0004
		0.1	0.3688	0.0422	0.0375	0.0019	0.0004
		1.0	0.3684	0.0424	0.0376	0.0023	0.0004
		CV	0.3680	0.0423	0.0376	0.0026	0.0005
	GBPLNet-IPW	0.01	0.3399	0.0462	0.0393	0.0307	0.0066
		0.1	0.3419	0.0479	0.0400	0.0288	0.0062

Appendix

Table 13: Counterfactual UCI and OpenML results, binary actions,

30

trials. Columns report welfare mean, welfare variance, welfare standard error, regret mean, and regret standard error across trials. For PluginReg, welfare standard errors are computed as the square root of the reported welfare variance divided by the number of trials; a dash indicates that a regret standard error is not reported.

Generalized Bayesian Inference

See all 8 papers

Wasserstein Policy Learning for Distributional Outcomes
Jun 17, 2026Yiyan Huang, Cheuk Hang Leung, Qi Wu +1Offline RL Wasserstein Barycenters
School of Statistics and Data Science & Institute of Big Data Research, Shanghai University of Finance and Economics, Shanghai, China
PAC-Bayesian Reward-Certified Outcome Weighted Learning
Apr 2, 2026Yuya Ishikawa, Shu Tamano
Department of Clinical Biostatistics, Graduate School of Medical and Dental Sciences, Institute of Science Tokyo, 1-5-45 Yushima, Bunkyo-ku, Tokyo, 113-8510, Japan · Biostatistics Division, Center for Research Administration and Support, National Cancer Center, 5-1-1 Tsukiji, Chuo-ku, Tokyo 104-0045, Japan · Department Multidisciplinary Sciences, Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, Meguro-Ku, Tokyo 153-8902, Japan +1
Policy Learning with Weak Signals
Oct 7, 2026Benedikt Koch, Winston Chou, Aurélien Bibaut +1Policy Learning Causal Effect Estimation
Harvard University · Netflix

Jun 17, 2026J/K move · Enter open · S save

Wasserstein Policy Learning for Distributional Outcomes

Yiyan Huang, Cheuk Hang Leung, Qi Wu, Zhiheng Zhang

School of Statistics and Data Science & Institute of Big Data Research, Shanghai University of Finance and Economics, Shanghai, China

Offline RL Wasserstein Barycenters Causal Inference

Open paper AskPDF arXiv

Offline policy learning has received growing attention in causal inference. The primary objective is to learn a policy (individualized treatment rule) as a mapping from covariates to treatment that maximizes the empirical welfare defined as the mean of scalar-valued potential outcomes. In this paper, we study offline policy learning with distribution-valued outcomes, where each potential outcome is a probability measure on

\mathbb{R}

and the reward is defined through a utility functional applied to the Wasserstein barycenter of induced outcome distributions. We establish statistical guarantees for the policy learning framework based on both Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators. By handling the challenging uniform deviation over the product of the combinatorial policy class and the infinite-dimensional quantile domain, we prove that the finite-sample regret has leading dependence

\widetilde{\mathcal{O}}(\sqrt{\mathrm{N\text{-}dim}(Π)/N})

. In the one-dimensional Wasserstein setting and under the stated regularity conditions, the leading regret rate is still governed by the policy-class complexity. Moreover, we provide a minimax lower bound establishing the sharpness of the leading dependence on

N

and

\mathrm{N\text{-}dim}(Π)

Abstract

Generalized Bayesian Inference

Explore similar work

Wasserstein Policy Learning for Distributional Outcomes

PAC-Bayesian Reward-Certified Outcome Weighted Learning

Policy Learning with Weak Signals