Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes

Mingyuan Xu; Zongqi Xia; Tianxi Cai; Doudou Zhou; Nian Si

stat.MEFeb 2, 2026

Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes

Authors: Mingyuan Xu, Zongqi Xia, Tianxi Cai, Doudou Zhou, Nian Si

Organizations: Department of Statistics and Data Science, National University of Singapore · Department of Neurology, University of Pittsburgh · Department of Biostatistics, Harvard T.H. Chan School of Public Health · Department of Biomedical Informatics, Harvard Medical School ·

Abstract

Learning a common policy from multiple source environments requires accounting for cross-source heterogeneity, and the offline setting introduces an additional challenge of limited data coverage. We formulate this problem as a distributionally robust Markov decision process under a shared linear MDP representation with source-specific transition and reward components. The uncertainty set is constructed through feature-wise mixtures of these components, yielding a $d$ -rectangular structure that captures cross-source heterogeneity and admits a robust Bellman recursion. Building on this formulation, we develop Group-Robust Pessimistic Value Iteration (GR-PEVI), an offline policy-learning algorithm that combines feature-wise worst-case aggregation with a data-dependent pessimism penalty. Under a robust partial coverage condition, we establish finite-sample suboptimality guarantees for GR-PEVI and extend the analysis to clustered sources, characterizing the trade-off between the statistical gain from pooling similar sources and the approximation error induced by within-cluster heterogeneity. Simulation studies demonstrate the effectiveness of GR-PEVI, with empirical behavior consistent with the theoretical analysis. We apply GR-PEVI to a multi-source electronic health record study of multiple sclerosis treatment decisions to illustrate its use in a clinical setting.

Figures & tables

Figure 1: Comparison of robust suboptimality across methods: (A) Distribution of suboptimality over 50 replications. (B) Mean suboptimality with 95% confidence intervals.

Figure 2: Worst-case robust suboptimality

\widehat{\mathcal{E}}(c,N_{\min})

versus

N_{\min}

for different values of

c

, based on 500 replications. Error bars and slope intervals are 95% bootstrap confidence intervals. The dotted line shows a reference

N_{\min}^{-1/2}

rate, and the dashed line shows the worst-case robust suboptimality of the uniform random policy.

Characteristic	Overall	MGB-young	MGB-old	UPMC
Patients, $n$	5,700	2,370	1,925	1,405
Female sex, $n$ (%)	4,244 (74.5%)	1,750 (73.8%)	1,398 (72.6%)	1,096 (78.0%)
Age, mean (SD)	40.9 (11.4)	32.3 (6.5)	51.3 (6.7)	41.2 (11.0)
White race, $n$ (%)	4,943 (86.9%)	1,941 (81.9%)	1,748 (90.8%)	1,254 (90.0%)
Standard-efficacy DMT, $n$ (%)	4,357 (76.4%)	1,851 (78.1%)	1,459 (75.8%)	1,047 (74.5%)
High-efficacy DMT, $n$ (%)	1,343 (23.6%)	519 (21.9%)	466 (24.2%)	358 (25.5%)

Table 1: Patient characteristics by source. Continuous variables are reported as mean (SD), and categorical variables as

n

(%).

Method	Mean	SD	Median	Win count	GR-PEVI win rate
GR-PEVI	3.2042	0.0760	3.2107	30/50	–
Operator-Avg PEVI	3.1535	0.0803	3.1594	7/50	72%
Avg-PEVI	3.1463	0.0814	3.1364	6/50	82%
Majority-PEVI	3.1319	0.1050	3.1287	7/50	76%
Pooled PEVI	3.0151	0.0803	3.0276	0/50	98%

Table 2: Worst-source WAIPW over 50 train–test splits. SD is computed across splits; win count is the number of splits with the highest value; GR-PEVI win rate is the proportion of paired splits in which GR-PEVI outperformed each baseline.

Method	MGB-young	MGB-old	UPMC
GR-PEVI	$\bm{+3.04}$ (1.85)	$+1.99$ (2.51)	$\bm{+3.83}$ (4.34)
Operator-Avg PEVI	$-1.07$ (2.08)	$\bm{+2.72}$ (2.72)	$+3.14$ (4.76)
Avg-PEVI	$+0.37$ (2.10)	$+0.80$ (2.98)	$+3.11$ (5.58)
Majority-PEVI	$-0.07$ (2.57)	$+0.26$ (3.69)	$+3.13$ (6.11)
Pooled PEVI	$-3.08$ (2.18)	$-3.32$ (2.25)	$-3.44$ (3.30)

Table 3: Source-specific relative improvements (%) over the observed behavior policy across 50 train–test splits. Entries report the mean (SD) across splits; bold entries indicate the largest mean in each source.

Method	Main setting	Milder transition heterogeneity
GR-PEVI	0.017 (0.007)	0.020 (0.006)
Pooled PEVI	0.329 (0.042)	0.274 (0.034)
Avg-PEVI	0.951 (0.021)	0.872 (0.020)
Majority-PEVI	0.895 (0.022)	0.778 (0.025)
Operator-Avg PEVI	0.242 (0.021)	0.196 (0.015)

Table S1 : Robust suboptimality under the main and milder transition-heterogeneity settings. Entries are means over 50 Monte Carlo replications, with standard deviations in parentheses. Lower values indicate better performance.

Figure S1 : Comparison of robust suboptimality across methods under increasing within-cluster heterogeneity. Mean suboptimality with 95% confidence intervals over

R=50

replications.

Generic name	Brand name(s)	RxNorm ID
High efficacy
Natalizumab	Tysabri	354770
Ocrelizumab	Ocrevus	1876366
Ofatumumab	Arzerra; Kesimpta	712566
Mitoxantrone	Novantrone	7005
Alemtuzumab	Campath; Lemtrada	117055

Table S2 : Disease-modifying therapies for multiple sclerosis grouped by efficacy category.

Method	Mean	SD	Median	Win count	GR-PEVI win rate
GR-PEVI	4.6629	0.0247	4.6576	38/50	–
Operator-Avg PEVI	4.6345	0.0128	4.6369	6/50	86%
Majority-PEVI	4.6269	0.0237	4.6277	4/50	84%
Avg-PEVI	4.6095	0.0348	4.6187	2/50	92%
Pooled PEVI	4.3767	0.0577	4.3824	0/50	100%

Table S3 : Worst-source WAIPW over 50 train–test splits. SD is computed across splits; win count is the number of splits with the highest value; GR-PEVI win rate is the proportion of paired splits in which GR-PEVI outperformed each baseline.

Method	MICU/SICU	TSICU	SICU	MICU	CCU	CVICU
GR-PEVI	$+1.61$ (0.65)	$\mathbf{+2.07}$ (0.89)	$\mathbf{+2.18}$ (0.71)	$+1.75$ (0.55)	$+1.32$ (0.59)	$\mathbf{+1.90}$ (0.62)
Operator-Avg PEVI	$+0.70$ (0.25)	$+0.92$ (0.45)	$+0.64$ (0.39)	$+1.22$ (0.24)	$\mathbf{+2.71}$ (0.79)	$+0.53$ (0.30)
Avg-PEVI	$+1.30$ (0.34)	$+0.32$ (0.96)	$+0.84$ (0.73)	$+1.59$ (0.45)	$+1.98$ (1.15)	$+0.68$ (0.44)
Majority-PEVI	$+1.56$ (0.28)	$+0.65$ (0.68)	$+0.80$ (0.51)	$+1.69$ (0.31)	$+2.35$ (0.83)	$+0.72$ (0.31)
Pooled PEVI	$\mathbf{+1.86}$ (0.43)	$-5.05$ (1.24)	$+0.45$ (0.86)	$\mathbf{+2.26}$ (0.46)	$+2.55$ (0.82)	$+1.53$ (0.54)

Table S4 : Source-specific relative improvements (%) over the observed behavior policy across 50 train–test splits. Entries report the mean (SD) across splits; bold entries indicate the largest mean in each source.

Robust Markov Decision Processes

Robust Markov decision processes optimize decisions under uncertain dynamics, seeking reliable performance against adverse transition models.

See all 27 papers

Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions
Aug 6, 2026Yuepeng Yang, Yuxin Chen, Yuejie ChiRobust Markov Decision Processes Average-Reward RL
*Department of Statistics and Data Science, Yale University. · †Department of Statistics and Data Science, the Wharton School, University of Pennsylvania.
Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data
May 14, 2025Rui Miao, Babak Shahbaba, Annie QuOffline RL Policy Learning
National Heart, Lung, and Blood Institute · University of California, Irvine
Distributionally-Robust Policy Learning with Observational Data
Oct 18, 2026Debmalya MandalPolicy Learning Distribution Shift Robustness
Dept. of Computer Science University of Warwick, UK

Aug 6, 2026J/K move · Enter open · S save

Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions

Yuepeng Yang, Yuxin Chen, Yuejie Chi

*Department of Statistics and Data Science, Yale University. · †Department of Statistics and Data Science, the Wharton School, University of Pennsylvania.

Robust Markov Decision Processes Average-Reward RL Markov Models Minimax Estimation Distributionally Robust RL Sample-Efficient RL

Open paper AskPDF arXiv

Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty. We study how many samples are necessary and sufficient to learn an

\varepsilon

-optimal robust policy under the average-reward criterion. A generative model provides samples from the nominal transition kernel, whereas policy performance is evaluated over

(s,a)

-rectangular total-variation uncertainty sets of radius at most

σ

. Let

H_0

and

H_σ

denote the nominal and robust optimal bias spans, respectively. We identify

σH_0

as the perturbation scale separating high- and low-tolerance regimes. Our matching upper and lower bounds show that, up to logarithmic factors, the minimax total sample complexity is

NSA \asymp \frac{SA}{\varepsilon^2}\begin{cases} \min\{H_0,H_σ\}, & \varepsilon\gtrsimσH_0,\\ \min\{H_0,H_σ\}+σH_σ^2, & \varepsilon\lesssimσH_0. \end{cases}

Here

S

and

A

are the numbers of states and actions, and

N

is the number of samples per state-action pair. The sample complexity consists of a linear-span term that resembles the nominal AMDP results and a robustness-specific term that appears only in the low-tolerance regime. We attain these rates using reduction-based plug-in procedures that select the reduction---nominal or robust---and its discount factor: a span-informed procedure that makes these choices using known span parameters, and a span-agnostic procedure that calibrates both choices from data.

Abstract

Robust Markov Decision Processes

Explore similar work

Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions

Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data

Distributionally-Robust Policy Learning with Observational Data