Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data

Rui Miao; Babak Shahbaba; Annie Qu

stat.MLMay 14, 2025

Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data

Authors: Rui Miao, Babak Shahbaba, Annie Qu

Organizations: National Heart, Lung, and Blood Institute · University of California, Irvine

Abstract

Offline reinforcement learning (RL) aims to find optimal policies in dynamic environments in order to maximize the expected total rewards by leveraging pre-collected data. Learning from heterogeneous data is one of the fundamental challenges in offline RL. Traditional methods focus on learning an optimal policy for all individuals with pre-collected data from a single episode or homogeneous batch episodes, and thus, may result in a suboptimal policy for a heterogeneous population. In this paper, we propose an individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs). The proposed heterogeneous model with individual latent variables enables us to efficiently estimate the individual Q-functions, and our Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on the average regret under a weak partial coverage assumption on behavior policies. In addition, our simulation studies and a real data application demonstrate the superior numerical performance of the proposed method compared with existing methods.

Figures & tables

Parameters	Subgroup (a)	Subgroup (b)	Subgroup (c)
$c_{1}^{i}$	0	0.6	-0.7
$c_{2}^{i}$	-0.6	0.4	0.5

Table 1: State transition parameters for individual

i

from different groups.

Figure 1: Boxplots for values of estimated policies for

n=30,60

and

T=50,100

and 200. Red dashed lines separate three groups (a), (b) and (c). In each group, the blue dotted line separates the P4L method of different

K

from the benchmark methods.

	CartPole (length/force)
Setting (A)	(2/0.85)	(5/0.85)	(10/0.85)
P4L $(K=2)$	58.0±12.7	72.8±16.8	121.1±31.7
P4L $(K=3)$	92.4±17.6	182.8 ±4.3	226.0 ±12.3
P4L $(K=4)$	74.3±16.8	179.6±8.3	165.5±29.5
P4L $(K=5)$	86.2±9.3	163.2±13.5	192.8±23.0
P4L $(K=Auto)$	98.2 ±12.1	174.2±6.9	213.0±19.0

Table 2: Values of learned policies from five RL methods on CartPole with different settings. Values in braces are standard errors. In Setting (A), we fix the force at 0.85 and let length vary from 2 to 10. In Setting (B), we fix the length at 5 and let force vary from 2 to 10. In Setting (C), we mix three different combinations of (length/force).

Method	Clinician	P4L	ACPI	VL	FQI
Value	-6.13 (1.82)	-5.40 (2.10)	-6.54 (2.31)	-7.32 (2.55)	-7.09 (2.65)

Table 3: Values of learning optimal policies with standard errors in parentheses.

Figure 3: Values of learned optimal policies estimated by PerSim.

Figure 4: The structure of the neural network in the empirical study.

Figure 5: Boxplots for values of estimated policies for

n=30,60

and

150

and

T=50,100

and 200. Red dashed lines separate three groups (a), (b) and (c). In each group, the blue dotted line separates the P4L method of different

K

from the benchmark methods.

Figure 6: MountainCar and CartPole environments.

	MountainCar (gravity)
Settings	(0.01)	(0.025)	(0.035)
P4L $(K=2)$	-106.3±44.1	-432.3±117	-471.8±43.0
P4L $(K=3)$	-56.80±1.83	-189.4 ±6.44	-210.6 ±4.27
P4L $(K=4)$	-74.23±16.5	-492.3±13.3	-500.0±0.0
P4L $(K=5)$	-67.60±22.3	-267.8±202	-295.1±180
P4L $(K=Auto)$	-63.20±4.3	-233.1±24.1	-263.1±28.0

Table 4: Values of learned policies from four RL methods on MountainCar with different settings.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
Jul 28, 2026Zeyu Bian, Ying Zhou, Yifan CuiCausal Effect Estimation Off-Policy Evaluation
Department of Statistics, Florida State University · Department of Statistics, University of Connecticut · Center for Data Science, Zhejiang University
Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
Feb 27, 2026Xiang Li, Yuheng Zhang, Nan JiangNatural Policy Gradient Offline RL
Nanjing University · UIUC
Peng's Q( $λ$ ) for Conservative Value Estimation in Offline Reinforcement Learning
May 14, 2026Byeongchan Kim, Min-hwan OhReinforcement Learning Value Function Estimation
Seoul National University

Jul 28, 2026J/K move · Enter open · S save

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

Zeyu Bian, Ying Zhou, Yifan Cui

Department of Statistics, Florida State University · Department of Statistics, University of Connecticut · Center for Data Science, Zhejiang University

Causal Effect Estimation Off-Policy Evaluation Offline RL

Open paper AskPDF arXiv

Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.

Abstract

Offline RL

Explore similar work

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies

Peng's Q(λλλ) for Conservative Value Estimation in Offline Reinforcement Learning

Peng's Q( $λ$ ) for Conservative Value Estimation in Offline Reinforcement Learning