Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning

Tatjana Krau; Jorge Mandlmaier; Tobias Damm; Frieder Heieck

cs.LGMar 10, 2026

Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning

Authors: Tatjana Krau, Jorge Mandlmaier, Tobias Damm, Frieder Heieck

Organizations: Institute of Production and Informatics, 87527 Sonthofen, Germany · Department of Mathematics, RPTU University Kaiserslautern-Landau, 67663 Kaiserslautern, Germany

Abstract

Reinforcement Learning (RL) has demonstrated strong potential for industrial process control, yet policies trained in simulation often suffer from a significant sim-to-real gap when deployed on physical hardware. This work systematically analyzes how core Markov Decision Process (MDP) design choices -- state composition, target inclusion, reward formulation, termination criteria, and environment dynamics models -- affect this transfer. Using a color mixing task, we evaluate different MDP configurations and mixing dynamics across simulation and real-world experiments. We validate our findings on physical hardware, demonstrating that physics-based dynamics models achieve up to 50% real-world success under strict precision constraints where simplified models fail entirely. Our results provide practical MDP design guidelines for deploying RL in industrial process control.

Figures & tables

Fig. 1 : Visual representation of the three base ink colors: cyan, magenta and yellow.

ID	Volume Encoding	Example	Description
0	$V_{\mathrm{sum}}$	$200\,\mu$ l	Absolute total volume
1	$[V_{1},V_{2},V_{3}]$	$[200,20,0]\,\mu$ l	Absolute per-channel
2	$V_{\mathrm{sum,rel}}$	$1.1$	Relative total volume
3	$[V_{1,\mathrm{rel}},V_{2,\mathrm{rel}},V_{3,\mathrm{rel}}]$	$[1.0,10.1,0]$	Relative per-channel
4	$[V_{1,\mathrm{ratio}},V_{2,\mathrm{ratio}},V_{3,\mathrm{ratio}}]$	$[0.909,0.091,0]$	Normalized ratios

TABLE I : State representations evaluated in this study. All states include

\mathbf{c}_{\mathrm{current}}\in[0,255]^{3}

and the volume encoding. States with target inclusion additionally contain

\mathbf{c}_{\mathrm{target}}\in[0,255]^{3}

. Example mixture:

200\,\mu

l Cyan +

20\,\mu

l Magenta.

Phase 1	Tested	Selected
Fixed: lerp , Noise, ARL , $T{=}20$ , $\tau{=}10$
Target inclusion	yes / no	yes
State	0 / 1 / 2 / 3 / 4	2, 3, 4
Reward	R1 / R2 / R3	R1, R2
Phase 2	Tested	Selected
Fixed: lerp , Noise, ARL , Target yes, State $\in\{2,3,4\}$ , Reward $\in\{\text{R1,R2}\}$

TABLE II : Overview of experimental phases. Left column lists parameters, middle column shows tested values, right column indicates selected configuration for subsequent phases.

Model	Phase	Target	State	Termination $T$	Tolerance $\tau$	Reward	Dynamics Model
M1	1	Yes	4	20	10	R1	lerp
M2	1	No	4	20	10	R1	lerp
M3	2	Yes	4	5	7.5	R1	lerp
M4	3	Yes	4	5	7.5	R1	KM
M5	3	Yes	4	5	7.5	R1	WGM

TABLE III : Models evaluated on hardware. Phase indicates where the configuration was determined. The remaining columns show the training parameters.

Phase	FP $\uparrow$	CV $\downarrow$	T7.5 $\downarrow$	NM $\downarrow$	CS $\uparrow$
1 - Target yes	9.75 $\pm$ 0.02	0.0119	15,018	0	0.998
1 - Target no	6.99 $\pm$ 0.23	0.07	178,858	20	0.93
2 - 10, 7.5	9.14 $\pm$ 0.06	0.03	28,672	1.7	0.94
2 - 5, 7.5	7.3 $\pm$ 0.14	0.06	92,842	14	0.75
3 - R1, State 4	6.81 $\pm$ 0.02	0.06	209,578	20.7	0.77
3 - R2	6.01 $\pm$ 0.05	0.08	–	21.3	0.71

TABLE IV : Simulation performance metrics for key configurations based on episode reward mean.

\uparrow

: higher is better;

\downarrow

: lower is better.

Target	lerp	KM	WGM
C1	11.3	15.0	13.0
C2	30.0	32.0	26.0
C3	36.6	40.0	33.0
C4	11.5	11.0	9.0

TABLE V : Minimum required tolerance

\tau_{\min}

for each target color to be reachable under each dynamics model.

Model	Metric	C1	C4	Avg
M1	$\bar{d}_{R}\pm\sigma$	12.98 $\pm$ 2.28	42.32 $\pm$ 2.62	17.10 $\pm$ 14.95
	$s_{R}$	5	5	4
	Succ (%)	0	0	43.75
M2	$\bar{d}_{R}\pm\sigma$	108.62 $\pm$ 1.86	46.74 $\pm$ 3.39	54.39 $\pm$ 32.45
	$s_{R}$	5	5	5
	Succ (%)	0	0	0

TABLE VI : Sim-to-Real Transfer Results.

\bar{d}_{R}\pm\sigma

: Mean final distance to target with the standard deviation,

s_{R}

: Steps taken in reality, Succ: Success in each run in %.

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 316 papers

Abstract Sim2Real through Approximate Information States
Apr 16, 2026Yunfu Deng, Yuhao Li, Josiah P. HannaSim-to-Real Transfer State Abstraction
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA
The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems
May 29, 2026Tobias Lademann, Théo Vincent, Jan Peters +1RL Control Partially Observable RL
Institute for Production Management, Technology and Machine Tools (PTW), Technical University of Darmstadt · DFKI GmbH, SAIROL · Department of Computer Science, Technical University of Darmstadt +1
Sim2Signal: Sim-to-Real Benchmarks for Traffic Signal Control
Sep 1, 2026Ferdous Al Rafi, Susrik Mukherjee, Latika Liladhar Dekate +6RL Benchmarks Sim-to-Real Transfer
Arizona State University

Apr 16, 2026J/K move · Enter open · S save

Abstract Sim2Real through Approximate Information States

Yunfu Deng, Yuhao Li, Josiah P. Hanna

Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA

Sim-to-Real Transfer State Abstraction Robotic RL

Open paper AskPDF arXiv

In recent years, reinforcement learning (RL) has shown remarkable success in robotics when a fast and accurate simulator is available for a given task. When using RL and simulation, more simulator realism is generally beneficial but becomes harder to obtain as robots are deployed in increasingly complex and widescale domains. In such settings, simulators will likely fail to model all relevant details of a given target task and this observation motivates the study of sim2real with simulators that leave out key task details. In this paper, we formalize and study the abstract sim2real problem: given an abstract simulator that models a target task at a coarse level of abstraction, how can we train a policy with RL in the abstract simulator and successfully transfer it to the real-world? Our first contribution is to formalize this problem using the language of state abstraction from the RL literature. This framing shows that an abstract simulator can be grounded to match the target task if the grounded abstract dynamics take the history of states into account. Based on the formalism, we then introduce a method that uses real-world task data to correct the dynamics of the abstract simulator. We then show that this method enables successful policy transfer both in sim2sim and sim2real evaluation.

Abstract

Sim-to-Real Transfer

Explore similar work

Abstract Sim2Real through Approximate Information States

The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems

Sim2Signal: Sim-to-Real Benchmarks for Traffic Signal Control