Risk-Aware Reinforcement Learning for Mobile Manipulation

Michael Groom; James Wilson; Nick Hawes; Lars Kunze

cs.ROMar 4, 2026

Risk-Aware Reinforcement Learning for Mobile Manipulation

Authors: Michael Groom, James Wilson, Nick Hawes, Lars Kunze

Organizations: Oxford Robotics Institute, Department of Engineering Science, University of Oxford, Oxford, United Kingdom · Dyson Institute of Engineering & Technology, Malmesbury, United Kingdom · Bristol Robotics Laboratory, School of Engineering, University of West England, Bristol, United Kingdom

Abstract

For robots to successfully transition from lab settings to everyday environments, they must begin to reason about the risks associated with their actions and make informed, risk-aware decisions. This is particularly true for robots performing mobile manipulation tasks, which involve both interacting with and navigating within dynamic, unstructured spaces. However, existing whole-body controllers for mobile manipulators typically lack explicit mechanisms for risk-sensitive decision-making under uncertainty. To our knowledge, we are the first to (i) learn risk-aware visuomotor policies for mobile manipulation conditioned on egocentric depth observations with runtime-adjustable risk sensitivity, and (ii) show risk-aware behaviours can be transferred through Imitation Learning (IL) to a visuomotor policy conditioned on egocentric depth observations. Our method achieves this by first training a privileged teacher policy using Distributional Reinforcement Learning (DRL), with a risk-neutral distributional critic. Distortion risk-metrics are then applied to the critic's predicted return distribution to calculate risk-adjusted advantage estimates used in policy updates to achieve a range of risk-aware behaviours. We then distil teacher policies with IL to obtain risk-aware student policies conditioned on egocentric depth observations. We perform extensive evaluations demonstrating that our trained visuomotor policies exhibit risk-aware behaviour (specifically achieving better worst-case performance) while performing reactive whole-body motions in unmapped environments, leveraging live depth observations for perception.

Figures & tables

Fig. 1 : Predicted value distributions from a risk-aware critic during a successful pick attempt at key time steps (PDFs via critic-predicted quantiles) shown for different risk attitudes: risk-neutral ( $\beta=0.0$ ), risk-seeking ( $\beta=-1.0$ ), and risk-averse ( $\beta=+1.0$ ). Risk sensitivity alters the perceived relative likelihood of outcomes, producing risk-aware behaviour. Note changing axis ranges.

Fig. 2 : An overview of our proposed framework. Phase 1: A risk-aware Distributional Reinforcement Learning teacher policy $\pi_{\theta}$ is trained. A critic predicts value distributions $Z_{\phi}(s)$ which are distorted by a risk metric to calculate a risk distorted expected value, which is used to update the teacher policy $\pi_{\theta}$ , which is also conditioned on the selected risk-sensitivity. Phase 2: A student policy $\pi_{\psi}$ conditioned on high-dimensional depth observations is learnt through Imitation Learning with the risk-aware teacher policy $\pi_{\theta}$ . The risk-sensitivity parameter $\beta$ is assumed to be provided by an external operator or planner at runtime.

Panel of Figure 3 — Fig. 3 : Training environments. Left: Navigation task, reaching a 3D target (axes) while avoiding static and dynamic obstacles. Right: Pick task, grasping and lifting a cube to a goal (red sphere).

Action Component	Navigation	Pick
Base cmd. vel. $\mathbf{v}^{\text{base}}_{t}=(v_{x},v_{y},\omega_{z})$	✓	✓
Arm joint positions $\mathbf{q}_{t}$	✓	✓
Gripper close/open $a_{t}^{\text{grip}}\in\{0,1\}$	—	✓

TABLE I : Actions per task (applied at 30 Hz).

Hyperparameter	Navigation Task	Pick Task
Learning rate	$1\times 10^{-3}$	$1\times 10^{-3}$
Discount factor $\gamma$	0.99	0.99
GAE( $\lambda$ ), $\lambda$	0.95	0.95
Batch size	128	256
Epochs per update	5	5
Clip parameter $\epsilon$	0.2	0.15

TABLE II : Hyperparameters for RL training across navigation and pick tasks. The same hyperparameters were used across all baselines trained on a task. No extensive hyperparamter searches were performed. Bracketed values denote student distillation settings.

Component	Noise	Params	Navigation			Pick
Component	Noise	Params	TA	TC	S	TA	TC	S
High-level / exteroception
Egocentric depth $\mathbf{d_{t}}$	$\mathcal{U}(-a,a)$	$a=0.1\,\mathrm{m}$	—	—	✓ *	—	—	✓ *
Height scan (1D) $\mathbf{\tilde{h}_{t}}$	—	—	✓	✓	—	✓	✓	—
Risk sensitivity $\mathbf{\beta}$	—	—	✓	—	✓	✓	—	✓
Proprioceptive state $s_{t}^{\text{robot}}$

TABLE III : Observations and noise terms. ✓= included; ✓ * = included with noise; — = not included. TA (Teacher Actor) and S (Students) can receive noise; TC (Teacher Critics) have no additive noise (following [ 34 ] ). Transforms are in the robot base frame; base pose/velocity via wheel odometry.

Task	Aspect	Item	Setting / Details	Trigger / When
Navigation	Reward schedule	Action rate $\\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\\|$	Update weight $w_{i}=-1\times 10^{-4}$	Iter. 400
		Arm joint velocity $\\|\dot{\mathbf{q}}^{\text{arm}}_{t}\\|$	Update weight $w_{i}=-2.5\times 10^{-3}$	Iter. 400
		Base velocity $\\|\mathbf{v}^{\text{base}}_{t}\\|$	Update weight $w_{i}=-5\times 10^{-3}$	Iter. 400
		EE acceleration $\\|\dot{\mathbf{v}}^{\text{ee}}_{t}\\|$	Update weight $w_{i}=-2.5\times 10^{-3}$	Iter. 800
	Level progression	Levels	10 total; tracked per environment	—
		Level change rules	Success: $+1$ (cap at max; at max sample levels uniformly). Failure/termination: $-1$ (min 0).	On episode outcome

TABLE IV : Curricula for navigation and pick teacher policies. Rows cover (i) reward weight updates and (ii) level/difficulty progression, including triggers and parameter ranges.

Component / Term	Formula	Weights $w_{i}$
		Navigation	Pick
EE goal	$15\cdot\,\mathbb{I}\big(\lVert\mathbf{x_{t}^{ee}}-\mathbf{g^{ee}}\rVert<0.15\,\mathrm{m}\big)$	10	—
EE shaped	$\lVert\mathbf{p_{t}^{ee}}-\mathbf{g^{ee}}\rVert-\lVert\mathbf{p_{t-1}^{ee}}-\mathbf{g^{ee}}\rVert$	10	—
EE–object	$1-\tanh\!\big(5\cdot\,\lVert\mathbf{p}_{t}^{ee}-\mathbf{p}_{t}^{obj}\rVert\big)$	—	1.0
Grasp success $\mathbb{I}_{t}^{\text{grasp}}$	$\displaystyle\mathbb{I}\Big(\ \lVert\mathbf{F}_{L,R}\rVert>\mathbf{F}_{\min}\ \wedge\ \angle(\mathbf{F}_{L,R},\mathbf{d}_{L,R})\leq\mathbf{\theta}_{\max}\ \Big)$	—	5.0
Obj.–goal	$1-\tanh\!\big(\lVert\mathbf{p}_{t}^{obj}-\mathbf{g^{goal}}\rVert\big)$	—	5.0

TABLE V : Reward terms formulas and weights.

r_{t}=\Delta t\sum_{i}w_{i}r_{t}^{(i)}

\Delta t=\text{simulation dt}=8.33{\times}10^{-3}

. Terms

\lVert\mathbf{F}_{L,R}\rVert

\mathbf{F}_{\min}

d_{L,R}

\theta_{\max}

represent absolute contact force in left/right gripper fingers, threshold contact force

(0.5N)

, the open directions of the gripper fingers, and minimum angle between contact force and open direction

(85^{\circ})

for a valid grasp.

T

is the number of timesteps per episode.

Panel of Figure 9 — Fig. 4 : Training curves for (a) navigation and (b) pick tasks. Teachers evaluate $16\times$ more steps per episode due to higher parallel environment counts ( $4096$ vs. $256$ ). The drop at episode 600 in (b) marks the switch to student-driven environment stepping.

Panel of Figure 10 — Fig. 5 : Task rates for policies evaluated on the navigation and pick tasks. (a) Left to right: task success rate; contact rate with the environment and the dynamic obstacle; cumulative return. (b) Left to right: task success rate; time out rate (not termination); time to reach goal success; time to task failure termination.

Fig. 6 : Navigation task curriculum progression. The privileged teacher (Scandots) makes stable progress, while the depth-based policy (Depth) fails to learn, likely due to noisy gradients from small batch sizes necessitated by the high VRAM cost of rendering many parallel high-dimensional depth images.

Fig. 7 : Difference in cumulative individual reward terms during the navigation task evaluation.

Fig. 8 : Plots of policies evaluated on a single instance of the pick task. Right: Cumulative return. Left:

20\%

Conditional Value at Risk of cumulative return.

20\%

is used as a balanced trade off between statistical reliability on the limited number of considered rollouts while placing emphasis on worst-case outcomes.

Risk-Sensitive RL

Risk-sensitive reinforcement learning optimizes policies for downside risk, tail outcomes, uncertainty, or safety constraints beyond expected reward.

See all 85 papers

Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots
Mar 21, 2026Kai Li, Shiyu ZhaoVision-Based Robot Control Mobile Robotics
College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.
StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception
May 11, 2026Evans Han, Yunfan Jiang, Yingke Wang +6Robot Policy Learning Visuomotor Policy Learning
1Stanford University · 2Northwestern University · 3Lambda, Inc
MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation
Sep 22, 2026Jinhe Tang, Ruixiao Dai, Weiming ZhiRobot Trajectory Tracking Robot Policy Learning
School of Computer Science, The University of Sydney, Australia

Mar 21, 2026J/K move · Enter open · S save

Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

Kai Li, Shiyu Zhao

College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

Vision-Based Robot Control Mobile Robotics Robot Navigation Cross-Modal Knowledge Distillation Feature-Based Knowledge Distillation

Open paper AskPDF arXiv

Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion. On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost. To address these issues, we propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy. The key idea is to train the student not only to mimic the expert actions but also to align with the latent embeddings of the omni view depth teacher. Experiments demonstrate that omni-view and depth inputs improve the scene transfer and navigation performance, and that the proposed distillation method enhances the performance of a singleview monocular policy, compared with policies solely imitating actions. Real world experiments further validate the effectiveness and practicality of our approach. Code will be released publicly.

Abstract

Risk-Sensitive RL

Explore similar work

Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation