Risk-Aware Reinforcement Learning for Mobile Manipulation

Michael Groom; James Wilson; Nick Hawes; Lars Kunze

cs.ROMar 4, 2026

Risk-Aware Reinforcement Learning for Mobile Manipulation

Authors: Michael Groom, James Wilson, Nick Hawes, Lars Kunze

Organizations: Oxford Robotics Institute, Department of Engineering Science, University of Oxford, Oxford, United Kingdom · Dyson Institute of Engineering & Technology, Malmesbury, United Kingdom · Bristol Robotics Laboratory, School of Engineering, University of West England, Bristol, United Kingdom

Abstract

For robots to successfully transition from lab settings to everyday environments, they must begin to reason about the risks associated with their actions and make informed, risk-aware decisions. This is particularly true for robots performing mobile manipulation tasks, which involve both interacting with and navigating within dynamic, unstructured spaces. However, existing whole-body controllers for mobile manipulators typically lack explicit mechanisms for risk-sensitive decision-making under uncertainty. To our knowledge, we are the first to (i) learn risk-aware visuomotor policies for mobile manipulation conditioned on egocentric depth observations with runtime-adjustable risk sensitivity, and (ii) show risk-aware behaviours can be transferred through Imitation Learning (IL) to a visuomotor policy conditioned on egocentric depth observations. Our method achieves this by first training a privileged teacher policy using Distributional Reinforcement Learning (DRL), with a risk-neutral distributional critic. Distortion risk-metrics are then applied to the critic's predicted return distribution to calculate risk-adjusted advantage estimates used in policy updates to achieve a range of risk-aware behaviours. We then distil teacher policies with IL to obtain risk-aware student policies conditioned on egocentric depth observations. We perform extensive evaluations demonstrating that our trained visuomotor policies exhibit risk-aware behaviour (specifically achieving better worst-case performance) while performing reactive whole-body motions in unmapped environments, leveraging live depth observations for perception.

Figures & tables

Fig. 1 : Predicted value distributions from a risk-aware critic during a successful pick attempt at key time steps (PDFs via critic-predicted quantiles) shown for different risk attitudes: risk-neutral ( $\beta=0.0$ ), risk-seeking ( $\beta=-1.0$ ), and risk-averse ( $\beta=+1.0$ ). Risk sensitivity alters the perceived relative likelihood of outcomes, producing risk-aware behaviour. Note changing axis ranges.

Fig. 2 : An overview of our proposed framework. Phase 1: A risk-aware Distributional Reinforcement Learning teacher policy $\pi_{\theta}$ is trained. A critic predicts value distributions $Z_{\phi}(s)$ which are distorted by a risk metric to calculate a risk distorted expected value, which is used to update the teacher policy $\pi_{\theta}$ , which is also conditioned on the selected risk-sensitivity. Phase 2: A student policy $\pi_{\psi}$ conditioned on high-dimensional depth observations is learnt through Imitation Learning with the risk-aware teacher policy $\pi_{\theta}$ . The risk-sensitivity parameter $\beta$ is assumed to be provided by an external operator or planner at runtime.

Panel of Figure 3 — Fig. 3 : Training environments. Left: Navigation task, reaching a 3D target (axes) while avoiding static and dynamic obstacles. Right: Pick task, grasping and lifting a cube to a goal (red sphere).

Action Component	Navigation	Pick
Base cmd. vel. $\mathbf{v}^{\text{base}}_{t}=(v_{x},v_{y},\omega_{z})$	✓	✓
Arm joint positions $\mathbf{q}_{t}$	✓	✓
Gripper close/open $a_{t}^{\text{grip}}\in\{0,1\}$	—	✓

TABLE I : Actions per task (applied at 30 Hz).

Hyperparameter	Navigation Task	Pick Task
Learning rate	$1\times 10^{-3}$	$1\times 10^{-3}$
Discount factor $\gamma$	0.99	0.99
GAE( $\lambda$ ), $\lambda$	0.95	0.95
Batch size	128	256
Epochs per update	5	5
Clip parameter $\epsilon$	0.2	0.15

TABLE II : Hyperparameters for RL training across navigation and pick tasks. The same hyperparameters were used across all baselines trained on a task. No extensive hyperparamter searches were performed. Bracketed values denote student distillation settings.

Component	Noise	Params	Navigation			Pick
Component	Noise	Params	TA	TC	S	TA	TC	S
High-level / exteroception
Egocentric depth $\mathbf{d_{t}}$	$\mathcal{U}(-a,a)$	$a=0.1\,\mathrm{m}$	—	—	✓ *	—	—	✓ *
Height scan (1D) $\mathbf{\tilde{h}_{t}}$	—	—	✓	✓	—	✓	✓	—
Risk sensitivity $\mathbf{\beta}$	—	—	✓	—	✓	✓	—	✓
Proprioceptive state $s_{t}^{\text{robot}}$

TABLE III : Observations and noise terms. ✓= included; ✓ * = included with noise; — = not included. TA (Teacher Actor) and S (Students) can receive noise; TC (Teacher Critics) have no additive noise (following [ 34 ] ). Transforms are in the robot base frame; base pose/velocity via wheel odometry.

Task	Aspect	Item	Setting / Details	Trigger / When
Navigation	Reward schedule	Action rate $\\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\\|$	Update weight $w_{i}=-1\times 10^{-4}$	Iter. 400
		Arm joint velocity $\\|\dot{\mathbf{q}}^{\text{arm}}_{t}\\|$	Update weight $w_{i}=-2.5\times 10^{-3}$	Iter. 400
		Base velocity $\\|\mathbf{v}^{\text{base}}_{t}\\|$	Update weight $w_{i}=-5\times 10^{-3}$	Iter. 400
		EE acceleration $\\|\dot{\mathbf{v}}^{\text{ee}}_{t}\\|$	Update weight $w_{i}=-2.5\times 10^{-3}$	Iter. 800
	Level progression	Levels	10 total; tracked per environment	—
		Level change rules	Success: $+1$ (cap at max; at max sample levels uniformly). Failure/termination: $-1$ (min 0).	On episode outcome

TABLE IV : Curricula for navigation and pick teacher policies. Rows cover (i) reward weight updates and (ii) level/difficulty progression, including triggers and parameter ranges.

Component / Term	Formula	Weights $w_{i}$
		Navigation	Pick
EE goal	$15\cdot\,\mathbb{I}\big(\lVert\mathbf{x_{t}^{ee}}-\mathbf{g^{ee}}\rVert<0.15\,\mathrm{m}\big)$	10	—
EE shaped	$\lVert\mathbf{p_{t}^{ee}}-\mathbf{g^{ee}}\rVert-\lVert\mathbf{p_{t-1}^{ee}}-\mathbf{g^{ee}}\rVert$	10	—
EE–object	$1-\tanh\!\big(5\cdot\,\lVert\mathbf{p}_{t}^{ee}-\mathbf{p}_{t}^{obj}\rVert\big)$	—	1.0
Grasp success $\mathbb{I}_{t}^{\text{grasp}}$	$\displaystyle\mathbb{I}\Big(\ \lVert\mathbf{F}_{L,R}\rVert>\mathbf{F}_{\min}\ \wedge\ \angle(\mathbf{F}_{L,R},\mathbf{d}_{L,R})\leq\mathbf{\theta}_{\max}\ \Big)$	—	5.0
Obj.–goal	$1-\tanh\!\big(\lVert\mathbf{p}_{t}^{obj}-\mathbf{g^{goal}}\rVert\big)$	—	5.0

TABLE V : Reward terms formulas and weights.

r_{t}=\Delta t\sum_{i}w_{i}r_{t}^{(i)}

\Delta t=\text{simulation dt}=8.33{\times}10^{-3}

. Terms

\lVert\mathbf{F}_{L,R}\rVert

\mathbf{F}_{\min}

d_{L,R}

\theta_{\max}

represent absolute contact force in left/right gripper fingers, threshold contact force

(0.5N)

, the open directions of the gripper fingers, and minimum angle between contact force and open direction

(85^{\circ})

for a valid grasp.

T

is the number of timesteps per episode.

Panel of Figure 9 — Fig. 4 : Training curves for (a) navigation and (b) pick tasks. Teachers evaluate $16\times$ more steps per episode due to higher parallel environment counts ( $4096$ vs. $256$ ). The drop at episode 600 in (b) marks the switch to student-driven environment stepping.

Panel of Figure 10 — Fig. 5 : Task rates for policies evaluated on the navigation and pick tasks. (a) Left to right: task success rate; contact rate with the environment and the dynamic obstacle; cumulative return. (b) Left to right: task success rate; time out rate (not termination); time to reach goal success; time to task failure termination.

Fig. 6 : Navigation task curriculum progression. The privileged teacher (Scandots) makes stable progress, while the depth-based policy (Depth) fails to learn, likely due to noisy gradients from small batch sizes necessitated by the high VRAM cost of rendering many parallel high-dimensional depth images.

Fig. 7 : Difference in cumulative individual reward terms during the navigation task evaluation.

Fig. 8 : Plots of policies evaluated on a single instance of the pick task. Right: Cumulative return. Left:

20\%

Conditional Value at Risk of cumulative return.

20\%

is used as a balanced trade off between statistical reliability on the limited number of considered rollouts while placing emphasis on worst-case outcomes.

Risk-Sensitive RL

Risk-sensitive reinforcement learning optimizes policies for downside risk, tail outcomes, uncertainty, or safety constraints beyond expected reward.

See all 85 papers

Mar 21, 2026

Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion. On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost. To address these issues, we propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy. The key idea is to train the student not only to mimic the expert actions but also to align with the latent embeddings of the omni view depth teacher. Experiments demonstrate that omni-view and depth inputs improve the scene transfer and navigation performance, and that the proposed distillation method enhances the performance of a singleview monocular policy, compared with policies solely imitating actions. Real world experiments further validate the effectiveness and practicality of our approach. Code will be released publicly.

Kai Li, Shiyu Zhao

College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

Vision-Based Robot Control Mobile Robotics Robot Navigation Cross-Modal Knowledge Distillation Feature-Based Knowledge Distillation

Ask PDF

May 11, 2026

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.

Evans Han, Yunfan Jiang, Yingke Wang +6

1Stanford University · 2Northwestern University · 3Lambda, Inc

Robot Policy Learning Visuomotor Policy Learning Robotic Manipulation Stereo Vision

Ask PDF

Sep 22, 2026

MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation

Successful mobile manipulation requires coordinated base and arm motion while maintaining accurate spatial positioning. However, demonstration-trained policies can struggle to realise the intended base motion reliably, leading to spatial misalignment and subsequent manipulation failures. We present MAVP (Map-Aware Visuomotor Policies), a framework that improves execution reliability by predicting explicit base-pose targets and tracking them using localisation feedback. MAVP reconstructs a static map from teleoperated demonstrations and expresses demonstrated base trajectories in a shared map frame, providing consistent spatial supervision across demonstrations. At execution time, the policy receives RGB observations, joint states, and the robot's current map-frame base pose, and jointly predicts target base poses, arm actions, and gripper actions. A low-level controller tracks the predicted base targets using feedforward motion and pose error feedback, enabling correction of execution deviations. We additionally use pose-noise augmentation during training to improve robustness to errors in the policy's pose input. Across six real-world manipulation tasks and three policy families, MAVP achieves higher task success rates than unanchored velocity control in all tasks. Videos and additional results are available at https://123qwedsa123.github.io/mavp/.

Jinhe Tang, Ruixiao Dai, Weiming Zhi

School of Computer Science, The University of Sydney, Australia

Robot Trajectory Tracking Robot Policy Learning Mobile Manipulation Visuomotor Policy Learning

Ask PDF

Abstract

Risk-Sensitive RL

Explore similar work

Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

MAVP: Map-Aware Visuomotor Policies for Mobile Manipulation