For practical motion generation, it is essential to handle multimodality (multiple actions can be correct for the same observation) while maintaining low inference cost and strong generalization from limited demonstrations. We introduce an attentive mixture-density policy trained with the Energy Score that represents future visuomotor trajectories as explicit probability distributions with a single forward pass. Concretely, we output Gaussian mixture parameters for joint commands (and optionally attention points) and optimize them with the Energy Score (computed as a sum of 1D CRPS under a diagonal covariance assumption) instead of the negative log-likelihood (NLL) to avoid MDN singularities and gradient-scale instabilities. Our architecture couples explicit spatial attention with mixture-density outputs, enabling robust multimodal behavior selection under visual noise and small-data regimes. On an 8-DOF manipulator, the method generates distinct avoidance trajectories and stabilizes learning in obstacle avoidance. Moreover, in a randomized moving-object grasping task, where spatiotemporal multimodality and fast timing decisions are required, the proposed model outperforms Diffusion Policy by 15-25% under several unseen conditions, demonstrating superior generalization and real-time feasibility.
Figures & tables
Fig. 1 : Architecture of the proposed model. The output layer (green box) represents the multimodality of future predictions by outputting joint angles and attention points as GMM parameters ( π,μ,σ ). The Energy Score (CRPS) is adopted as the loss function. The yellow box indicates the reconstruction process optimized for the modified prediction format.
Fig. 2 : Left: Experimental setup for Exp. 1. The target object is positioned behind an obstacle. Right: Movement patterns of the moving object in Exp. 2. During training, four reciprocating patterns are used: vertical, horizontal, right-diagonal, and left-diagonal. During inference, an additional unlearned pattern with random turn-back angles is introduced.
Fig. 3 : Comparison of Exp. 1 results. The left panel shows the success rate, and the right panel shows the failure rate and its breakdown. The left graph confirms that the proposed model achieves success rates equivalent to or higher than representative conventional models such as ACT and DP. In the right graph, the area enclosed by the crimson frame indicates the proportion of ”mode averaging” generated; it can be seen that this proportion is significantly reduced in the proposed model compared to SARNN without GMM.
Fig. 4 : PCA visualization of internal states in Exp. 1 using only successful grasping trials. We plot nine trajectories in total: three trials each at the three seen target positions.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Imitation learning has become a cornerstone for solving complex robotic manipulation tasks. In particular, multimodality, which enables robots to capture diverse yet valid behavioral patterns, has driven the rapid emergence of generative policies as a dominant paradigm in robot learning. However, achieving such multimodality typically relies on stochastic noise initialization and iterative denoising procedures, resulting in substantial training complexity and low inference efficiency. Meanwhile, not all phases of a robotic task inherently require behavioral diversity. Motivated by this insight, we propose the Modality-Adaptive Robot Sampling (MARS) policy, which adaptively invokes tailored stochasticity only when it is truly beneficial, while reverting to an efficient deterministic learning during single-modal phases. In other words, the proper amount of noise is injected only at the proper time. By selectively activating multimodal generation, MARS policy bridges the gap between the multimodal capability of generative policies and the superior training and inference efficiency of deterministic models. Empirical studies across 8 simulated and 4 real-world tasks demonstrate that MARS exhibits robust multimodal expressivity and high efficiency, with a 16.67% success rate improvement and an 83.20% inference latency reduction in real-world tests. Counterintuitively, MARS also outpaces deterministic policies in training efficiency on near-deterministic tasks by more effectively modeling nuanced action diversity.
Generative action models based on diffusion and flow matching have been increasingly adopted in vision-language-action (VLA) policies for their ability to capture diverse behaviors, including multiple valid action sequences under the same observation and instruction. Their iterative sampling procedures, however, require repeated network evaluations to generate each action chunk, increasing inference latency in closed-loop control. We propose ESP (Energy-Score Policy), a teacher-free approach that maps policy context and noise directly to an action chunk in a single network evaluation. ESP trains the action head with the energy score rather than mean squared error. Whereas squared-error regression targets the conditional mean, the energy score is strictly proper: its expected value is uniquely minimized by the target distribution. This provides a principled objective for learning multimodal action distributions without iterative sampling, with exact recovery at the population optimum when the model can represent the target distribution. Experiments on both simulation and real-world manipulation tasks demonstrate competitive task success with substantially lower action-generation latency than the flow matching baseline. These results support direct distributional learning as an efficient alternative to iterative generative robot policies.
Lilika Makabe, Heecheol Kim, Yasuyuki Matsushita
Microsoft Research Asia - Tokyo · Graduate School of Information Science and Technology, the University of Osaka
We address the problem of fine-tuning pre-trained generative policies with reinforcement learning (RL) while preserving the multimodality of their action distributions. Existing methods for RL fine-tuning of generative policies (e.g., diffusion policies) improve task performance but often collapse diverse behaviors into a single reward-maximizing mode. To mitigate this issue, we propose an unsupervised mode discovery framework that uncovers latent behavioral modes within generative policies. The discovered modes enable the use of mutual information as an intrinsic reward, regularizing RL fine-tuning to enhance task success while maintaining behavioral diversity. Experiments on robotic manipulation tasks demonstrate that our method consistently outperforms conventional fine-tuning approaches, achieving higher success rates and preserving richer multimodal action distributions.
Alberta Longhini, David Emukpere, Jean-Michel Renders +1
Department of Computer Science, Stanford University, CA, USA