Ergodic Control and Controlled Diffusion for Robot Learning: Review and Tutorial
Authors: Max Muchen Sun, Cem Bilaloglu, Ananya Rao, Stefan Ivic, Guillaume Sartoretti, Kathleen Fitzsimons, Ian Abraham, Sylvain Calinon, +1 more
Organizations: Northwestern University · Idiap Research Institute · Carnegie Mellon University · University of Rijeka · National University of Singapore · Pennsylvania State University · University of Sydney/Yale University
Diffusion learning leverages the statistical mechanism of diffusion processes for learning, reasoning, and inferring complex distributions from data. Recent advances in diffusion learning have been transformative, with robot learning emerging as a key opportunity area, with applications spanning perception, control, and decision-making. At the same time, the statistical mechanism of diffusion processes can be controlled to shape the temporal evolution of the state distribution underlying robot trajectories, inducing ergodic behavior in robotic systems. The frameworks of controlled diffusion and ergodic control were developed around the same time as diffusion learning, and their theories and algorithms have increasingly converged. Ergodicity induced by controlled diffusion has several significant implications for robot learning, distinct from applying diffusion learning to robotics problems: it formally enforces statistical properties required to ensure optimality of robot learning, enables non-myopic search over uncertain information landscapes for data collection, and enables behavior specification based on spatial rather than temporal characteristics of trajectories. This survey introduces the intuition behind controlled diffusion for robot learning, explores its connection to diffusion learning, presents theoretical foundations and numerical tutorials for solving controlled diffusion and ergodic control problems, and reviews applications across robotics. Finally, we discuss key challenges and future opportunities in leveraging controlled diffusion for robot learning.
Figures & tables
Figure 1.1 : Illustrations of ergodic trajectories and ergodic control in action.
Figure 2.1 : Illustration of the “rectangle game”. We assume the binary measurements are sampled from an evenly distributed grid within an arbitrary but fixed squared area. The learned rectangle is computed as the smallest rectangle that contains all the positive signals and none of the negative signals.
Figure 2.3 : There exists a computable lower bound of the error between the ground truth rectangle and the learned rectangle from a finite number of samples. Furthermore, this bound decreases as the number of samples increases.
Figure 2.5 : Illustration of the BoxGPT example. Training and inference of the generative model are straightforward; what is not straightforward is data collection.
Figure 2.6 : Exhaustive sampling is an effective but inefficient data collection strategy.
Figure 2.7 : Based on the model’s current predictions, sampling measurements from different regions—regardless of their exact values—has varying impacts on model improvement. Identifying the optimal data distribution to sample from is the key to optimal data collection for learning.
Figure 2.9 : Increasing the sampling frequency along a trajectory where the time-averaged statistics fail to converge to a stationary distribution may not yield a better dataset.
Figure 2.11 : Temporal vs. spatial specifications for motion synthesis. (a) Conventional methods focus on the temporal sequence of robot actions. (b) Controlled diffusion focuses on the spatial statistics of the robot trajectory.
Figure 3.1 : For generative models, the diffusion process transforms a simple distribution into a complex one. Controlled diffusion follows the same principle, but applies the diffusion process to motion synthesis, generating trajectories that match desired time-averaged statistics. There are two perspectives on controlled diffusion: in the first, trajectories are generated incrementally over time (incremental generation), while in the second, they are iteratively optimized across the entire horizon (trajectory optimization).
Figure 4.1 : Diagrams for specifying an ergodic control implementation.
Fourier
Kernelized
MMD
KL
Sinkhorn
Target representation
Grid / Samples
Normalized density
Samples
Score function
Grid / samples
Time complexity
O(MKd)
O(M2d)
O((M2+MN)d)
O(M2d)
O(L(M2+MN)d)
Space complexity
O(Kd)
O(M2)
O(M2+MN)
O(M2)
O(M2+MN)
Dimension scaling
Exponential
Linear
Linear
Linear
Linear
Domain
Bounded rectangular Euclidean domain
Any domain with a suitable kernel
Any domain with a suitable kernel
Euclidean; specialized extensions for manifolds
Any domain with a suitable kernel
Advantages
Numerically stable and robust
Scalability and flexibility across domains
Works directly with samples
Supports unnormalized target densities
Handles irregular or disjoint supports
Table 4.1 : Comparison of ergodic metrics.
Figure 4.2 : The number of Fourier coefficients determines the reconstruction accuracy of the target distribution and the resulting ergodic trajectory.
Figure 4.3 : Kernel ergodic control effectively combined information maximization behavior with uniform coverage behavior to synthesize optimal ergodic trajectories.
Figure 4.4 : The iterative LQR process is shown to be equivalent to the flow matching [ 113 ] framework in generative models subject to dynamics constraints [ 176 ] .
Figure 4.5 : (a) Example of ergodic exploration using HEDAC FEM method presented in [ 81 ] . The triangular discretization allows for curved outer and inner boundaries of the domain. (b) 3D trajectories filling a torus with HEDAC FEM implementation from [ 79 ] .
Figure 5.1 : Filtered Ergodic Control. Shown is an example ergodic trajectory (orange) for a planar drone-like system covering the darker areas in the background. Obstacles to be avoided are shown in green where the non-occupied space defines the safe set for the drone. The red arrow illustrates the magnitude of the ergodic control vector uE while the blue arrow is the closest resulting safe controller that avoids the obstacle.
Figure 5.2 : Integrated Safety Ergodic Control. Here, the optimized ergodic trajectory (orange) has the safety barrier constraint satisfied throughout the planning horizon. The blue vector is the commanded ergodic control with integrated safety is that follows the optimized path. Performance of the ergodic controller is preserved while maintaining safety throughout.
Figure 5.3 : Time-Evolution of a 2D drone following a robust ergodic controller. Top row: the time-averaged spatial distribution and the information distribution. As time increases, the time-averaged trajectory statistics approach the information distribution, which is the goal of ergodic exploration. Bottom: evolution of the trajectory through phase space, superimposed on a cross-section of the value function with respect to position and velocity, where zk are fixed to their value at that instant in the trajectory.
Figure 5.4 : SVGD Prevents Mode Collapse and Promotes Diverse Ergodic Controllers. Here, we demonstrate mode collapse of inferring ergodic trajectories, i.e., when there is no repulsive force to diversify solutions. (Left) Ergodic trajectories randomly initialized converge to similar solutions. (Right) With the repulsive force from SVGD, the trajectory solutions leverage the multimodal nature of the ergodic metric, yielding diverse solutions, and ultimately solution robustness to uncertainty. Mode collapse can occur with increased dimensionality of the optimization variables which can be mitigated with the right choice of kernel [ 115 ] .
Figure 5.5 : GPU Acceleration of SVGD and the Ergodic metric for Robust Control. (Left) Example SVGD for robust ergodic control in an online exploration problem (cover dark areas) with dynamics obstacles (green). A total of 12 ergodic trajectories are calculated (shown in black). The red line indicates the best possible solution according to a heuristic ( [ 104 ] ), and blue line indicates visited areas. (Right) Computational scaling across devices. The Stein variational ergodic controller can optimize several paths that can be used as redundancies to attain robustness in uncertain and dynamic environments.
Figure 5.6 : Time Optimal Ergodic Trajectory Evolution [ 44 ] in clutter. As one lower the upper bound γ , satisfying the ergodic metric requires the solver to search for larger tf which allows the robot trajectory to cover more of the space. Here, ϕ is given as a uniform distribution.
Figure 5.7 : Energy-Aware Ergodic Coverage . State of charge awareness can maintain a persistent level of ergodicity. Each agent shown trades off uniform coverage over the space and battery charge. High density areas are defined via μ1,μ2,μ3,μ4 whereas charging stations are denoted by α1,α2,α3,α4 .
Figure 5.8 : Ergodicity as a function of state-of-charge . The top plot shows the evolution of the ergodicity for a set of agents. The bottom shows the average ergodicity. Initially, the agents are at their charging stations. As agents begin to lose battery charge, they they start moving to the charging stations – ergodicity increases while other agents maintain the average ergodicity to be under the value γ .
Figure 5.9 : The Laplacian eigenbasis extends Fourier analysis to curved manifolds, including surfaces reconstructed from point clouds. This basis enables efficient closed-loop ergodic control on such surfaces, supporting operations such as cleaning or surface inspection.
Figure 5.10 : Example spectral decomposition of an information prior with two heterogeneous agents: A given information prior is decomposed directly in the spectral domain, and the spectral coefficients assigned to the different agents based on human intuition. Here, the UAV with omnidirectional motion capabilities and high-range, low-fidelity sensing (e.g., onboard camera) is assigned low-order coefficients, allowing it to quickly cover the lowest-frequency portions of the information space. Conversely, a ground vehicle with curve-constrained motion constraints, equipped with a low-range, high-fidelity sensor (e.g., LiDAR) is assigned higher-frequency coefficients, allowing it to easily chain peaks of information.
Figure 5.11 : Learning spectral allocations [ 149 ] : Relying on the A3C algorithm with three different distribution schemes, agents learn their allocation to search subtasks. Each distribution method has different initial conditions. In all three methods, the output is a weighted set of spectral coefficients of the information distribution being searched for each agent. The learning process seeks to minimize the joint ergodic metric of the team’s trajectories.
Figure 6.1 : Examples of ergodic control applications in information gathering and active perception. From top to bottom: (1) Information gathering with UAVs over uneven terrains; (2) Information gathering with human-swarm systems; (3) Information gathering over disaster sites; (4) Active perception with binary sensors; (5) Active perception with generative models.
Figure 6.2 : Examples of ergodic control applications in manipulation. From left to right: (1) Learning insertion policies from human demonstrations; (2) Reactive erasing in dynamic environments; (3) Erasing over curved surfaces with tactile feedback.
Figure 6.3 : Examples of ergodic control applications in human-robot interaction. From left to right: (1) Ergodicity as a measure of task-relevant information in patients’ motion during rehabilitation; (2) Ergodic control ensures safety while maintaining adaptivity in physical therapy; (3) Ergodic control for learning from physical demonstration.
Figure 6.4 : Examples of ergodic control applications in manufacturing. From left to right: (1) Micro-patterned surface manufacturing with swarm robots; (2) Learning assembly from demonstrations; (3) Autonomous inspection of manufactured components.
Figure 6.5 : Examples of ergodic control applications in robot art.
Diffusion models can represent complex, multimodal trajectory distributions, but extracting uncertainty from them typically requires costly Monte Carlo sampling. This limits their use in real-time control, where robots must rapidly assess risk and maintain safety margins. We introduce Score-Curvature for Online Precision Estimation (SCOPE), a lightweight module that augments diffusion trajectory models with control-ready uncertainty. SCOPE learns a structured precision matrix around each nominal trajectory by distilling score-curvature information and producing calibrated Gaussian tubes with low overhead and without repeated Monte Carlo sampling. These tubes provide per-timestep covariance estimates that can be used both as predicted occupancy for moving agents and as adaptive exploration guides for robot control. We evaluate SCOPE with mode-conditioned multimodal diffusion backbones in pedestrian forecasting, crowd navigation, Maze2D control, and real-world Franka Panda manipulation. Across these settings, SCOPE provides fast uncertainty estimation, which leads to better closed-loop performance. Project page: https://zackaxue.github.io/SCOPE-project-page/
Zhiwei Xue, Jia Yue Kam, Jinhang Qiu +5
National University of Singapore. · LAAS-CNRS. · Smart Systems Institute, NUS.
A key challenge towards reliable robotic control is devising computational models that can both learn policies and guarantee robustness when deployed in the field. Inspired by the free energy principle in computational neuroscience, to address these challenges, we propose a model for policy computation that jointly learns environment dynamics and rewards, while ensuring robustness to epistemic uncertainties. Expounding a distributionally robust free energy principle, we propose a modification to the maximum diffusion learning framework. After explicitly characterizing robustness of our policies to epistemic uncertainties in both environment and reward, we validate their effectiveness on continuous-control benchmarks, via both simulations and real-world experiments involving manipulation with a Franka Research~3 arm. Across simulation and zero-shot deployment, our approach narrows the sim-to-real gap, and enables repeatable tabletop manipulation without task-specific fine-tuning.
Hozefa Jesawada, Giovanni Russo, Abdalla Swikir +1
Mechanical Engineering Program, New York University Abu Dhabi, Abu Dhabi, UAE. · Department of Information and Electrical Engineering & Applied Mathematics, University of Salerno, Fisciano, Italy. · Mohammed bin Zayed University for Artificial Intelligence, Abu Dhabi, UAE.
Generative policies such as diffusion models provide expressive data-driven priors for robot behavior, yet the situations a robot faces at execution time rarely match its training data and impose constraints and objectives the policy never learned. Existing inference-time guidance methods either rely on gradients that can push samples away from the learned prior, or on sampling procedures that can be inefficient, collapse modes, or average them into out-of-distribution predictions. We introduce MOSAIC, a mode- and support-aware framework for gradient-free inference-time control of robotic diffusion models. MOSAIC treats the initial noise and all stochastic reverse-step perturbations as a joint input to the pretrained denoising process and adapts a distribution over this input using importance weights computed from fully denoised outputs. Physical feasibility and consistency with the learned support are balanced using adaptive penalties, while mode-wise importance-weighted updates preserve modes and avoid destructive averaging. The same formulation can jointly steer multiple independently trained diffusion models through shared coupling costs, enabling coordinated behavior without requiring a joint model. We evaluate MOSAIC on navigation, dexterous grasping, motion planning with unseen geometry, and collaborative box manipulation with independently trained robot priors. Across these settings, MOSAIC adapts and composes frozen diffusion priors under new deployment constraints without retraining.
Sabrina Bodmer, Rahel Rickenbach, René Zurbrügg +2
Department of Mechanical and Process Engineering, ETH Zürich, Switzerland. · Automatic Control Laboratory, EPFL, Switzerland.