Potential-Field Action Representation for Reinforcement Learning in Contact-Rich Manipulation
Authors: Xinyu Liu, Gökhan Solak, Arash Ajoudani
Organizations: Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, 16121 Genoa, Italy, and also with the Ph.D. Program of National Interest in Robotics and Intelligent Machines (DRIM), Universit`a di Genova, 16126 Genoa, Italy · Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, 16163 Genoa, Italy
Model-free reinforcement learning can acquire contact-rich robotic manipulation skills through trial-and-error interaction, but it often requires the policy to learn both task strategy and low-level motion generation. In this setting, the action representation is critical because it determines how policy outputs are converted into robot motion, shaping both exploration and physical execution. Direct Cartesian command interfaces require the policy to generate motion at every decision step, coupling task-level adaptation with continuous low-level control and increasing the learning burden. We propose PA-RL, a reinforcement-learning framework that uses artificial potential fields as the action representation. Instead of commanding motion directly, the policy adapts the parameters of an energy-like potential field, which generates a state-dependent guidance direction executed through a Cartesian impedance controller. We evaluate PA-RL on peg-in-hole insertion, a representative contact-rich task with nonlinear dynamics and discontinuous contact transitions. In simulation, PA-RL is compared with Cartesian velocity, Cartesian pose, and variable-impedance action spaces using the same RL algorithm. PA-RL is the only method to reach a 100% evaluation success rate within the allotted training time, while the best baseline reaches 92.6%. It also reduces joint-torque variation by 55.4% and Cartesian acceleration variation by 70.8% relative to the best baseline, without explicit motion-quality penalties in the reward. The simulation-trained policy further completes 9/9 real-robot insertions without fine-tuning, demonstrating the deployment feasibility of the learned potential-field interface.
Figures & tables
Fig. 1: Overview of the PA-RL framework for the peg-in-hole task. The policy action consists of the positions and parameters of several attractors with linear and gaussian bases, defining a state-dependent artificial potential field over the goal-relative workspace that is illustrated as a vector field (green) along its gradient. The velocity reference is generated following the negative gradient of the potential field.
Fig. 2: Overview of the PA-RL pipeline. The upper diagram shows the closed-loop execution architecture and the lower panels illustrate a representative peg-in-hole example. Here, pc,t and pg,t are peg-tip position and goal position, and force–torque sensor wrench wt . The action specifies the attractor offsets pai,t , weights λai,t , and Gaussian sigma σai,t .
Parameter
Distribution or range
Initial hole offset [Δx,Δy]
[−50,50]mm per axis
Goal-pos. estimation error [ex,ey]
N(0,(1.5mm)2I)
Goal-distance noise [nx,ny,nz]
N(0,diag(0.5,0.5,0.3)2)mm
Force observation noise
N(0,(0.2N)2I)
Torque observation noise
N(0,(0.02Nm)2I)
TABLE I: Task randomization and observation-noise settings.
Fig. 3: Simulation and real-robot setups for the peg-in-hole task. (a) MuJoCo environment with a Franka Emika Panda, cylindrical peg, and fixed hole. (b) Physical setup with the same robot and peg–hole geometry, instrumented with an ATI force–torque sensor.
Fig. 4: Learning efficiency over three seeds. Solid lines and shaded regions denote the across-seed mean and one standard deviation. The endpoint distribution expressed as the final peg-tip XY error relative to the true insertion goal.
Fig. 5: Fixed-grid evaluation task performance. Each evaluation process contains 27 episodes with nine hole positions on a 3×3 grid and three trials per position under resampled goal-position estimation error and observation noise. Solid lines and shaded regions denote the across-seed mean and one standard deviation.
Fig. 6: Motion quality at the selected evaluation experiment. For each method and seed, the checkpoint with the highest evaluation success rate is selected. Metrics are computed over all 27 evaluation episodes. Bars and error bars denote the across-seed mean and one standard deviation; markers denote individual seeds.
Fig. 7: Real-robot deployment over a 3×3 grid. (a) Per-trial task performance with final peg-tip-to-goal distance, peak measured contact force, and completion time. (b) Top-view peg-tip trajectories relative to the true insertion goal and (c) Side-view trajectories. Colors indicate the different initial relative peg–hole positions.
Contact-rich manipulation requires policies to generate precise actions by reasoning over contact forces, robot configurations, and interaction histories beyond visual observations. Existing methods passively condition on force feedback rather than actively predicting future contact dynamics, limiting their ability to generate high-precision actions. To address this problem, we introduce Predictive Sensorimotor Representation (PSR) learning, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy. Specifically, during a pretraining stage, a multimodal Transformer is trained to learn a hierarchy of predictive representations by jointly forecasting future interaction dynamics. The learned hierarchy subsequently augments the action stream, enabling the resulting policy to exploit contact-relevant cues at multiple depths. We further instantiate PSR within a Vision-Language-Action (VLA) model, resulting in PSR-VLA, and evaluate it on six real-world contact-rich manipulation tasks. Experimental results show that PSR-VLA achieves 91.7% overall success, improving over π0.5, ForceVLA-π0.5, and ForceVLA2-π0.5 by 30.0, 22.5, and 19.2 percentage points, respectively. These results demonstrate the effectiveness of the proposed PSR for force-aware, contact-rich manipulation. Videos of the tasks and stability tests are available at https://psr-vla.pages.dev/.
Contact-rich manipulation requires robots to coordinate not only where to move, but also how contact and force should evolve throughout an interaction. Yet most robot policies represent actions primarily as motion trajectories, using tactile signals as inputs or auxiliary prediction targets without explicitly encoding contact dynamics in the action representation. We propose ILA (Interaction Latents as Actions), which uses decodable interaction latents that jointly capture the evolution of motion, contact, and force application as policy action units, providing a shared representation for action generation, evaluation, and offline improvement. We learn these latents through visually conditioned joint action-tactile reconstruction with a time-conditioned implicit decoder. Using them as flow-matching generation targets supports post-training of pretrained vision-language-action models (VLAs) and world action models (WAMs), as well as training flow-matching policies from scratch. A shared decoder maps generated latents to executable actions. For offline improvement, we learn a tactile-conditioned expert reference model from interaction latents encoded from expert demonstrations. The resulting expert-consistency rewards guide weighted learning from successful rollouts, while observation-compatible expert references guide correction of low-reward predictions from failed rollouts. Across three policy families, ILA improves success rates averaged over four real-world tasks by 21.0-36.5 percentage points relative to tactile-conditioned direct-action baselines. Ablations highlight the benefit of encoding contact evolution in action targets beyond tactile conditioning, while the same latent space supports expert-guided offline improvement. Project website: https://mrsecant.github.io/ILA/.
In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.
Department of Computing Science, University of Alberta · National Research Council Canada · School of Electrical Engineering and Computer Science, University of Ottawa +3