Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient

Hongrui Zheng; Cristian Ioan Vasile; Antonio Loquercio; Rahul Mangharam

cs.ROSep 25, 2026

Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient

Authors: Hongrui Zheng, Cristian Ioan Vasile, Antonio Loquercio, Rahul Mangharam

Organizations: Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA · Mechanical Engineering and Mechanics Department, Lehigh University, Bethlehem, PA, USA

Abstract

Learning robot policies for tasks with sparse success signals is challenging when completion depends on coordinated actions, precise contact outcomes, or satisfying several conditions together. Intricate physical interactions with the world further complicate these requirements. Prior work using conventional reward shaping mechanisms provides dense feedback but local progress might not translate into eventual task completion. We present Signal Temporal Logic-guided Stein Variational Policy Gradient (STL-SVPG), a population-based method that uses smooth STL robustness as a trajectory-level training objective. Differentiating this objective through the dynamics assigns credit to policy actions according to their effect on the complete task specification, rather than local progress alone. We evaluate the approach on six quadcopter and manipulator tasks that involves event-triggered responses, strictly ordered behavior, responses within specified deadlines, and physical interaction with the world. STL-SVPG achieves the highest mean success rate among the compared methods on five of six benchmarks. Simulation-trained policies trained in simulation transfer temporal and contact task behavior to the real world.

Figures & tables

d_{i,e,t}=\frac{\partial\widetilde{\rho}_{\phi}(\tau_{i,e})}{\partial u_{i,e,t}}

Algorithm 1 STL-SVPG

Fig. 2: Example of successful STL-SVPG rollout in all tasks. In Unlock and both Random Cue tasks, the policy learned stable control that satisfies the timing and order requirements. In Crazyflie and Panda Ping-pong, it learned the ball return strikes reliably. And finally in Pillar Pull, it learned to bump two pillars without disturbing the balance of the plate and the block, without specifying which two pillars to be removed.

Task	STL-SVPG P64	PPO	SAC	TD-MPC2
CF Unlock	87.67 $\pm$ 20.23	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.02 $\pm$ 0.04
CF Random Cue	71.93 $\pm$ 2.95	13.98 $\pm$ 4.18	19.53 $\pm$ 9.72	74.33 $\pm$ 17.14
CF Ping-Pong	11.30 $\pm$ 4.36	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00
CF Pillar Pull	5.35 $\pm$ 4.09	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.78 $\pm$ 1.56
Panda Random Cue	99.98 $\pm$ 0.04	83.33 $\pm$ 40.82	25.00 $\pm$ 41.83	96.72 $\pm$ 2.98
Panda Ping-Pong	35.78 $\pm$ 12.99	0.00 $\pm$ 0.00	6.70 $\pm$ 9.23	0.32 $\pm$ 0.38

TABLE I: Overall baseline comparison using task success rates (%, mean

\pm

sample standard deviation over six training seeds). Evaluation uses 1000 held-out episodes per policy per seed.

Task	STL-SVPG	Dense diff.	Dense FD
CF Unlock	87.67 $\pm$ 20.23	45.17 $\pm$ 32.58	0.00 $\pm$ 0.00
CF Random Cue	71.93 $\pm$ 2.95	66.07 $\pm$ 6.29	0.47 $\pm$ 0.79
CF Ping-Pong	11.30 $\pm$ 4.36	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00

TABLE II: Objective comparison and black-box dense baseline using task success rates (%, mean

\pm

sample standard deviation over six training seeds). Evaluation uses 1000 held-out episodes per policy per seed.

Task	STL-SVPG P64	STL-SVPG P1
CF Unlock	87.67 $\pm$ 20.23	0.00 $\pm$ 0.00
CF Random Cue	71.93 $\pm$ 2.95	68.63 $\pm$ 9.79
CF Ping-Pong	11.30 $\pm$ 4.36	4.03 $\pm$ 5.00
CF Pillar Pull	5.35 $\pm$ 4.09	0.00 $\pm$ 0.00
Panda Random Cue	99.98 $\pm$ 0.04	93.22 $\pm$ 9.58
Panda Ping-Pong	35.78 $\pm$ 12.99	7.78 $\pm$ 10.46

TABLE III: Population ablation with the same per-particle rollout and iteration schedule using task success rates (%, mean

\pm

sample standard deviation over six training seeds). Evaluation uses 1000 held-out episodes per policy per seed.

Fig. 3: Top-down view of all 64 policies on one reset from a Branching training instance: 30 A–C successes, 29 B–D successes, and five failures.

Fig. 4: Real world policy rollout trajectory of Crazyflie Random Cue from two starting positions. Gray, red, blue, and green portions of the trajectory denotes different phases of the task.

Explore similar work

STL-SVPIO: Signal Temporal Logic guided Stein Variational Path Integral Optimization
Mar 6, 2026Hongrui Zheng, Zirui Zang, Ahmad Amine +2Trajectory Optimization Robot Task Planning
Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA. · Mechanical Engineering and Mechanics Department, Lehigh University, Bethlehem, PA, USA.
Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning
Sep 29, 2026Merve Atasever, Keyan Azbijari, Cagan Bakirci +6Robot Policy Learning Signal Temporal Logic
Department of Computer Science, University of Southern California Los Angeles, CA, USA · University of Florida Gainesville, FL, USA
Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications
Jul 1, 2026Merve Atasever, Cagan Bakirci, Alfredo Reina Corona +2Reinforcement Learning Robot Skill Learning
University of Southern California, Los Angeles, California, USA

Mar 6, 2026J/K move · Enter open · S save

STL-SVPIO: Signal Temporal Logic guided Stein Variational Path Integral Optimization

Hongrui Zheng, Zirui Zang, Ahmad Amine, Cristian Ioan Vasile+1

Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA. · Mechanical Engineering and Mechanics Department, Lehigh University, Bethlehem, PA, USA.

Trajectory Optimization Robot Task Planning Motion Planning

Open paper AskPDF arXiv

Signal Temporal Logic (STL) enables formal specification of complex spatiotemporal constraints for robotic task planning. However, synthesizing long-horizon continuous control trajectories from complex STL specifications is fundamentally challenging due to the nested structure of STL robustness objectives. Existing solver-based methods, such as Mixed-Integer Linear Programming (MILP), suffer from exponential scaling, whereas sampling methods, such as Model-Predictive Path Integral control (MPPI), struggle with sparse, long-horizon costs. We introduce Signal Temporal Logic guided Stein Variational Path Integral Optimization (STL-SVPIO), which reframes STL as a globally informative, differentiable reward-shaping mechanism. By leveraging Stein Variational Gradient Descent and differentiable physics engines, STL-SVPIO transports a mutually repulsive swarm of control particles toward high robustness regions. Our method transforms sparse logical satisfaction into tractable variational inference, mitigating the severe local minima traps of standard gradient-based methods. We demonstrate that STL-SVPIO significantly outperforms existing methods in both robustness and efficiency for traditional STL tasks. Moreover, it solves complex long-horizon tasks, including multi-agent coordination with synchronization and queuing while baselines either fail to discover feasible solutions, or become computationally intractable. Finally, we use STL-SVPIO in agile robotic motion planning tasks with nonlinear dynamics, such as 7-DoF manipulation and half cheetah back flips to show the generalizability of our algorithm.

Abstract

Explore similar work

STL-SVPIO: Signal Temporal Logic guided Stein Variational Path Integral Optimization

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications