From LLM-Generated Specifications to Learned Quadruped Locomotion

Merve Atasever; Keyan Azbijari; Cagan Bakirci; Alfredo Reina Corona; Tolga Izdas; Richard Yang; Erdem Biyik; Jyotirmoy V. Deshmukh

cs.ROSep 7, 2026

From LLM-Generated Specifications to Learned Quadruped Locomotion

Authors: Merve Atasever, Keyan Azbijari, Cagan Bakirci, Alfredo Reina Corona, Tolga Izdas, Richard Yang, Erdem Biyik, Jyotirmoy V. Deshmukh

Organizations: Department of Computer Science University of Southern California Los Angeles, CA, USA · University of Florida Gainesville, FL, USA

Abstract

Quadruped robot locomotion policies are often trained using reinforcement learning, which in turn relies heavily on hand-crafted reward functions. Designing reward functions requires substantial manual engineering, and it is often unclear which local rewards will induce the desired global behavior. Shaped rewards from formal specifications in languages like Signal Temporal Logic (STL) can make rewards more interpretable, but writing STL specifications itself still requires domain expertise. We study whether large language models (LLMs) can fill this gap by generating Parametric Signal Temporal Logic (PSTL) specifications that are subsequently used for policy learning. Given a natural language locomotion objective and a constrained specification grammar, GPT-5.5 and Qwen 3.6 independently propose STL templates for command tracking, safety, and gait structure. We instantiate the parameters of the generated PSTL templates using expert trajectories and retain only specifications that are consistent with demonstrated expert behavior. The resulting specifications are then transformed into smooth, finite-history reward functions and used to train a quadruped locomotion policy with Proximal Policy Optimization (PPO) in MuJoCo XLA (MJX). We evaluate both gait-aware and gait-agnostic settings. The former specifies walking-trot, trot, and bound regimes, while the latter allows contact patterns to emerge from the task objective. We compare against hand-engineered rewards, Text2Reward-style LLM-generated reward code, and an expert-switching oracle. Gait-aware Qwen 3.6 specifications achieved 100% survival and command success across all tested speeds (0.3--2.1 m/s) and matched the target gait at high speeds, whereas Text2Reward achieved 0% for both metrics at $\geq 1.9$ m/s. Videos: https://stl-locomotion.github.io/

Figures & tables

Heuristic
$v_{x}$ (m/s)	CoT $\downarrow$	Survival $\uparrow$	Success $\uparrow$
0.3	$1.20\pm 0.00$	$100\%$	$100\%$
0.5	$1.00\pm 0.00$	$100\%$	$100\%$
0.7	$1.00\pm 0.00$	$100\%$	$100\%$
1.0	$1.20\pm 0.00$	$100\%$	$100\%$
1.3	$1.30\pm 0.00$	$100\%$	$100\%$

TABLE I: Gait-agnostic locomotion results across commanded forward velocities. Lower CoT is better; higher survival and success are better. Each entry reports mean

\pm

standard deviation over 20 rollouts.

$v_{x}$	CoT $\downarrow$	Survival $\uparrow$	Success $\uparrow$	Gait Match $\uparrow$
Expert Oracle
0.3	$0.92\pm 0.02$	$100\%$	$100\%$	Success
0.5	$0.90\pm 0.01$	$100\%$	$100\%$	Success
0.7	$0.96\pm 0.01$	$100\%$	$100\%$	Success
1.0	$1.08\pm 0.01$	$100\%$	$100\%$	Success
1.3	$1.27\pm 0.01$	$100\%$	$100\%$	Success

TABLE II: Multi-gait locomotion results across commanded forward velocities. Lower CoT is better; higher survival, command success, and gait success are better. Gait success measures whether the desired gait associated with the commanded speed regime is successfully exhibited.

Method	$H$	CoT $\downarrow$	Survival $\uparrow$	Success $\uparrow$
GPT-GA	1	0.99	100%	100%
	5	0.99	100%	100%
	10	1.00	100%	100%
	20	1.15	100%	44%
	30	1.14	100%	67%
Qwen-GA	1	$1.11$	$55.6\%$	$55.6\%$

TABLE III: Ablation of STL robustness window size

H

for gait-agnostic policies. Metrics are averaged across all evaluated commanded forward velocities.

Method	$H$	CoT $\downarrow$	Survival $\uparrow$	Success $\uparrow$
GPT-MG	1	1.40	100%	55.6%
	5	1.60	100%	55.6%
	10	1.37	100%	55.6%
	20	1.40	100%	66.7%
	30	1.36	99.4%	66.7%
Qwen-MG	1	$1.28$	$100\%$	$100\%$

TABLE IV: Ablation of STL robustness window size

H

for multi-gait policies. Metrics are averaged across all evaluated commanded forward velocities.

LLM-Guided RL

Large language model-guided reinforcement learning uses language models to shape rewards, exploration, planning, and policy learning.

See all 140 papers

Jul 1, 2026

Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications

Reinforcement learning (RL) for quadruped locomotion commonly depends on fixed, hand-crafted, and Markovian reward functions that limit both interpretability of learned policies and lack explicit control over gait behaviors. We introduce a framework where distinct gaits are specified using parameterized constraints expressed in Signal Temporal Logic (STL). These include safety bounds, gait synchronization constraints, command tracking, and actuation bounds. From these specifications, we develop a reward shaping mechanism that provides learning agents a dense, continuous reward landscape that encodes desired behavior. We define parametric STL templates for three speed regimes (walking-trot, trot, bound), calibrate their parameters from reference rollouts, and compute rewards from using smooth approximations of STL robustness over the rollouts. The generated rewards can be used to provide shaped gradients compatible with Proximal Policy Optimization (PPO). We instantiate the approach on Google's Barkour quadruped robot in MuJoCo XLA (MJX). We use parallelization within the simulator to improve training speeds and use domain randomization to robustify learned policies. We show that compared to a baseline of hand-crafted rewards, the STL-shaped rewards yield tighter velocity tracking and more stable training. Videos can be found on our project website: https://stl-locomotion.github.io/.

Merve Atasever, Cagan Bakirci, Alfredo Reina Corona +2

University of Southern California, Los Angeles, California, USA

Reinforcement Learning Robot Skill Learning RL for Legged Locomotion Policy Learning Reward Design for RL Quadruped Locomotion

Ask PDF

Sep 21, 2026

MimicAgent: Quadruped Skills via Prompt-to-Trajectory Generation

We present MimicAgent, a prompt-to-trajectory generation framework for learning dynamic quadruped skills. Although reward shaping is extensively used when training quadruped policies, navigating the resulting reward landscape is notoriously difficult, requiring hours of "graduate student descent". Eureka attempts to automate reward design with LLMs, but we find that it struggles to generalize across diverse skills and morphologies. Our key observation is that it is far easier for a human - and by association, an LLM - to generate reference motions than to shape reward functions. Our hypothesis is motivated by the success of example-guided RL for humanoids, which exploits large-scale motion capture datasets as references for training locomotion policies. Unlike humanoids, quadrupeds lack such reference motion data. Towards this end, we propose MimicAgent, an agentic harness that, given a skill prompt, generates quadruped reference trajectories with coding agents. These coarse reference trajectories are then used to train example-guided RL policies that are deployable in simulation and in the real-world. Notably, we find that when prompting Claude Fable 5.1 within our agentic harness, 87% of prompts yield semantically aligned reference trajectories.

Lucky Kant Nayak, Narayanan Palghat Parameswaran, Neehar Peri +1

Carnegie Mellon University

Reinforcement Learning Trajectory Generation LLM Agent Trajectory Synthesis RL for Legged Locomotion Policy Learning Quadruped Locomotion

Ask PDF

Sep 29, 2026

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves

85.8\%

average success-once and

67.0\%

success-at-end, compared with

81.5\%/59.5\%

for native dense PPO and

65.0\%/42.3\%

for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve

100\%

success across velocities from

0.3

2.1\,\mathrm{m/s}

while remaining competitive in high-speed energy efficiency. Project webpage: video2stl.

Merve Atasever, Keyan Azbijari, Cagan Bakirci +6

Department of Computer Science, University of Southern California Los Angeles, CA, USA · University of Florida Gainesville, FL, USA

VLMs for Robotics Signal Temporal Logic Cross-Embodiment Robot Learning Robot Policy Learning Policy Learning Robot Learning+1

Ask PDF

Abstract

LLM-Guided RL

Explore similar work

Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications

MimicAgent: Quadruped Skills via Prompt-to-Trajectory Generation

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning