Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

Merve Atasever; Keyan Azbijari; Cagan Bakirci; Bo-Ruei Huang; Tolga Izdas; Zahra Shahrooei; Richard Yang; Erdem Biyik

cs.ROSep 29, 2026

Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning

Authors: Merve Atasever, Keyan Azbijari, Cagan Bakirci, Bo-Ruei Huang, Tolga Izdas, Zahra Shahrooei, Richard Yang, Erdem Biyik, +1 more

Organizations: Department of Computer Science, University of Southern California Los Angeles, CA, USA · University of Florida Gainesville, FL, USA

Abstract

Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves $85.8\%$ average success-once and $67.0\%$ success-at-end, compared with $81.5\%/59.5\%$ for native dense PPO and $65.0\%/42.3\%$ for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve $100\%$ success across velocities from $0.3$ to $2.1\,\mathrm{m/s}$ while remaining competitive in high-speed energy efficiency. Project webpage: video2stl.

Figures & tables

Figure 1: Overview of Video2STL framework. A VLM parses an observation video into a semantic event trace and generates parametric STL formulas. Predicate thresholds and temporal bounds are then fitted to target-embodiment trajectories. Smooth rolling-window robustness over short-horizon formulas provides dense local guidance, while a causal prefix monitor over the long-horizon backbone yields milestone progress rewards for RL policy training.

	Video2STL (Qwen 3.8)			Video2STL (GPT-5.6)			Text2Reward			Heuristic
$\mathbf{v_{x}}$ (m/s)	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$
0.3	100%	100%	$2.63\pm 0.08$	100%	100%	$2.63\pm 0.06$	100%	100%	$0.91\pm 0.01$	100%	100%	$1.20\pm 0.00$
0.5	100%	100%	$1.86\pm 0.03$	100%	100%	$1.80\pm 0.04$	100%	100%	$0.80\pm 0.00$	100%	100%	$1.00\pm 0.00$
0.7	100%	100%	$1.58\pm 0.02$	100%	100%	$1.52\pm 0.03$	100%	100%	$0.75\pm 0.00$	100%	100%	$1.00\pm 0.00$
1.0	100%	100%	$1.37\pm 0.01$	100%	100%	$1.26\pm 0.01$	100%	100%	$0.74\pm 0.00$	100%	100%	$1.20\pm 0.00$
1.3	100%	100%	$1.22\pm 0.01$	100%	100%	$1.17\pm 0.01$	100%	100%	$0.79\pm 0.00$	100%	100%	$1.30\pm 0.00$

Table 1: Quadruped locomotion performance across commanded forward velocities. Survival and success are reported as percentages, and CoT denotes cost of transportation. Higher is better for survival and success; lower is better for CoT.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	GPT-5.6			Gemini 3.1			Qwen 3.8
$\mathbf{v_{x}}$ (m/s)	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$	Surv. $\uparrow$	Succ. $\uparrow$	CoT $\downarrow$
0.3	100%	100%	$2.63\pm 0.06$	100%	25%	$2.37\pm 0.09$	100%	100%	$2.63\pm 0.08$
0.5	100%	100%	$1.80\pm 0.04$	100%	100%	$1.97\pm 0.04$	100%	100%	$1.86\pm 0.03$
0.7	100%	100%	$1.52\pm 0.03$	100%	100%	$1.66\pm 0.03$	100%	100%	$1.58\pm 0.02$
1.0	100%	100%	$1.26\pm 0.01$	100%	100%	$1.26\pm 0.02$	100%	100%	$1.37\pm 0.01$
1.3	100%	100%	$1.17\pm 0.01$	100%	100%	$1.14\pm 0.01$	100%	100%	$1.22\pm 0.01$

Appendix

Table 2: Quadruped locomotion performance using STL specifications generated by different VLMs. Survival and success are reported as percentages, and CoT denotes cost of transportation. Higher is better for survival and success; lower is better for CoT.

ID	Formula	$\mathbf{w_{i}}$
$\varphi_{1}$	$\bigwedge_{\ell\in\mathcal{L}}\mathcal{R}(D_{\ell})$	0
$\varphi_{2}$	$\bigwedge_{\ell\in\mathcal{L}}\mathcal{R}(L_{\ell})$	0
$\varphi_{3}$	$\mathcal{R}(D_{\mathrm{FL}}\wedge D_{\mathrm{HR}})\wedge\mathcal{R}(D_{\mathrm{FR}}\wedge D_{\mathrm{HL}})$	0
$\varphi_{4}$	$\mathcal{R}(L_{\mathrm{FL}}\wedge L_{\mathrm{HR}})\wedge\mathcal{R}(L_{\mathrm{FR}}\wedge L_{\mathrm{HL}})$	0
$\varphi_{5}$	$\mathcal{R}(C_{\mathrm{FL}}\wedge C_{\mathrm{HR}})\wedge\mathcal{R}(C_{\mathrm{FR}}\wedge C_{\mathrm{HL}})$	1
$\varphi_{6}$	$\mathcal{R}(S_{\mathrm{FL}}\wedge S_{\mathrm{HR}})\wedge\mathcal{R}(S_{\mathrm{FR}}\wedge S_{\mathrm{HL}})$	1

Appendix

Table 3: GPT-5.6 specifications and final weights. Gray formulas were filtered out;

\varphi_{11}

was rejected before training.

D_{\ell},L_{\ell},C_{\ell},S_{\ell}

denote touchdown, liftoff, contact, and swing.

\mathcal{R}(p)=\mathbf{G}_{[0,H]}(\mathbf{F}_{[0,h_{c}]}p)

and

\mathcal{B}(p,q)=\mathbf{G}_{[0,H]}(p\rightarrow\mathbf{F}_{[0,h_{s}]}q)

Z,P

denote trunk-height and pitch stability. All tanh scales are

s_{i}=1

Parameter	Role	Value
Temporal
$H$	history-window setting, all modes	30 steps
$h_{c}$	recurrence bound	$(29,20,15)$ steps
$h_{s}$	response bound	$(14,10,7)$ steps
Mode hysteresis	walk/trot entry, exit	$0.72,\ 0.65$ m/s
	trot/bound entry, exit	$1.55,\ 1.45$ m/s

Appendix

Table 4: GPT-5.6 grounding and reward parameters. All values are fixed in the reward implementation, except the support count, which is specified by the VLM, and the filtering rule. Triples follow walk/trot/bound mode order;

\Delta t=0.02

ID	Formula	$\mathbf{w_{i}}$
$\varphi_{1}$	$\mathcal{R}(D_{\mathrm{HR}})\wedge\mathcal{R}(L_{\mathrm{HR}})$	0
$\varphi_{2}$	$\mathcal{R}(D_{\mathrm{FR}})\wedge\mathcal{R}(L_{\mathrm{FR}})$	0
$\varphi_{3}$	$\mathcal{R}(D_{\mathrm{HL}})\wedge\mathcal{R}(L_{\mathrm{HL}})$	0
$\varphi_{4}$	$\mathcal{R}(D_{\mathrm{FL}})\wedge\mathcal{R}(L_{\mathrm{FL}})$	0
$\varphi_{5}$	$\mathcal{B}(D_{\mathrm{HR}},D_{\mathrm{FR}})$	1
$\varphi_{6}$	$\mathcal{B}(D_{\mathrm{HL}},D_{\mathrm{FL}})$	0.70

Appendix

Table 5: Qwen specifications and final weights. Gray formulas were filtered out;

\varphi_{11}

was rejected before training.

D_{\ell},L_{\ell},C_{\ell},S_{\ell}

denote touchdown, liftoff, contact, and swing.

\mathcal{R}(p)=\mathbf{G}_{[0,H]}(\mathbf{F}_{[0,h_{c}]}p)

and

\mathcal{B}(p,q)=\mathbf{G}_{[0,H]}(p\rightarrow\mathbf{F}_{[0,h_{s}]}q)

Z,P,R

denote trunk-height, pitch, and roll stability. All tanh scales are

s_{i}=1

Parameter	Role	Value
Temporal
$H$	history-window setting, all modes	30 steps
$h_{c}$	recurrence bound	$(29,20,15)$ steps
$h_{s}$	response bound	$(14,10,7)$ steps
Mode hysteresis	walk/trot entry, exit	$0.72,\ 0.65$ m/s
	trot/bound entry, exit	$1.55,\ 1.45$ m/s

Appendix

Table 6: Qwen grounding and reward parameters, supplied by the stage-3 configuration except the VLM-specified support count and the filtering rule. Triples follow walk/trot/bound mode order;

\Delta t=0.02

ID	Formula	$\mathbf{w_{i}}$
$\varphi_{1}$	$\mathbf{G}_{[0,H]}Z\wedge\mathbf{G}_{[0,H]}P$	1
$\varphi_{2}$	$\mathbf{G}_{[0,H]}(\mathit{support\_count\_at\_least}(2))$	1
$\varphi_{3}$	$\mathcal{R}(D_{\mathrm{FL}})\wedge\mathcal{R}(D_{\mathrm{HR}})$	1
$\varphi_{4}$	$\mathcal{R}(L_{\mathrm{HL}})$	1
$\varphi_{5}$	$\mathcal{B}(D_{\mathrm{HL}},D_{\mathrm{FL}})$	1
$\varphi_{6}$	$\mathcal{B}(D_{\mathrm{HR}},D_{\mathrm{FR}})$	1

Appendix

Table 7: Gemini specifications and final aggregation priors. Gray formulas were filtered out.

D_{\ell}

and

L_{\ell}

denote touchdown and liftoff.

\mathcal{R}(p)=\mathbf{G}_{[0,H]}(\mathbf{F}_{[0,h_{c}]}p)

and

\mathcal{B}(p,q)=\mathbf{G}_{[0,H]}(p\rightarrow\mathbf{F}_{[0,h_{p}]}q)

Z

and

P

denote trunk-height and pitch stability. The first six rows are Gemini-generated; the remaining rows are implementation-supplied safety and tracking terms.

Parameter	Role	Value
Temporal
$H$ ( $H_{\mathrm{episode}}$ )	outer always bound	$(30,24)$ steps
$h_{c}$	recurrence bound	25 steps
$h_{p}$	phase-response bound	13 steps
$W$	velocity/yaw tracking bound	$(30,24)$ steps
Mode hysteresis	trot entry / walk return	$0.72,\ 0.65$ m/s

Appendix

Table 8: Gemini grounding and reward parameters. Parenthesized pairs follow walk/trot mode order; unpaired values are shared;

\Delta t=0.02

Explore similar work

Jul 20, 2026cs.RO

STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models

Natural-language robot instructions often specify more than a coarse task goal: they may impose spatial, temporal, and logical requirements that must remain satisfied throughout execution. We present STeP, a specification-based agentic framework that uses Signal Temporal Logic (STL) as an explicit interface between high-level language reasoning and low-level robot execution. Rather than encoding such requirements implicitly in a learned policy, STeP formalizes them as task specifications that can be decomposed across multi-stage manipulation, enforced during execution, monitored online, and used as structured feedback for replanning. We evaluate STeP on standard LIBERO and LIBERO-PRO, and introduce LIBERO-Constrained, a new benchmark for manipulation tasks with spatial, temporal, and logical requirements, together with real-world tabletop experiments. On LIBERO-PRO, STeP retains 54%-89% success across five of six evaluated perturbation settings, substantially outperforming VLA and code-as-policy baselines under distribution shift. On LIBERO-Constrained, STeP achieves 80% safe success across 49 task-constraint instances; on real-world tasks, it improves safe success over a specification-free model-based baseline across all four task categories, with gains of up to 45 percentage points. These results support explicit formal specifications as a practical interface between foundation-model reasoning and reliable robot execution.

Kasra Torshizi, Anukriti Singh, Sidharth Mathur +3

Department of Computer Science University of Maryland

Signal Temporal Logic Action Generation

Ask PDF

Jul 1, 2026cs.RO

Learning Gait-Aware Quadruped Locomotion with Temporal Logic Specifications

Reinforcement learning (RL) for quadruped locomotion commonly depends on fixed, hand-crafted, and Markovian reward functions that limit both interpretability of learned policies and lack explicit control over gait behaviors. We introduce a framework where distinct gaits are specified using parameterized constraints expressed in Signal Temporal Logic (STL). These include safety bounds, gait synchronization constraints, command tracking, and actuation bounds. From these specifications, we develop a reward shaping mechanism that provides learning agents a dense, continuous reward landscape that encodes desired behavior. We define parametric STL templates for three speed regimes (walking-trot, trot, bound), calibrate their parameters from reference rollouts, and compute rewards from using smooth approximations of STL robustness over the rollouts. The generated rewards can be used to provide shaped gradients compatible with Proximal Policy Optimization (PPO). We instantiate the approach on Google's Barkour quadruped robot in MuJoCo XLA (MJX). We use parallelization within the simulator to improve training speeds and use domain randomization to robustify learned policies. We show that compared to a baseline of hand-crafted rewards, the STL-shaped rewards yield tighter velocity tracking and more stable training. Videos can be found on our project website: https://stl-locomotion.github.io/.

Merve Atasever, Cagan Bakirci, Alfredo Reina Corona +2

University of Southern California, Los Angeles, California, USA

Locomotion Gait Dynamics Signal Temporal Logic

Ask PDF

Jul 13, 2026cs.RO

WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos

Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.

Jiahao Liu, Yupeng Zheng, Zhongpu Xia +10

CASIA · Anyverse Dynamics · UCAS +2

Latent Actions

Ask PDF