Vision-Predictor-Action: Towards Efficient Embodied AI via Neuro-Symbolic JEPA Predictors

Yeonseok Lee

cs.LOOct 11, 2026

Vision-Predictor-Action: Towards Efficient Embodied AI via Neuro-Symbolic JEPA Predictors

Authors: Yeonseok Lee

Organizations: SLING AI Inc., Incheon, Republic of Korea

Abstract

Vision-Language-Action (VLA) models commonly generate robot actions token by token through Large Language Model (LLM) decoders, causing inference latency to grow with action sequence length and limiting closed-loop control frequency. We present Goal-Conditioned Vision-Predictor-Action (VPA), which replaces autoregressive decoders with a lightweight latent control loop. A shared vision encoder and a frozen text encoder embed observations, goal images, and instructions; a neuro-symbolic selector chooses a discrete skill; a Joint-Embedding Predictive Architecture (JEPA) predictor forecasts the next latent state; and a flow-matching solver generates an entire action chunk in a few integration steps, so that sequential network evaluations per decision step do not depend on the action horizon. We analyze inference complexity, regularization against representation collapse, and conditions under which latent sub-goal thresholding limits error accumulation across task phases. As an initial proof of concept, we evaluated VPA on the LIBERO-10 suite. The trainable model has under 20M parameters (checkpoint under 75 MB, excluding a frozen 63.4M-parameter CLIP text encoder); a decision step of a single-camera reference configuration takes 10-14 ms on an Apple M4; and trained only on LIBERO-10 demonstrations in under three hours on a single GPU, a two-camera policy succeeded in 40.0% of 200 evaluation episodes. These preliminary results from a single training run suggest the approach is workable. Code and weights are available at https://github.com/ylee-sling/Vision-Predictor-Action and https://huggingface.co/ylee-sling/vpa-libero10, and archived at https://doi.org/10.5281/zenodo.22960920.

Figures & tables

Panel of Figure 1 — Figure 1: VPA at a glance. Top: autoregressive VLAs decode actions token by token through a large language model, so the number of sequential forward passes grows with the action horizon; inference rates of about $1\text{--}5$ Hz have been reported for RT-2 [ 1 ] . Middle: VPA replaces the decoder with a short chain of small networks that produces an entire action chunk in $K+3$ sequential evaluations. Bottom: initial results of our reference implementation (Section 6 ). Latency was measured for the single-camera reference configuration (Section 5.1 ); the parameter count and checkpoint size exclude the frozen CLIP text encoder ( $63.4$ M parameters), which is downloaded separately; the success rate comes from a single training run; and the models were trained only on the LIBERO-10 demonstrations. The VLA and VPA figures do not come from a controlled comparison.

Panel of Figure 2 — Figure 2: High-level architectural overview of the Goal-Conditioned Vision-Predictor-Action (VPA) framework. Multimodal observations are projected into latent and text embeddings that condition the selection of an active primitive $u_{t}$ , the forecast of the next latent state $\hat{z}_{t+1}$ and its uncertainty $\sigma_{t+1}$ by a JEPA predictor, and a continuous flow-matching solver that generates an action chunk $A_{t}$ , of which an uncertainty-dependent prefix of length $H_{t}$ is executed.

Panel of Figure 3 — Figure 3: VPA Closed-Loop Inference Pipeline (Action Trajectory Generation). This diagram shows the online forward pass executed at each decision step during robot deployment. The current observation $I_{t}$ , the active milestone goal $I_{g}^{(m_{t})}$ , and the natural language instruction $x$ are encoded and used to select a primitive $u_{t}$ , forecast $(\hat{z}_{t+1},\sigma_{t+1})$ , and form the unified conditioning vector $e_{t}$ (Eq. 12 ), from which the solver synthesizes an action chunk $A_{t}$ . The predictive uncertainty $\sigma_{t+1}$ determines the executed prefix length $H_{t}$ (Eq. 15 ), and the milestone pointer $m_{t}$ is updated by the threshold rule of Eq. 9a .

Panel of Figure 4 — Figure 4: Offline Representation-Learning Objective. The online encoder $E_{\psi}$ and the predictor $P_{\omega}$ are trained jointly. The target latent $z_{t+1}$ is produced by a momentum (exponential-moving-average) copy $E_{\bar{\psi}}$ of the encoder under stop-gradient, and the VICReg variance and covariance terms are evaluated on the batch $Z$ of online latents, since the target branch receives no gradients. During this stage, the predictor is conditioned on the discrete primitive labels $u_{t}$ supplied by logged demonstration trajectories (Eq. 10 ); the frozen text encoder is not involved. At the end of the stage, $E_{\psi}$ and $P_{\omega}$ are frozen, which fixes the latent scale $\bar{\gamma}^{\star}$ and the deployment threshold $\tau^{\star}$ (Eq. 9 ).

$K$	$H$	$t_{E}$	$t_{h}$	$t_{P}$	$t_{l}(H)$	Sum	Full step
1	8	3.732	0.333	0.692	1.473	6.229	10.481
1	16	3.766	0.336	0.711	1.482	6.295	10.617
1	32	3.714	0.334	0.687	1.485	6.220	10.484
1	64	3.740	0.333	0.692	1.568	6.333	10.726
1	128	3.867	0.337	0.715	1.865	6.783	11.166
2	8	3.756	0.334	0.701	1.457	7.707	12.235

Table 1: Measured latency of one VPA decision step on an Apple M4 (MPS backend, batch size one, float32) for the single-camera benchmark configuration described in the text. Entries are medians of

100

runs, in milliseconds. “Sum” is the network time

t_{E}+t_{h}+t_{P}+K\cdot t_{l}(H)

of Eq. 17 ; “Full step” is the complete decision step, which additionally includes non-network operations and the synchronization overhead of the per-stage timing. Because each entry is a separate median, the columns need not add up exactly.

Component	Setting
Observation $I_{t}$	third-person view, or third-person and wrist views; $128\times 128$ RGB
Proprioception $S_{t}$	end-effector position and orientation, gripper and joint states ( $d_{s}=15$ )
Actions	six-dimensional end-effector displacement and gripper command ( $d_{a}=7$ )
Encoder $E_{\psi}$	Vision Transformer, patch size $16$ , width $384$ , depth $6$ , $6$ heads, $d=256$ ; random initialization
Text encoder	frozen CLIP ViT-B/32 text encoder, $c_{\text{text}}\in\mathbb{R}^{512}$
Primitives	$N_{u}=4$ (reach, grasp, carry, release), derived from the gripper command

Table 2: Settings of the LIBERO-10 experiments. The listed values are defaults of the reference implementation and were not tuned; both models share them except for the camera views.

	Task	One camera	Two cameras
0	Alphabet soup and tomato sauce into the basket	$0$	$10$
1	Cream cheese box and butter into the basket	$0$	$45$
2	Turn on the stove and put the moka pot on it	$30$	$40$
3	Black bowl into the bottom drawer, then close it	$35$	$55$
4	White mug on the left plate, yellow-and-white mug on the right plate	$0$	$40$
5	Book into the back compartment of the caddy	$20$	$50$

Table 3: LIBERO-10 success rates (%) of the one-camera (third-person view) and two-camera (third-person and wrist views) models. Each task was evaluated on

20

trials; the aggregate row lists

95\%

Wilson intervals in brackets. One training run per model.

Executed horizon	Successes	Success (%)	$95\%$ interval
Fixed, $H_{t}=2$	$32/200$	$16.0$	$11.6\text{--}21.7$
Fixed, $H_{t}=4$	$52/200$	$26.0$	$20.4\text{--}32.5$
Fixed, $H_{t}=8$	$67/200$	$33.5$	$27.3\text{--}40.3$
Eq. 15 , $\beta=40.4$ (adaptive $H_{t}$ )	$61/200$	$30.5$	$24.5\text{--}37.2$
Eq. 15 , $\beta=1$ ( $H_{t}=15$ )	$80/200$	$40.0$	$33.5\text{--}46.9$

Table 4: Success rate of the two-camera model on LIBERO-10 as a function of the number of actions executed per decision step (

H=16

). All rows use the same checkpoint and the same

200

episodes; intervals are

95\%

Wilson intervals. The value

\beta=40.4

was chosen so that the 90th percentile of

\sigma_{t+1}

in the

\beta=1

run gives

H_{t}=H_{\min}

; it was not tuned for task success.

Panel of Figure 5 — Figure 5: Planned Closed-Loop Evaluation in Simulation (Future Work). To benchmark high-frequency reactive control, the Goal-Conditioned VPA framework will be deployed in GPU-accelerated simulators subjected to dynamic physical and visual perturbations. High-frequency sensory streams are processed by the VPA pipeline, whose sequential depth of $K+3$ network evaluations per decision step is independent of the action horizon (Proposition 5.1 ), and the resulting action chunks are executed open-loop for an uncertainty-dependent number of steps $H_{t}$ . This setting is intended to test, under realistic closed-loop conditions, whether VPA avoids the low inference rates reported for autoregressive VLA architectures, such as about $1\text{--}5$ Hz for RT-2 [ 1 ] . Because the simulator exposes the privileged physical state $y_{t}$ , the evaluation will also log the quantities required to test the conditional analysis of Section 5.3 .

Flow Matching for Action Generation

See all 21 papers

Oct 7, 2026

Juno: Taming Predictive Latents for Vision-Language-Action Models

Joint-embedding predictive architectures (JEPAs) predict masked or future observations in representation space, offering a natural source of predictive latents for vision-language-action (VLA) models. Yet making these latents useful across pretraining, policy learning, and deployment requires addressing three failures: mismatch with embodiment-specific control, interference with action learning, and teacher miscalibration under distribution shifts. We introduce Juno, a unified framework built around one action-conditioned JEPA that serves as a control-aligned representation backbone, a predictive teacher, and an adaptable dynamics model. During pretraining, we train it on embodiment-matched trajectories and use a dynamic CLS loss to transfer motion-weighted patch dynamics to a compact global state. During policy learning, we fuse current-frame JEPA patches into VLA perception and use a decoupled reasoning branch with separate transformation parameters to distill future latent states for action generation. During deployment, we adapt the world model on all observed transitions, including failed rollouts, freeze the adapted teacher, and re-align the policy on verified executions using LoRA adapters and a trainable action head, without expert corrections or task rewards. On SimplerEnv, Juno raises average success from

60.9\%

68.5\%

over Qwen3GR00T, the strongest baseline, and test-time adaptation further reaches

72.7\%

; on a real robot, it retains

70\%

75\%

success under background, height, and object shifts where the base policy collapses to

0\%

Yuchen Zhu, Chenyi Xu, Yulin Zhang +2

University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Hangzhou Dianzi University +1

Vision-Language-Action Models World Models for Robotics Joint-Embedding Predictive Architecture Robotic Manipulation Test-Time Adaptation of VLA Models

Ask PDF

Sep 10, 2026

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies

Vision-language-action (VLA) policies leverage pretrained vision-language backbones to achieve strong cross-task generalization. A leading design couples this backbone with a dedicated continuous action head trained via diffusion or flow matching. However, such heads rely on iterative multi-step sampling, for example 10 Euler steps in

π_{0.5}

. This creates an inference bottleneck that produces stop-and-go movement in the robot and slower task completion. We introduce IMLE-VLA, which replaces the iterative action head with a single-step conditional generator trained via conditional Implicit Maximum Likelihood Estimation (cIMLE). The cIMLE objective promotes multimodal action coverage, avoiding the mode collapse of naive regression heads while eliminating multi-step sampling entirely. When IMLE-VLA is applied to

π_{0.5}

, it increases inference frequency 3.67x (55 Hz vs. 15 Hz), enabling up to 11x higher action throughput. On the 40-task LIBERO benchmark, IMLE-VLA achieves the highest average success rate (98.0%) among all baselines while leading in inference frequency. Under the test-time perturbations of LIBERO-plus, IMLE-VLA retains

π_{0.5}

's robustness while other baselines degrade sharply, confirming that the cIMLE head preserves generalization. Real-world experiments on a Franka Emika Panda across four tasks demonstrate smoother motion (2.2x to 3.0x lower jerk) and faster task completion, with IMLE-VLA outperforming

π_{0.5}

on every task and reducing average VLA inference time per episode by 3.9x to 6.6x. Videos and code are available at https://kianhk6.github.io/IMLE-VLA/

Kian Hosseinkhani, Qinhe Peng, George Shramko +6

Simon Fraser University · University of Pennsylvania · Alberta Machine Intelligence Institute +1

Efficient VLA Model Inference Vision-Language-Action Models Policy Learning VLA Model Training

Ask PDF

Oct 8, 2026

PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies

Learning to predict how the world evolves can provide vision-language-action (VLA) policies with predictive context for long-horizon control, but its effectiveness depends on what future representation is modeled and how it conditions action generation. We introduce PLaW-VLA, which models task-relevant future states in a pretrained prediction-oriented representation space, reducing the need to predict control-irrelevant visual details. Built on a Mixture-of-Transformers architecture, PLaW-VLA conditions action generation on observation history, current task semantics, and predicted future states through structured causal attention. Experiments show a +11.8 percentage-point (pp) gain over reactive policies on RoboTwin Hard Horizon III and a +1.77 pp gain over reconstruction-oriented latent prediction on zero-shot LIBERO-Plus, supporting improved long-horizon control and generalization under distribution shift, respectively. By avoiding low-level visual reconstruction, PLaW-VLA lowers the burden of future prediction, enabling a lightweight latent world model with parallel future prediction and about 1/19 the inference latency of generative world-action modeling at comparable policy performance.

Yu Liu, Hetian Guo, Tianlv Huang +11

Jilin University · Astribot · Harbin Institute of Technology, Shenzhen +2

Vision-Language-Action Models Latent World Models World Model Learning Long-Horizon Robotic Manipulation Efficient Inference for World Action Models

Ask PDF

Abstract

Flow Matching for Action Generation

Explore similar work

Juno: Taming Predictive Latents for Vision-Language-Action Models

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies

PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies