Vision-Language-Action (VLA) models have shown strong performance on robotic manipulation, but they often struggle to generalize to unseen tasks, configurations, and long-horizon settings. A key challenge is that VLAs overfit to training scenes and fail to follow novel language instructions. Off-the-shelf vision-language models (VLMs) often provide stronger generalization, but cannot directly control robot actions. To combine the common sense of VLMs with VLA control, we propose Guided Trace VLA (GT-VLA), a steerable framework that accepts guidance from an external generalist VLM through trace-conditioned action generation. GT-VLA uses a generalist model to identify semantic guidance for the current skill, converts this guidance into a 2D visual trace, and conditions its action policy on the resulting trace-rendered observation. This design separates semantic target acquisition, trace generation, and low-level action execution, allowing high-level guidance to propagate to robot actions. GT-VLA uses a Mixture-of-Experts architecture with skill-specific trace and action modules for robust execution. We evaluate GT-VLA on LIBERO and a physical robot platform, showing improved generalization over recent VLA baselines in both settings. The code and additional supplemental materials are available on our project website at https://ivaniz.github.io/gt-vla/.
Figures & tables
Fig. 1: GT-VLA overview. An off-the-shelf generalist VLM decomposes tasks into skills and provides skill-specific guidance. This is converted into an image-space trace conditioning for the action policy, ensuring execution follows high-level intent.
Fig. 2: GT-VLA architecture. Left : a skill-routed trace expert generates a 2D trace anchored to the semantic target g(i) and end-effector position xt . Right : a skill-routed action expert predicts actions from the trace-overlaid observation, and an MLP predicts skill progress. A deterministic router selects experts for the current skill.
Fig. 3: Comparing steerability of GT-VLA against baselines and ablations. On the left we show a task from Libero-Object, which is an OOD task. GT-VLA (top-left) is steerable by the external semantic target. G-VLA executes memorized actions despite having correct semantic guidance (mid-left). AtomicVLA (bottom-left) also fails to generalize to this instruction. In real-world experiments, we show a 4-step OOD task. GT-VLA completes all skills, while AtomicVLA fails to adapt.
Fig. 4: Trace-conditioning augmentations (Sec. III-C ), shown on LIBERO (top row) and the real robot (bottom row).
Models
Goal
Spatial
Object
Avg.
π0.5
16.2 (13.1–19.7)
29.6 (25.6–33.8)
1.6 (0.7–3.1)
15.8
AtomicVLA
30.2 (26.2–34.4)
46.8 (42.4–51.3)
24.6 (20.9–28.6)
33.9
SEAL
31.0 (27.0–35.3)
45.0 (40.6–49.5)
6.0 (4.1–8.5)
27.3
GT-VLA (ours)
38.8 (34.5–43.2)
50.8 (46.3–55.3)
41.8 (37.4–46.3)
43.8
TABLE I: Success rate on LIBERO out-of-distribution evaluation suites, with 50 trials × 10 tasks per category (%). 95% Wilson confidence intervals are shown in parentheses.
TABLE II: Ablation results on LIBERO (%). Results use 50 trials × 10 tasks per category, except the generalist VLM swap rows, which use 20 trials × 10 tasks. Parentheses show 95% Wilson confidence intervals.
Fig. 6: Semantic target perturbations. Top: GT-VLA success rates across LIBERO suites when conditioned on noisy semantic targets. Bottom: Visual illustration of perturbations. Pink cross indicates given point, white circle indicates perturbation max. radius, red dot indicates perturbed final point.
Fig. 7: GT-VLA failure breakdowns for simulation, hardware, and noise injections.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Part
Parameters
Vision encoder (Gemma) [ 1 ]
400M
Shared VLM (Gemma) [ 1 ]
2.6B
Action heads (shared)
200M
Action head (per expert)
230M
Trace heads (shared)
72M
Trace head (per expert)
57M
Appendix
TABLE III: Approximate parameter allocations for GT-VLA and baselines.
Component
Runtime
Frequency
VLM planning
∼7s
1/task
VLM guidance
∼5s
1/skill
Trace generation
∼52ms
Every 2 action chunks
Action generation
∼74ms
N/A
Appendix
TABLE IV: Inference time for different components of GT-VLA.
Model
Goal
Spatial
Object
Avg.
LoHo-Manip (reprod.)
30.8 (26.9–35.0)
23.2 (19.7–27.1)
12.8 (10.2–16.0)
22.3
GT-VLA (ours)
38.8 (34.5–43.2)
50.8 (46.3–55.3)
41.8 (37.4–46.3)
43.8
Appendix
TABLE V: Supplementary comparison with LoHo-Manip under the same LIBERO OOD protocol as Table I. Results use 50 trials × 10 tasks per suite (%). Parentheses show 95% Wilson confidence intervals. The GT-VLA row is repeated from Table I for direct comparison.
Setting
Objects
table1
orange plate, blue plate, basket, eggplant, corn, carrot, green bell pepper
table2
white plate, green plate, orange plate, eggplant, corn, carrot, green bell pepper, asparagus
table3
blue plate, green plate, red bell pepper, green bell pepper, corn
table4
white plate, basket, green bell pepper, carrot, eggplant, asparagus
Appendix
TABLE VI: Real-world training-data settings.
ID
Task Instruction
Setting
Initial State
0
move the basket onto the blue plate and put the green bell pepper in the basket
table1
–
1
move the orange plate onto the basket
table1
–
2
move the eggplant into the basket and place the blue plate on the basket
table1
eggplant on blue plate
3
move the green plate onto the orange plate and move the asparagus and corn onto the white plate
table2
–
4
put the carrot onto the green plate
table2
green bell pepper on green plate
5
move the eggplant and carrot onto the white plate
table2
corn on white plate
Appendix
TABLE VII: Real-world training tasks. Objects are initially placed on the table unless otherwise specified; “–” denotes no additional initial-state constraint.
Fig. 8: Visual illustration of a task setup from the composition-6 test suite. The seen skill configurations identified in the figure can be found in Table VII .
Fig. 9: Visual illustration of a task setup from the OOD-6 test suite. The boxed skill configurations are unseen from the training dataset.