VLA architectures that pair a pretrained VLM with a flow-matching action expert have emerged as a strong paradigm for language-conditioned manipulation. Yet the VLM, optimized for semantic abstraction and typically conditioned on static visual observations, tends to attenuate fine-grained geometric cues and often lacks explicit temporal evidence for the action expert. Prior work mitigates this by injecting auxiliary visual features, but existing approaches either focus on static spatial representations or require substantial architectural modifications to accommodate temporal inputs, leaving temporal information underexplored. We propose Plug-in Visual Injection (PVI), a lightweight, encoder-agnostic module that attaches to a pretrained action expert and injects auxiliary visual representations via zero-initialized residual pathways, preserving pretrained behavior with only single-stage fine-tuning. Using PVI, we obtain consistent gains over the base policy and a range of competitive alternative injection strategies, and our controlled study shows that temporal video features (V-JEPA2) outperform strong static image features (DINOv2), with the largest gains on multi-phase tasks requiring state tracking and coordination. Real-robot experiments on long-horizon bimanual cloth folding further demonstrate the practicality of PVI beyond simulation.
Figures & tables
Figure 1: Overview of Plug-in Visual Injection (PVI). Typical VLAs condition the VLM on static images with language, providing limited temporal context; moreover, the VLM’s output representations may under-emphasize fine-grained geometric cues. PVI bypasses this bottleneck by injecting auxiliary visual representations directly into the frozen action expert via a trainable plug-in, with no backbone modification and no re-pretraining required.
Figure 2: Architecture overview. The frozen main DiT blocks receive semantic embeddings from a frozen VLM. A trainable DiT copy (PVI) conditions on auxiliary visual features and injects them into the main stream via zero-initialized linear projections to produce continuous actions.
Figure 3: Overview of PVI and four candidate strategies for injecting auxiliary visual features into the DiT action expert. We compare PVI (ours), which injects V-JEPA2 features via a trainable copy branch with zero-initialized injection layers, against input-level fusion ( Concat ), attention-level dual injection ( ControlVLA-style ), and parallel-branch designs with feature concatenation or residual addition ( ReferenceNet- and ControlNet-style ).
Task
GR00T N1.5
Ours
Concat
ControlNet
ControlVLA
ReferenceNet
adjust_bottle
92
95
90
86
91
96
beat_block_hammer
39
84
47
30
35
37
click_alarmclock
48
87
54
46
53
48
move_can_pot
37
51
36
30
37
37
click_bell
46
91
75
41
61
42
move_playingcard_away
23
84
39
29
26
61
Table 1: Comparing injection strategies in simulation (single-task fine-tuning). Success rates (%) over 100 evaluation rollouts per task with randomized initial conditions (different rollout seeds). Evaluated tasks span diverse manipulation primitives, including alignment and perception, press and switch, relocation and placement, articulation, and coordination with tools and stacking.
Figure 4: Encoder-agnostic visual injection and representation comparison. Per-task success rates and overall average across 10 simulation tasks, comparing a fine-tuned GR00T N1.5 baseline against three PVI instantiations with different auxiliary encoders.
Variant
Frames@4fps
Avg. (%)
Baseline (fine-tuned GR00T N1.5)
–
39.8
Temporal context (PVI@V-JEPA2)
PVI@V-JEPA2
2
71.6
PVI@V-JEPA2
4
71.8
PVI@V-JEPA2
8
69.4
PVI@V-JEPA2
16
66.6
Table 2: Ablations on temporal context and stabilization choices. Average success rates (%) over 100 evaluation rollouts per task with randomized initial conditions (different rollout seeds).
Figure 5: PVI enables long-horizon bimanual manipulation of deformable objects on real hardware. The task comprises eight sequential subtasks, each illustrated by three keyframes: sleeve localization and precise pinching (Step 1), global shape adjustment via dragging without wrinkling (Step 2), symmetric bimanual sleeve folding (Step 3), long-horizon edge-to-edge vertical half-fold (Step 4), contact-rich sliding with tension control (Step 5), asymmetric anchoring and collar lifting (Step 6), multi-stage final fold with shape retention (Step 7), and stable lift-and-stack (Step 8). All subtasks are driven by a single PVI-augmented policy conditioned on language instructions, without task-specific engineering or manual resets between steps.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Baseline
PVI
Concat
Quantity
Trainable
Total
Trainable
Total
Trainable
Total
VLM
0.000∣
1.655
0.000∣
1.655
0.000∣
1.655
Main DiT
0.550∣
0.550
0.000∣
0.550
0.550∣
0.550
Others
0.518∣
0.518
0.317∣
0.518
0.518∣
0.518
Aux encoder
0.000∣
0.000
0.000∣
1.012
0.000∣
1.012
Plug-in branch
0.000∣
0.000
0.585∣
0.591
0.003∣
0.003
Appendix
Table 3: Parameter breakdown for the main injection-method comparison in Sec. 4.1 (Part I). All numbers are in billions of parameters (B).
ControlVLA-style
ControlNet-style
ReferenceNet-style
Quantity
Trainable
Total
Trainable
Total
Trainable
Total
VLM
0.000∣
1.655
0.000∣
1.655
0.000∣
1.655
Main DiT
0.550∣
0.550
0.000∣
0.550
0.550∣
0.550
Others
0.518∣
0.518
0.317∣
0.518
0.317∣
0.518
Aux encoder
0.000∣
1.012
0.000∣
1.012
0.000∣
1.012
Plug-in branch
0.062∣
0.062
0.586∣
0.593
0.571∣
0.578
Appendix
Table 4: Parameter breakdown for the main injection-method comparison in Sec. 4.1 (Part II). All numbers are in billions of parameters (B).
PVI@V-JEPA2
PVI@DINOv2
PVI@DINOv2+V-JEPA2
Quantity
Trainable
Total
Trainable
Total
Trainable
Total
VLM
0.000∣
1.655
0.000∣
1.655
0.000∣
1.655
Main DiT
0.000∣
0.550
0.000∣
0.550
0.000∣
0.550
Others
0.317∣
0.518
0.317∣
0.518
0.317∣
0.518
Aux encoder
0.000∣
1.012
0.000∣
1.136
0.000∣
2.149
Plug-in branch
0.585∣
0.591
0.585∣
0.591
0.588∣
0.594
Appendix
Table 5: Parameter breakdown for the encoder comparison in Sec. 4.2. All numbers are in billions of parameters (B).
Task
Baseline
PVI
adjust_bottle
100%
99%
beat_block_hammer
67%
96%
click_alarmclock
88%
99%
move_can_pot
54%
52%
click_bell
99%
97%
move_playingcard_away
61%
73%
Appendix
Table 6: Per-task success rates on the 20-task mixture. All policies are trained with 50 demonstrations per task and evaluated over 100 rollouts per task. PVI improves the average success rate from 61.15% to 69.15% (+8.00 pp).
Task
Baseline
PVI
adjust_bottle
99%
100%
beat_block_hammer
43%
86%
click_alarmclock
85%
93%
move_can_pot
43%
51%
click_bell
98%
98%
move_playingcard_away
64%
90%
Appendix
Table 7: Per-task success rates on the 50-task mixture (Part I). All policies are trained with 50 demonstrations per task and evaluated over 100 rollouts per task. Across the full 50-task mixture, PVI improves the average success rate from 61.32% to 63.56% (+2.24 pp), outperforming the fine-tuned GR00T N1.5 baseline on 32 of 50 tasks and tying on 2 tasks.
Task
Baseline
PVI
move_stapler_pad
6%
8%
move_pillbottle_pad
39%
63%
pick_dual_bottles
46%
62%
place_a2b_right
56%
38%
place_bread_basket
67%
61%
place_bread_skillet
53%
56%
Appendix
Table 8: Per-task success rates on the 50-task mixture (Part II).
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China