We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only minimal embodiment-specific data. This task is challenging, as object trajectory generation from pre-manipulation images and natural language instructions requires appropriate instruction-flow alignment. To tackle this challenge, we propose the flow-based Language Instruction-guided open-Loop ACtion generator (LILAC). This flow-based Vision-Language-Action model (VLA) generates object-centric 2D optical flow from an RGB image and a natural language instruction, and converts the flow into a 6-DoF manipulator trajectory. LILAC incorporates two key components: Semantic Alignment Loss, which strengthens language conditioning to generate instruction-aligned optical flow, and Prompt-Conditioned Cross-Modal Adapter, which aligns learned visual prompts with image and text features to provide rich cues for flow generation. Experimentally, our method outperformed existing approaches in generated flow quality across multiple benchmarks. Furthermore, in physical object manipulation experiments using free-form instructions, LILAC demonstrated a superior task success rate compared to existing methods. The project page is available at https://lilac-75srg.kinsta.page/.
Figures & tables
Fig. 1: Overview of LILAC, 2D object-centric optical flow-based Vision-and-Language trajectory generation framework. In this figure, ‘Act. De-Tokenizer’ represents Action De-Tokenizer. Given a natural language instruction, LILAC generates 2D flow from an RGB image and the instruction, and converts the flow into a 6-DoF robot trajectory.
Fig. 2: Overview of the LILAC framework. The inputs to the flow generation module are a single RGB image ximg and a natural language instruction xinst . In this module, first, visual prompts and bounding boxes are generated by the MLLM. Subsequently, 2D flow is autoregressively generated by the Prompt-Conditioned Multi-Modal Adapter and the Semantic Reconstruction Decoder. After generating the 2D flow, based on the flow, ximg , xinst , and xdepth , a 6-DoF manipulator trajectory is generated by the Action De-Tokenizer.
Fig. 3: Examples of visual prompts generated by GPT-4o. GPT-4o outputs the start and end points of an arrow that represents the rough shape of the action from a text prompt and an RGB image as input. In this figure, the red arrows indicate the rendered visual prompts.
Method
Fractal
BridgeData V2
ADE ↓
AUC ↑
P@5 ↑ [%]
P@10 ↑ [%]
P@20 ↑ [%]
ADE ↓
AUC ↑
P@5 ↑ [%]
P@10 ↑ [%]
P@20 ↑ [%]
Im2Flow2Act [ 16 ]
56.46
0.190
13.4
16.2
27.4
63.19
0.177
13.3
15.4
24.5
FLIP [ 15 ]
44.41
0.165
6.9
14.1
28.4
41.95
0.200
9.8
18.0
32.2
LILAC w/o srl
28.94
0.357
14.8
32.0
60.2
31.14
0.325
16.3
30.0
51.2
LILAC w/o vp
33.65
0.253
9.0
20.9
45.8
37.09
0.211
9.7
17.9
35.6
LILAC (full)
26.98
0.434
21.4
42.1
66.8
29.44
0.396
22.9
38.2
57.6
TABLE I: Quantitative comparison on the Fractal and BridgeData V2 subsets in the Robot Flow benchmark. In addition, the rows “LILAC w/o srl” and “LILAC w/o vp” respectively show the results of the ablation studies on Semantic Reconstruction Loss and the visual prompt, respectively. Lower values of ADE are better. Meanwhile, higher values of AUC and P@K are better. Best values per column are shown in bold .
Fig. 4: Qualitative results on the Robot Flow benchmark. The given xinst was (a) “Take the yellow block to put it on top of the tower.” and (b) “Close top drawer.”, respectively.
Fig. 5: Qualitative results with and without visual prompts. ‘Visual prompt’, ‘w/o Visual prompt’, and ‘w/ Visual prompt’ denote the visual prompt generated by the MLLM, the flow generated without a visual prompt, and the flow generated with a visual prompt, respectively. In each example, the given instruction sentence is as follows: (a) “Move 7up can near brown chip bag.” (b) “Pick black chip bag.”
Error category
#Samples
Incorrect end point
12
Overly short visual prompt
6
Incorrect start point
5
Invalid visual prompt
5
Reversed direction
5
No visual prompt generated
4
TABLE II: Error categories in the visual prompt failure analysis. We analyzed 100 examples randomly sampled from the Fractal subset, and identified 37 failure cases.
Fig. 6: Top row: (a) Representative example of Incorrect end point. In this example, the instruction “Move water bottle near paper bowl.” was given. (b) Representative example of ‘Overly short visual prompt’. The given instruction sentence was “Pick apple.” Bottom row: Example of flow generation conditioned on an incorrect visual prompt. The instruction “Place green jalapeno chip bag into top drawer.” was given.
Fig. 7: (a) HSR used as the robot platform; (b) objects used for data collection; (c) objects used for evaluation.
Fig. 8: Success rates of different methods on physical experiments. We compared our proposed method against three baseline approaches (Im2Flow2Act, FLIP, and π0 ) across five manipulation tasks and report the average success rate. Each task was evaluated over 20 trials. Bold numbers indicate the highest success rate for each task.
Fig. 9: Compounding error at test time. The horizontal axis represents the action step, and the vertical axis represents the Euclidean distance to the human demonstration trajectory.
Fig. 10: Qualitative results of LILAC executing tasks on the real HSR platform. In each example, the following instructions are given: (a) “Could you take the coke.” (b) “Place the cup near the orange.” (c) “Put the brick close to the bottle.” In this figure, (a) and (b) represent successful execution examples, and (c) shows a failure case. The 2D Flow column shows the 2D flow generated by the flow generation module.
Method
Inference speed [ms/action]
π0 [ 7 ]
541.0
Im2Flow2Act [ 16 ]
298.7
FLIP [ 15 ]
21.6
LILAC
376.6
TABLE III: Inference speed for trajectory generation in physical experiments. Lower is better. The fastest result is shown in bold.
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Zhejiang University, Hangzhou, China. · Simon Fraser University, Burnaby, BC, Canada. +1