Task-Prototype Guided Flow Matching for Few-Shot Generalization in Vision-Language Robot Manipulation

Yizhao Wang; Guantao Zhang; Jingbo Wang

cs.ROAug 18, 2026

Task-Prototype Guided Flow Matching for Few-Shot Generalization in Vision-Language Robot Manipulation

Authors: Yizhao Wang, Guantao Zhang, Jingbo Wang

Organizations: School of Computer Science, Henan Institute of Science and Technology

Abstract

Vision-language robot manipulation policies can follow semantic instructions, but adapting them to a new procedure from only a few demonstrations remains difficult because language underspecifies contact timing, motion phases, corrective behavior, and execution style. This paper presents Task-Prototype Guided Flow Matching (TP-Flow), a few-shot manipulation framework that converts support demonstrations into structured task-prototype tokens and uses them to guide both the initial flow prior and the velocity field. TP-Flow employs symmetric cross-attention with learnable queries to extract phase-level prototypes, parameterizes a task-adaptive initial distribution, and injects prototype information through gated adaptive normalization. It is trained with an episodic support-query objective and prototype contrastive regularization, so few-shot adaptation is simulated during training while nuisance information is suppressed. On the LEROBOT-ARM-SO101 platform, TP-Flow achieves 66.8%, 79.6%, and 82.1% success rates under 1-, 4-, and 6-shot settings, with a 75.5% few-shot AUC. At 1-shot, it improves over CFM, Pooled-Demo CFM, and In-Context Flow by 29.8, 14.5, and 9.9 percentage points. It also improves held-out target-group generalization across novel-object transfer, goal recombination, long-horizon composition, and contact/correction tasks. TP-Flow maintains real-time execution with six online prototype tokens, 54.3 ms latency, 3.9 GB peak memory, and a 10 Hz control rate, while reducing the noisy-support success drop to 6.2%. Theoretical diagnostics show that prototype distance aligns with action-distribution distance, the adaptive prior reduces transport cost, and gated modulation keeps measured trajectory deviations below the derived ODE bound. The code repository is omitted for anonymous review.

Figures & tables

Figure 1: Overview of TP-Flow. Few-shot support demonstrations are encoded into structured task-prototype tokens. The prototypes parameterize a task-adaptive flow prior and modulate the velocity field through gated adaptive normalization.

Figure 2: Internal structure of the shared multimodal observation encoder $f_{\mathrm{obs}}$ . The RGB branch, robot-state branch, and language branch produce $\mathbf{v}_{t}$ , $\mathbf{s}_{t}$ , and $\mathbf{l}$ , which are fused by a query-fusion block to obtain the current condition $\mathbf{h}_{t}$ . The same encoder weights are reused for support frames to compute $\mathbf{h}_{k,j}^{S}$ ; action and temporal information are then added by the support-token construction in Eqs. 7 – 8 .

Figure 3: Internal structure of the gated prototype-modulated velocity network. The flow state

\mathbf{x}_{s}

, flow time

s

, and observation condition

\mathbf{h}_{t}

are embedded into

\mathbf{y}^{(0)}

. Each velocity layer queries the task prototypes

\mathbf{P}_{\tau}

to generate scale, shift, and gate parameters for gated adaptive normalization, and the output head predicts

v_{\theta}(\mathbf{x}_{s},s,\mathbf{h}_{t},\mathbf{P}_{\tau})

Figure 4: LEROBOT-ARM-SO101 robotic platform used in all experiments. The platform provides the visual-proprioceptive observation stream and gripper-based manipulation interface used by TP-Flow.

Split	Task group	Representative task examples
Source	short-horizon prior	grasp a red cube; place a blue cube into a gray tray; push a green block to a target mark; close a book; grasp a spray bottle; press a stapler
Target	spatial transfer	place the cube in the left tray; move a cup to the back-right region; push a puck to a shifted mark; place a marker into a pen holder; place headphones into a paper box
Target	object transfer	grasp a spray bottle with changed pose; pick a soft sponge block; place a transparent cup upright; grasp a thin marker pen; handle folded headphones near a paper box
Target	goal transfer	pour water from a bottle into a cup; place a cup on a coaster; put a marker into a pen holder; cover a tea canister with its lid; transfer a block from bowl to tray
Target	long-horizon	pour water into a cup after bottle alignment; cover the tea canister after lid pickup; place headphones into a paper box; carry a cube around an obstacle and place it
Diag.	robustness	cluttered support demo; eight-frame support subsampling; perturbed support object pose; failed lid alignment; stapler press with shifted contact point; small support-view occlusion

Table 1: LEROBOT-ARM-SO101 task groups and few-shot generalization settings.

Hyperparameter	Value
RGB resolution	$224\times 224$
Robot-state dimension	$d_{q}=10$
Action dimension	$d_{a}=7$
Hidden dimension	$d=512$
Support shots	$K\in\{1,2,4,6\}$
Support tokens per demo	$T_{k}^{\prime}=16$

Table 2: Main implementation hyperparameters used for TP-Flow.

Method	SR@1 $\uparrow$	SR@2 $\uparrow$	SR@4 $\uparrow$	SR@6 $\uparrow$	FS-AUC $\uparrow$	Gain@1 $\uparrow$	Horizon $\downarrow$
ResNet-T	$28.4$	$31.2$	$34.7$	$36.1$	$32.6$	$-8.6$	$146.3$
Transformer BC	$31.6$	$34.0$	$37.9$	$39.2$	$35.7$	$-5.4$	$139.7$
OpenVLA	$42.8$	$45.5$	$48.9$	$50.1$	$46.8$	$+5.8$	$121.5$
ACT	$39.5$	$43.2$	$47.6$	$49.3$	$44.9$	$+2.5$	$118.9$
Diffusion Policy	$45.1$	$49.7$	$53.8$	$55.6$	$51.1$	$+8.1$	$113.4$
FlowPolicy	$48.7$	$53.5$	$57.9$	$59.8$	$55.0$	$+11.7$	$106.8$

Table 3: Experimental few-shot task-performance results on held-out LEROBOT-ARM-SO101 target tasks. SR denotes success rate, FS-AUC averages success over

K\in\{1,2,4,6\}

, and Gain@1 is measured against CFM at 1-shot.

Target group	Best-base method	Best-base SR $\uparrow$	TP-Flow SR $\uparrow$	Gain $\uparrow$	Avg. horizon $\downarrow$
Spatial transfer	In-Context Flow	$73.6$	$\mathbf{83.9}$	$+10.3$	$\mathbf{76.4}$
Object transfer	In-Context Flow	$63.4$	$\mathbf{77.8}$	$+14.4$	$\mathbf{83.1}$
Goal transfer	Pooled-Demo CFM	$60.5$	$\mathbf{75.2}$	$+14.7$	$\mathbf{88.6}$
Long-horizon	In-Context Flow	$58.1$	$\mathbf{71.9}$	$+13.8$	$\mathbf{96.5}$
Robust diagnostic	Pooled-Demo CFM	$51.7$	$\mathbf{68.3}$	$+16.6$	$\mathbf{104.2}$

Table 4: Experimental target-group generalization results. Best-base is the strongest non-TP-Flow baseline within each group. Gain is the absolute success-rate improvement of TP-Flow over Best-base.

Figure 5: Few-shot success curves from the experimental results. The figure emphasizes how TP-Flow benefits most when the support set is extremely small.

Figure 6: Target-group generalization comparison from the experimental results. The figure shows whether prototype guidance is most useful for object transfer, goal recombination, long-horizon execution, or noisy-support settings.

Figure 7: Quality, efficiency, and robustness diagnostics from the experimental results. The figure visualizes the trade-off between success rate, smooth action generation, online cost, and robustness to noisy support demonstrations.

Method	SR@4 $\uparrow$	Smoothness $\downarrow$	Replan error $\downarrow$	Action norm $\downarrow$	Contact recovery $\uparrow$	Jerk events $\downarrow$
OpenVLA	$48.9$	$0.184$	$0.091$	$1.42$	$34.6$	$7.8$
ACT	$47.6$	$0.121$	$0.064$	$1.18$	$41.3$	$5.1$
Diffusion Policy	$53.8$	$0.098$	$0.058$	$1.12$	$46.7$	$4.4$
FlowPolicy	$57.9$	$0.087$	$0.052$	$1.06$	$50.8$	$3.9$
CFM	$42.1$	$0.112$	$0.069$	$1.23$	$38.2$	$5.6$
Pooled-Demo CFM	$62.4$	$0.083$	$0.049$	$1.05$	$55.9$	$3.6$

Table 5: Experimental action-quality and trajectory-stability results at 4-shot evaluation. Smoothness is the mean squared second-order action difference; lower values indicate smoother generated chunks.

Method	Clean SR $\uparrow$	Noise-drop $\downarrow$	Proto. stab. $\uparrow$	Latency ms $\downarrow$	Context len. $\downarrow$	Mem. GB $\downarrow$	Gate/noise corr. $\downarrow$
Pooled-Demo CFM	$62.4$	$15.8$	$0.71$	$42.6$	1	$3.2$	$0.62$
In-Context Flow	$68.0$	$13.1$	$0.78$	$91.4$	64	$6.8$	$0.55$
TP-Flow w/o adaptive prior	$73.5$	$9.7$	$0.84$	$55.9$	6	$3.9$	$0.39$
TP-Flow w/o gated modulation	$70.6$	$14.4$	$0.83$	$52.8$	6	$3.8$	$0.58$
TP-Flow w/o contrastive loss	$72.1$	$12.6$	$0.76$	$53.7$	6	$3.8$	$0.47$
TP-Flow	$\mathbf{79.6}$	$\mathbf{6.2}$	$\mathbf{0.91}$	$\mathbf{54.3}$	6	$\mathbf{3.9}$	$\mathbf{0.24}$

Table 6: Experimental robustness, efficiency, and internal diagnostic results. Noise-drop is the absolute SR decrease from clean support to perturbed support; context length is the online conditioning length used by the velocity network.

Figure 8: Photographic trajectory visualization in the main-result setting. The five-row by four-column grid shows real LEROBOT-ARM-SO101 rollout frames for five executable tasks. Each row corresponds to one task, and the columns show the initial query frame, approach or grasp phase, contact or task-motion phase, and final completion state.

Target group	Representative query tasks	Best-base method	Best-base SR $\uparrow$	TP-Flow SR $\uparrow$	Gain $\uparrow$	Horizon $\downarrow$
Seen-task variation	close a book; grasp a spray bottle with a changed initial pose; press a stapler from a shifted contact point	In-Context Flow	$76.2$	$\mathbf{84.7}$	$+8.5$	$\mathbf{74.9}$
Novel object	grasp a spray bottle; place folded headphones into a paper box; grasp a thin marker pen	In-Context Flow	$63.4$	$\mathbf{77.8}$	$+14.4$	$\mathbf{83.1}$
Goal recombination	pour water from a bottle into a cup; place a marker into a pen holder; cover a tea canister with its lid	Pooled-Demo CFM	$60.5$	$\mathbf{75.2}$	$+14.7$	$\mathbf{88.6}$
Long-horizon composition	align bottle and pour water into a cup; pick up a lid and cover the tea canister; place headphones into a paper box after reorientation	In-Context Flow	$58.1$	$\mathbf{71.9}$	$+13.8$	$\mathbf{96.5}$
Contact and correction	press a stapler; recover marker insertion after rim contact; correct lid pose before covering the tea canister	Pooled-Demo CFM	$51.7$	$\mathbf{68.3}$	$+16.6$	$\mathbf{104.2}$

Table 7: Cross-task and novel-object generalization results on held-out LEROBOT-ARM-SO101 target tasks. Best-base denotes the strongest non-TP-Flow method within each group.

Variant	SR@1 $\uparrow$	SR@4 $\uparrow$	FS-AUC $\uparrow$	Noise-drop $\downarrow$	Proto. stab. $\uparrow$	Latency ms $\downarrow$
Full TP-Flow	$\mathbf{66.8}$	$\mathbf{79.6}$	$\mathbf{75.5}$	$\mathbf{6.2}$	$\mathbf{0.91}$	$54.3$
w/o structured prototype	$55.2$	$68.8$	$64.0$	$12.9$	$0.74$	$\mathbf{50.6}$
w/o reverse attention path	$58.9$	$72.5$	$68.1$	$10.7$	$0.81$	$52.1$
w/o adaptive prior	$60.4$	$73.5$	$69.8$	$9.7$	$0.84$	$55.9$
w/o gated AdaLN	$57.8$	$70.6$	$66.5$	$14.4$	$0.83$	$52.8$
w/o contrastive loss	$59.6$	$72.1$	$68.7$	$12.6$	$0.76$	$53.7$

Table 8: Measured component-level ablation results on held-out LEROBOT-ARM-SO101 target tasks. All variants use the same source-task training split and the same

K=4

support-shot evaluation protocol unless otherwise stated.

Figure 9: Measured component-level ablation analysis. Panel (a) shows the SR@4 degradation caused by removing or replacing each component. Panel (b) relates prototype stability to the success-rate drop under noisy support demonstrations.

Figure 10: Measured sensitivity analysis of TP-Flow inference settings. Panel (a) varies the number of prototype tokens

M

, while panel (b) varies the number of ODE integration steps. The selected default values balance success rate and latency.

Method	Online conditioning	Scaling	Context len. $\downarrow$	Latency ms $\downarrow$	Peak mem. GB $\downarrow$	Control rate Hz $\uparrow$
CFM	none	$\mathcal{O}(1)$	0	$49.8$	$3.1$	$10.0$
Pooled-Demo CFM	pooled support vector	$\mathcal{O}(1)$	1	$\mathbf{42.6}$	$\mathbf{3.2}$	$10.0$
In-Context Flow	raw support tokens $\mathbf{Z}_{\tau}$	$\mathcal{O}(L_{\tau})$	64	$91.4$	$6.8$	$8.7$
TP-Flow w/ $M=4$	prototype tokens $\mathbf{P}_{\tau}$	$\mathcal{O}(M)$	4	$52.7$	$3.7$	$10.0$
TP-Flow w/ $M=6$	prototype tokens $\mathbf{P}_{\tau}$	$\mathcal{O}(M)$	6	$54.3$	$3.9$	$\mathbf{10.0}$
TP-Flow w/ $M=8$	prototype tokens $\mathbf{P}_{\tau}$	$\mathcal{O}(M)$	8	$57.1$	$4.1$	$10.0$

Table 9: Measured inference-efficiency comparison on LEROBOT-ARM-SO101. Latency is measured per generated action chunk under the 4-shot evaluation setting.

Figure 11: Measured prototype-geometry diagnostics. Panel (a) shows that prototype embeddings form local neighborhoods corresponding to manipulation groups. Panel (b) compares prototype distance

d_{P}

with empirical action-distribution distance

d_{A}

. Panel (c) reports how correlation and same-group retrieval improve as the number of support demonstrations increases.

Figure 12: Measured inference-efficiency profile. Panel (a) compares the latency scaling of raw in-context support conditioning and TP-Flow prototype conditioning as the available support context grows. Panel (b) decomposes action-chunk latency into observation encoding, support/prototype processing, flow generation, and other overhead.

Generalization in Robotic Manipulation

Generalization in robotic manipulation studies how robots transfer grasping and interaction skills to new objects, scenes, tasks, and embodiments.

See all 353 papers

TOPS: Reshaping Flow Source and Transport for Multi-Task Robot Manipulation
Oct 27, 2026Tuo An, Chuhao Zhou, Jiahua Dong +4Flow-Based Policy Learning Language-Conditioned Robot Manipulation
MARS Lab, Nanyang Technological University
NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields
Oct 1, 2026Shota Kobayashi, Koki Seno, Daichi Yashima +1Language-Conditioned Robot Manipulation Cross-Embodiment Robot Learning
Keio University, Japan
Cue the Flow: Steering Flow-Matching Policies for Open-World Delivery Manipulation
Sep 30, 2026Haoxuan Wang, Griffin Galimi, Junhua Huang +4Language-Conditioned Robot Manipulation Generalization in Robotic Manipulation
University of Illinois Chicago · University of California, Los Angeles

Oct 27, 2026J/K move · Enter open · S save

TOPS: Reshaping Flow Source and Transport for Multi-Task Robot Manipulation

Tuo An, Chuhao Zhou, Jiahua Dong, Jingliang Li+3

MARS Lab, Nanyang Technological University

Flow-Based Policy Learning Language-Conditioned Robot Manipulation Multi-Task Robotic Manipulation Flow Matching for Action Generation Conditional Flow Matching

Open paper AskPDF arXiv

Flow-based policies for language-conditioned multi-task manipulation commonly share a single

\textit{Gaussian}

source prior and a common linear transport path across tasks. However, reliably executing multiple tasks with distinct motion requirements remains challenging for a shared policy. We introduce

\textbf{TOPS}

(

\textbf{T}

ask-

\textbf{O}

riented

\textbf{P}

riors and

\textbf{S}

chedules), which adapts the flow source prior and interpolation schedule to each task without modifying the network architecture. TOPS first constructs multimodal task sources from reverse-mapped expert action sequences, allowing language instructions to select where generation begins. Guided by source-target covariance mismatch, it then slows the transition toward expert actions early in denoising and accelerates it later along directions requiring expansion. Under a

\textit{Gaussian}

endpoint model, our analysis shows that this schedule adjustment can lower an upper bound on velocity-prediction risk. Experiments on LIBERO, LIBERO-Plus, and CALVIN demonstrate improved performance across in-distribution, perturbation and cross-environment settings, alongside better data efficiency. Using one integration step, TOPS raises LIBERO-10 success from

53.44\%

69.67\%

compared with ten-step Standard Flow Matching (FM), while reducing inference time to approximately one-sixth of the baseline. TOPS also yields consistent gains when transferred to the large-scale flow-based

π_{0.5}

policy across all four evaluated settings. Real-world experiments further demonstrate improved task success, including an increase from

20\%

with standard FM to

80\%

with TOPS on the upper-drawer opening task.

Abstract

Generalization in Robotic Manipulation

Explore similar work

TOPS: Reshaping Flow Source and Transport for Multi-Task Robot Manipulation

NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields

Cue the Flow: Steering Flow-Matching Policies for Open-World Delivery Manipulation