The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models

Chen Xu; Rishi Shah; Hadas Kress-Gazit; Haruki Nishimura; Masha Itkina

cs.ROSep 22, 2026

The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models

Authors: Chen Xu, Rishi Shah, Hadas Kress-Gazit, Haruki Nishimura, Masha Itkina

Organizations: Toyota Research Institute · Woven by Toyota · Cornell University

Abstract

Modern robot imitation learning increasingly relies on generative policies based on diffusion or flow-matching models, which generate actions by transforming samples from a prior distribution. A key question is whether the choice of prior matters. Replacing the standard Gaussian with a closer-to-target, non-Gaussian prior has been shown to substantially improve performance when training from scratch. A natural next step is to ask whether these gains transfer to fine-tuning pretrained Large Behavior Models (LBMs) such as LBM 1.0, $π_{0.5}$ , and GR00T~N1.5, where one might expect even larger gains. Surprisingly, we find that this is not the case, except possibly at very low fine-tuning data fractions. Across over 100K simulation rollouts spanning all three aforementioned LBMs on 40+ tasks in two simulation platforms, and 1250 hardware rollouts on five bimanual manipulation tasks, non-Gaussian priors that are demonstrably closer to the target yield statistically indistinguishable or worse fine-tuning performance than a standard Gaussian prior. Diagnostic analyses suggest why: fine-tuned imitation learning policies converge to similar action predictions across priors, despite their fine-tuned encoder embeddings diverging substantially from the pretrained embeddings and each other. A learning-rate ablation further confirms that encoder training is the dominant factor in fine-tuning performance, substantially outweighing the effect of prior choice. We conclude with concrete directions for future research on when and why learned priors might still matter in fine-tuning. Project page: https://cxu-tri.github.io/non_gaussian_FT/

Figures & tables

Panel of Figure 1 — Fig. 1: Prior definition and visualization. Bottom: schematic of an example flow-matching policy. The action head $v_{\theta}$ is conditioned on the encoder embedding $e$ and takes in diffusion step $t$ as well as the action prior $p_{0}$ . Only the prior varies across our experiments. Top: samples drawn directly from two examples of the prior $p_{0}$ on the simulated DumpVegetablesIntoContainer task from LBM Eval [ 3 ] , with overlaid colored dots showing both arms’ predicted end-effector positions over the action chunk horizon. Color gradient indicates temporal ordering within the chunk, with lighter dots representing predictions further into the future. The Gaussian $\mathcal{Z}$ (left) produces incoherent, random motion, while $A_{\text{pre}}$ (right) already makes partial task progress by picking up the avocado and moving it to the bin, reflecting a prior close to the target action. We hypothesize that the close-to-target prior $A_{\text{pre}}$ would outperform $\mathcal{Z}$ after policy fine-tuning, but we empirically find that this is not the case.

Fig. 2: Action priors help when training from scratch but not when fine-tuning (Setting A, 5 LBM Eval kitchen tasks [ 3 ] , 1000 rollouts per prior; see Section VI-A for details). All fine-tuning prior variants start from the same pretrained LBM 1.0 checkpoint.

A_{\text{pre}}

denotes the action prior sampled from the pretrained policy. Left:

\ell_{2}

distance between prior samples and ground-truth actions, in un-normalized action coordinates;

A_{\text{pre}}

3\times

closer to the target data than the Gaussian. Middle: from-scratch training;

A_{\text{pre}}

significantly outperforms the Gaussian. Right: after fine-tuning, the Gaussian prior matches or, in this case, exceeds

A_{\text{pre}}

in success rate. The full comparison across all seven prior variants ( Fig. 3 ) confirms that most priors we consider yield statistically indistinguishable performance after fine-tuning, and, thus, provide no benefit to the policy.

Proposed priors (constructed from $\pi_{\text{pre}}$ )
Prior	Description
$\mathcal{Z}$	Isotropic Gaussian $\mathcal{N}(0,I)$
$A_{\text{pre}}$	Actions generated by $\pi_{\text{pre}}$
$A_{\text{pre}}{+}\sigma_{A}\mathcal{Z}$	Actions from $\pi_{\text{pre}}$ with added Gaussian noise (scale $\sigma_{A}$ )
$E_{\text{pre}}{+}\sigma_{E}\mathcal{Z}$	Encoder embeddings with added Gaussian noise (scale $\sigma_{E}$ )
Baselines from the literature

TABLE I: The seven prior variants compared in this work.

\pi_{\text{pre}}

is the pretrained LBM (LBM 1.0,

\pi_{0.5}

, or GR00T N1.5). Appendix A gives full sampling procedures and auxiliary-training details.

Setting	Description	Model	Figures	# Rollouts
Main experiments
A	Do closer-to-target priors improve fine-tuning performance?	LBM 1.0 [ 3 ]	2 , 3	14000
B sim	Do priors matter across varying data fractions?	LBM 1.0 [ 3 ]	4	56000
Analysis	How similar are fine-tuned models across priors?		7 – 10	$\dagger$
Encoder LR	Does the encoder matter more than the prior?		11	9600
B real	Do priors matter on robot hardware?	LBM 1.0 [ 3 ]	5	1250

TABLE II: Index of all experimental settings, ablations, and analyses. Each row links to its corresponding figure(s).

\dagger

Reuses the rollouts of its corresponding main setting (no additional evaluations).

Fig. 3: Setting A: full seven-prior comparison; each violin corresponds to 1000 rollouts. Top: from scratch;

A_{\text{pre}}

statistically outperforms

\mathcal{Z}

, which is second lowest in empirical success rate among all priors. Bottom: fine-tuned; all priors perform comparably to

\mathcal{Z}

with six of seven priors sharing CLD group “a” or “b”.

Fig. 4: Fine-tuning performance across data fractions (Setting B sim , eight tasks). CLD letters are computed within each data fraction independently. All priors are statistically indistinguishable or worse than

\mathcal{Z}

across fractions. The one exception is

E_{\text{pre}}{+}\sigma_{E}\mathcal{Z}

at 5% data, which significantly outperforms

\mathcal{Z}

Fig. 5: Setting B real : hardware A/B testing (five bimanual FR3 tasks, 50 rollouts per prior per task, 1250 total). Top: Success rate violins; four of five priors share CLD group “a” and BRIDGER (group “b”) underperforms. Bottom: Partial-task-progress violins; the same grouping holds.

Fig. 6: Setting C: LBM fine-tuning on RoboCasa, with shared y-axis (Success Rate, 0–1) across all four panels. Left two panels:

\pi_{0.5}

on 18 atomic (3600 rollouts per prior) and 16 composite (3200 rollouts per prior) tasks. Right two panels: GR00T N1.5 on the same splits. Both architectures share the same qualitative ordering;

\mathcal{Z}

and

A_{\text{pre}}{+}\sigma_{A}\mathcal{Z}

tie at the top on both task types, while

A_{\text{pre}}

and Retrieval priors underperform.

Fig. 7: LogME feature quality as percentage difference from

\mathcal{Z}

at each fraction (Setting B sim , eight tasks). A value of five means LogME for that prior is 5% higher than

\mathcal{Z}

(dashed green line at zero) at the same fraction. Ribbons show

\pm 1

standard error across the eight tasks. We normalize because raw LogME drops sharply from 5% to 25% data due to a small-sample artifact of the LogME computation (see raw values in Fig. 13 ); normalizing to

\mathcal{Z}

removes this artifact since we only compare across priors at the same sample size. CLD letters (computed on raw values) confirm all priors share group “a” at every fraction.

Fig. 8: Action-space similarity after fine-tuning (Setting B sim , eight tasks). Violins show per-observation distances between predicted and ground-truth actions at each fraction, with CLD letters from Welch’s

t

-test. Top:

\ell_{2}

distance (

\downarrow

). All four priors cluster within a narrow range (5.7–6.4), with the Gaussian consistently

{\sim}0.5

higher than the non-Gaussian priors. Bottom: cosine similarity (

\uparrow

). All four priors largely overlap at every fraction (means within

{\sim}0.79

–

0.83

). Note: per-prior sample sizes are very large (

n{\approx}16000

), so CLD letters here flag any detectable mean difference and may not reflect practical separation.

Fig. 9: How much does fine-tuning change the model’s internal representations? (Setting B sim , eight tasks.) Violins show per-observation

\ell_{2}

distances with CLD letters from Welch’s

t

-test at each fraction. Top: action change (

A_{\text{post}}

vs.

A_{\text{pre}}

); all priors cluster within a narrow band (2.5–2.9). Bottom: encoder change (

E_{\text{post}}

vs.

E_{\text{pre}}

); the four priors show large, visually apparent separation between violin distributions at every fraction (unlike the action change above, where the four CLD groups arise from small mean differences within an otherwise largely overlapping band), with

A_{\text{pre}}

restructuring the encoder substantially (mean

\ell_{2}

roughly 9–12 across fractions) while

\mathcal{Z}

moves much less (

\ell_{2}\approx 5

Fig. 10: Representation change, cosine-similarity view (Setting B sim , eight tasks); companion to Fig. 9 . Higher values indicate less directional rotation from the pretrained representation. Top: action change; all priors largely overlap (

\cos\approx 0.92

–

0.95

). Bottom: encoder change;

\mathcal{Z}

retains the highest similarity (

\cos\approx 0.84

) while

A_{\text{pre}}

rotates the most (

\cos\approx 0.55

–

0.64

), with the mixed priors in between.

Fig. 11: Encoder learning rate ablation. Top: Setting

B_{\mathrm{sim}}

(eight tasks, 100% data, three representative priors). Reducing the encoder LR to

1/10

of the action head’s (lighter violins, “b”) degrades performance by 15–17% across all three priors, while the choice of prior has minimal effect within each LR group (both groups share a single CLD letter). Bottom: Setting B real (Gaussian prior, five tasks). Reducing the encoder LR to

1/10

drops both success rate and partial task progress (different CLD groups), a larger effect than any prior-related difference in this study.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}{=}0.95$ , $\beta_{2}{=}0.999$ , $\epsilon{=}10^{-8}$ )
Weight decay	$10^{-6}$
Peak learning rate	$2\times 10^{-5}$
LR schedule	Cosine decay with 50-step linear warmup
Trainable parameters	Full model (CLIP ViT-B/16 encoder + DiT action head), with encoder LR multiplier $1.0$ ( $0.1$ in the encoder-LR ablation)
Global batch size	304 (38 per GPU $\times$ 8 GPUs)

Appendix

TABLE III: Fine-tuning hyperparameters for the LBM 1.0 backbone (Settings A, B sim , B real ). Values in parentheses apply to the corresponding ablations. The fine-tuning step budget in Setting B sim follows a discrete lookup by data fraction.

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.95$ , $\epsilon{=}10^{-8}$ )
Weight decay	$10^{-10}$
Peak learning rate	$2.5\times 10^{-5}$
LR schedule	Cosine decay to $2.5\times 10^{-6}$ with 100-step linear warmup
Gradient clipping	$\lVert g\rVert_{2}\leq 1.0$
Trainable parameters	Full model (SigLIP vision encoder + Gemma 2B VLM + Gemma 300M action expert)

Appendix

TABLE IV: Fine-tuning hyperparameters for the

\pi_{0.5}

backbone (Setting C).

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}{=}0.95$ , $\beta_{2}{=}0.999$ , $\epsilon{=}10^{-8}$ )
Weight decay	$10^{-5}$
Peak learning rate	$3\times 10^{-5}$
LR schedule	Cosine decay with 200-step linear warmup (5% of 4000 steps)
Gradient clipping	$\lVert g\rVert_{2}\leq 1.0$
Trainable parameters	Action-head projector + DiT (EAGLE backbone frozen)

Appendix

TABLE V: Fine-tuning hyperparameters for the GR00T N1.5 backbone (Setting C).

Short name	Full name	Scenario	# demos
Easier group (three tasks, used in Setting B sim only)
PutMug	PutMugInCenterOfTable	Breakfast (B)	294
PlaceAvocado	PlaceAvocadoFromBowlIntoBin	Shelf (S)	196
PutSpatula	PutSpatulaOnPlateFromUtensilCrock	DryingRack (D)	196
Harder group (five tasks, used in Settings A and B)
DumpVeg	DumpVegetablesFromSmallToLargeContainer	Kitchen (K)	392

Appendix

TABLE VI: Simulation tasks used in Settings A and B sim (LBM Eval benchmark). Demonstration counts follow LBM 1.0 [ 3 ] . Fractional data experiments in Setting B sim randomly subsample this full pool per task without replacement (e.g., 50% of DumpVeg uses 196 randomly selected demonstrations out of 392).

Task name	One-line description	# demos
FoodBank	Pack the food box with at least one of each type of food item on the table	354
GatheringIngredientsForCookies	Gather the ingredients for making cookies	199
BusBin	Place plates and silverware into the separate bus bin, and place compost and recycling into the correct bins	202
ClearKitchenCounter [ 3 ]	Place tools into a tray, clean a cutting board with a sponge, and sweep the remaining waste into a trash bin	305
BikeRotorInstall [ 3 ]	Seat a rotor on a bicycle wheel, place a lockring, and tighten it at least one revolution with a tool	533

Appendix

TABLE VII: Hardware tasks used in Setting B real (bimanual Franka FR3).

Fig. 12: Setting B sim : easier (top, three tasks) vs. harder (bottom, five tasks) breakdown.

Fig. 13: Raw LogME scores across data fractions (Setting B sim , eight tasks). Each violin shows the distribution of scores across tasks. The sharp drop from 5% to 25% is a small-sample artifact of the LogME computation, not necessarily a real change in feature quality. All priors share CLD group “a” at every fraction.

Fig. 14: Setting B FP : all methods, including Gaussian, use the flow-matching objective in both pretraining and fine-tuning (five tasks).

Fig. 15: LogME feature quality in Setting B FP (five tasks, 100% data). Each violin shows the distribution of scores across tasks. All four priors are indistinguishable.

Fig. 16: Action-space similarity in Setting B FP (five tasks, 100% data). Top:

\|A_{\text{post}}-A_{\text{target}}\|_{2}

. Bottom:

\cos(A_{\text{post}},A_{\text{target}})

. Violins show per-observation distributions with CLD letters from Welch’s

t

-test.

Fig. 17: Representation change in Setting B FP (

\ell_{2}

view, five tasks, 100% data). Top: action change

\|A_{\text{post}}-A_{\text{pre}}\|_{2}

; all priors cluster within a narrow band. Bottom: encoder change

\|E_{\text{post}}-E_{\text{pre}}\|_{2}

;

A_{\text{pre}}

restructures the encoder most and

\mathcal{Z}

the least, with the mixed priors in between.

Fig. 18: Representation change in Setting B FP , cosine-similarity view (companion to Fig. 17 ). Higher values indicate less directional rotation from the pretrained representation.

Fig. 19: Noise ablation (Setting B FP , five tasks). Performance is robust to noise scale for

E_{\text{pre}}

variants;

A_{\text{pre}}

variants degrade with higher noise.

Fig. 20: Setting B real prior-selection pilot (three tasks, 25 rollouts per method). Left: success rate. Right: partial task progress. All three priors are statistically tied on both metrics.

Fig. 21: Zero-shot prior-to-target distance for

\pi_{0.5}

on RoboCasa. Violins show the distribution of per-step prior-to-target

\ell_{2}

distances pooled across tasks (

n{=}18

atomic,

n{=}16

composite); dots mark the mean and CLD letters are from Welch’s

t

-test.

A_{\text{pre}}

remains substantially closer to the target than the Gaussian on both splits.

Fig. 22: Setting B sim per-task success rates at 100% data (LBM 1.0, eight tasks, 200 rollouts per method–task). CLD letters are assigned independently within each task. The vertical dotted line separates the 3 easier tasks (left) from the 5 harder Kitchen-K tasks (right).

Fig. 23: Setting B real per-task performance (LBM 1.0 on hardware, five tasks, 50 A/B-test rollouts per method–task). Top: success rate violins with CLD letters computed within each task. Bottom: partial-task-progress violins with Welch CLD.

Panel of Figure 31 — Fig. 24: Zero-shot rollouts of all seven priors on DumpVeg (columns: priors as in Table I ; rows: progression over a 20 s rollout). Colored dots show both arms’ predicted end-effector positions over the action chunk horizon (color gradient indicates temporal ordering). Frames without visible dots occur when predicted positions project outside the camera’s field of view (e.g., $A_{\text{pre}}$ at ${\sim}9$ – $19$ s, where arms moved off-screen after task progress) or cluster into a single indistinguishable point due to near-stationary predictions (e.g., Cocos at ${\sim}5$ s). The Gaussian baseline ( $\mathcal{Z}$ ) is erratic throughout, as expected from pure noise. Several learned priors (e.g., $A_{\text{pre}}$ , Cocos, Retrieval) occasionally produce rollouts that make legitimate task progress; the rest sit between these extremes.

Explore similar work

May 13, 2026

WarmPrior: Straightening Flow-Matching Policies with Temporal Priors

Generative policies based on diffusion and flow matching have become a dominant paradigm for visuomotor robotic control. We show that replacing the standard Gaussian source distribution with WarmPrior, a simple temporally grounded prior constructed from readily available recent action history, consistently improves success rates on robotic manipulation tasks. We trace this gain to markedly straighter probability paths, echoing the effect of optimal-transport couplings in Rectified Flow. Beyond standard behavior cloning, WarmPrior also reshapes the exploration distribution in prior-space reinforcement learning, improving both sample efficiency and final performance. Collectively, these results identify the source distribution as an important and underexplored design axis in generative robot control.

Sinjae Kang, Chanyoung Kim, Kaixin Wang +2

KAIST · Microsoft Research

Flow Matching Visuomotor Policy Learning Diffusion Policy Robotic Manipulation RL for Generative Models

Ask PDF

Jun 22, 2026

Flowing With Purpose: Latent Action Guided Flow Matching Policies For Robotic Manipulation

Flow matching has recently become a new standard for behavior cloning in robotic manipulation. However, state-of-the-art flow matching policies suffer from a systematic structural mismatch: they rely on a globally fixed isotropic source distribution despite the strongly fragmented and heteroscedastic structure of robotic action spaces. This agnostic initialization forces the model to learn highly entangled vector fields, bottlenecking training efficiency and limiting overall policy performance. To address this limitation, we introduce Latent Action Guided Flow Matching (LAFM), a novel framework that replaces the monolithic Gaussian with an adaptive library of learned prior distributions. By grounding these distributions using a latent action model, LAFM maps current observations to discrete motion primitives, selecting a specialized base distribution that provides an informed, structurally aligned initialization for the denoising process. This dynamic adaptivity naturally accommodates heteroscedasticity in human demonstrations and makes transport trajectories shorter and less entangled. Empirically, LAFM substantially outperforms standard flow matching formulations, increasing task success rates by 23.4% in real-world robotic deployments and by 10.4% on the LIBERO-90 benchmark. Furthermore, we demonstrate that LAFM achieves state-of-the-art results, surpassing massively pre-trained vision-language-action models while utilizing significantly smaller architectures.

Bruno Machado, Alexandre Chapin, Emmanuel Dellandrea +1

École Centrale de Lyon · LIRIS, UMR5205

Flow Matching Robotic Manipulation Robot Imitation Learning

Ask PDF

Oct 11, 2026

$Δ$ Flow Matching for Robot Foundation Models

Flow matching is a common generative recipe for robot foundation models. Vision-language-action models (VLAs) generate action chunks, while world-action models (WAMs) additionally generate future visual states. Both start generation from a fixed Gaussian prior, despite temporal continuity between past and future states and actions. This requires costly training and iterative sampling, especially for WAMs with an additional visual branch. We introduce

Δ

-flow, which constructs the source from recent history with mild noise rather than from a fixed Gaussian prior. This moves the source closer to the target, reducing transport to a small residual

Δ

that can be learned efficiently and enables single-step inference. We integrate

Δ

-flow into the VLA policy

π_{0.5}

and the WAM policies ImageWAM and Fast-WAM. On LIBERO, ImageWAM reaches 90% success in one epoch rather than six, and Fast-WAM reaches 95% in two epochs rather than eight. With LoRA fine-tuning,

π_{0.5}

exceeds its baseline's five-epoch performance after one epoch. Single-step inference gives

1.54\times

2.49\times

, and

4.45\times

speedups on

π_{0.5}

, ImageWAM, and Fast-WAM, respectively. It preserves action smoothness, achieving a

4.5\times

reduction in action jerk compared with one-step Gaussian sampling.

Δ

-flow reaches state-of-the-art generalization performance on LIBERO-Plus. Action-history sources can be applied to pretrained WAM policies without retraining. Replacing the Gaussian action source at inference largely preserves task success with a single sampling step and can slightly improve it. We further study how action representation and visual information in action learning affect history-to-future prediction in

Δ

-flow. These results show that

Δ

-flow is simple yet effective, improving the efficiency and generalization of robot foundation models.

Kuangji Zuo, Tuo An, Xinying Guo +9

MARS Lab, Nanyang Technological University

Robot Foundation Models Flow-Based Generative Modeling Efficient Inference for World Action Models Efficient VLA Model Inference

Ask PDF

Abstract

Explore similar work

WarmPrior: Straightening Flow-Matching Policies with Temporal Priors

Flowing With Purpose: Latent Action Guided Flow Matching Policies For Robotic Manipulation

ΔΔΔ Flow Matching for Robot Foundation Models

$Δ$ Flow Matching for Robot Foundation Models