cs.ROSep 22, 2026
SaveThe Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models
Organizations: Toyota Research Institute · Woven by Toyota · Cornell University
Abstract
Modern robot imitation learning increasingly relies on generative policies based on diffusion or flow-matching models, which generate actions by transforming samples from a prior distribution. A key question is whether the choice of prior matters. Replacing the standard Gaussian with a closer-to-target, non-Gaussian prior has been shown to substantially improve performance when training from scratch. A natural next step is to ask whether these gains transfer to fine-tuning pretrained Large Behavior Models (LBMs) such as LBM 1.0, , and GR00T~N1.5, where one might expect even larger gains. Surprisingly, we find that this is not the case, except possibly at very low fine-tuning data fractions. Across over 100K simulation rollouts spanning all three aforementioned LBMs on 40+ tasks in two simulation platforms, and 1250 hardware rollouts on five bimanual manipulation tasks, non-Gaussian priors that are demonstrably closer to the target yield statistically indistinguishable or worse fine-tuning performance than a standard Gaussian prior. Diagnostic analyses suggest why: fine-tuned imitation learning policies converge to similar action predictions across priors, despite their fine-tuned encoder embeddings diverging substantially from the pretrained embeddings and each other. A learning-rate ablation further confirms that encoder training is the dominant factor in fine-tuning performance, substantially outweighing the effect of prior choice. We conclude with concrete directions for future research on when and why learned priors might still matter in fine-tuning. Project page: https://cxu-tri.github.io/non_gaussian_FT/
Figures & tables
Fig. 1: Prior definition and visualization. Bottom: schematic of an example flow-matching policy. The action head is conditioned on the encoder embedding and takes in diffusion step as well as the action prior . Only the prior varies across our experiments. Top: samples drawn directly from two examples of the prior on the simulated DumpVegetablesIntoContainer task from LBM Eval [ 3 ] , with overlaid colored dots showing both arms’ predicted end-effector positions over the action chunk horizon. Color gradient indicates temporal ordering within the chunk, with lighter dots representing predictions further into the future. The Gaussian (left) produces incoherent, random motion, while (right) already makes partial task progress by picking up the avocado and moving it to the bin, reflecting a prior close to the target action. We hypothesize that the close-to-target prior would outperform after policy fine-tuning, but we empirically find that this is not the case.
Fig. 2: Action priors help when training from scratch but not when fine-tuning (Setting A, 5 LBM Eval kitchen tasks [ 3 ] , 1000 rollouts per prior; see Section VI-A for details). All fine-tuning prior variants start from the same pretrained LBM 1.0 checkpoint. denotes the action prior sampled from the pretrained policy. Left: distance between prior samples and ground-truth actions, in un-normalized action coordinates; is closer to the target data than the Gaussian. Middle: from-scratch training; significantly outperforms the Gaussian. Right: after fine-tuning, the Gaussian prior matches or, in this case, exceeds in success rate. The full comparison across all seven prior variants ( Fig. 3 ) confirms that most priors we consider yield statistically indistinguishable performance after fine-tuning, and, thus, provide no benefit to the policy.
| Prior | Description |
|---|---|
| Isotropic Gaussian | |
| Proposed priors (constructed from ) | |
| Actions generated by | |
| Actions from with added Gaussian noise (scale ) | |
| Encoder embeddings with added Gaussian noise (scale ) | |
| Baselines from the literature | |
TABLE I: The seven prior variants compared in this work. is the pretrained LBM (LBM 1.0, , or GR00T N1.5). Appendix A gives full sampling procedures and auxiliary-training details.
| Setting | Description | Model | Figures | # Rollouts |
| Main experiments | ||||
| A | Do closer-to-target priors improve fine-tuning performance? | LBM 1.0 [ 3 ] | 2 , 3 | 14000 |
| B sim | Do priors matter across varying data fractions? | LBM 1.0 [ 3 ] | 4 | 56000 |
| Analysis | How similar are fine-tuned models across priors? | 7 – 10 | ||
| Encoder LR | Does the encoder matter more than the prior? | 11 | 9600 | |
| B real | Do priors matter on robot hardware? | LBM 1.0 [ 3 ] | 5 | 1250 |
TABLE II: Index of all experimental settings, ablations, and analyses. Each row links to its corresponding figure(s). Reuses the rollouts of its corresponding main setting (no additional evaluations).
Fig. 3: Setting A: full seven-prior comparison; each violin corresponds to 1000 rollouts. Top: from scratch; statistically outperforms , which is second lowest in empirical success rate among all priors. Bottom: fine-tuned; all priors perform comparably to with six of seven priors sharing CLD group “a” or “b”.
Fig. 4: Fine-tuning performance across data fractions (Setting B sim , eight tasks). CLD letters are computed within each data fraction independently. All priors are statistically indistinguishable or worse than across fractions. The one exception is at 5% data, which significantly outperforms .
Fig. 5: Setting B real : hardware A/B testing (five bimanual FR3 tasks, 50 rollouts per prior per task, 1250 total). Top: Success rate violins; four of five priors share CLD group “a” and BRIDGER (group “b”) underperforms. Bottom: Partial-task-progress violins; the same grouping holds.
Fig. 6: Setting C: LBM fine-tuning on RoboCasa, with shared y-axis (Success Rate, 0–1) across all four panels. Left two panels: on 18 atomic (3600 rollouts per prior) and 16 composite (3200 rollouts per prior) tasks. Right two panels: GR00T N1.5 on the same splits. Both architectures share the same qualitative ordering; and tie at the top on both task types, while and Retrieval priors underperform.
Fig. 7: LogME feature quality as percentage difference from at each fraction (Setting B sim , eight tasks). A value of five means LogME for that prior is 5% higher than (dashed green line at zero) at the same fraction. Ribbons show standard error across the eight tasks. We normalize because raw LogME drops sharply from 5% to 25% data due to a small-sample artifact of the LogME computation (see raw values in Fig. 13 ); normalizing to removes this artifact since we only compare across priors at the same sample size. CLD letters (computed on raw values) confirm all priors share group “a” at every fraction.
Fig. 8: Action-space similarity after fine-tuning (Setting B sim , eight tasks). Violins show per-observation distances between predicted and ground-truth actions at each fraction, with CLD letters from Welch’s -test. Top: distance ( ). All four priors cluster within a narrow range (5.7–6.4), with the Gaussian consistently higher than the non-Gaussian priors. Bottom: cosine similarity ( ). All four priors largely overlap at every fraction (means within – ). Note: per-prior sample sizes are very large ( ), so CLD letters here flag any detectable mean difference and may not reflect practical separation.
Fig. 9: How much does fine-tuning change the model’s internal representations? (Setting B sim , eight tasks.) Violins show per-observation distances with CLD letters from Welch’s -test at each fraction. Top: action change ( vs. ); all priors cluster within a narrow band (2.5–2.9). Bottom: encoder change ( vs. ); the four priors show large, visually apparent separation between violin distributions at every fraction (unlike the action change above, where the four CLD groups arise from small mean differences within an otherwise largely overlapping band), with restructuring the encoder substantially (mean roughly 9–12 across fractions) while moves much less ( ).
Fig. 10: Representation change, cosine-similarity view (Setting B sim , eight tasks); companion to Fig. 9 . Higher values indicate less directional rotation from the pretrained representation. Top: action change; all priors largely overlap ( – ). Bottom: encoder change; retains the highest similarity ( ) while rotates the most ( – ), with the mixed priors in between.
Fig. 11: Encoder learning rate ablation. Top: Setting (eight tasks, 100% data, three representative priors). Reducing the encoder LR to of the action head’s (lighter violins, “b”) degrades performance by 15–17% across all three priors, while the choice of prior has minimal effect within each LR group (both groups share a single CLD letter). Bottom: Setting B real (Gaussian prior, five tasks). Reducing the encoder LR to drops both success rate and partial task progress (different CLD groups), a larger effect than any prior-related difference in this study.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW ( , , ) |
| Weight decay | |
| Peak learning rate | |
| LR schedule | Cosine decay with 50-step linear warmup |
| Trainable parameters | Full model (CLIP ViT-B/16 encoder + DiT action head), with encoder LR multiplier ( in the encoder-LR ablation) |
| Global batch size | 304 (38 per GPU 8 GPUs) |
TABLE III: Fine-tuning hyperparameters for the LBM 1.0 backbone (Settings A, B sim , B real ). Values in parentheses apply to the corresponding ablations. The fine-tuning step budget in Setting B sim follows a discrete lookup by data fraction.
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW ( , , ) |
| Weight decay | |
| Peak learning rate | |
| LR schedule | Cosine decay to with 100-step linear warmup |
| Gradient clipping | |
| Trainable parameters | Full model (SigLIP vision encoder + Gemma 2B VLM + Gemma 300M action expert) |
TABLE IV: Fine-tuning hyperparameters for the backbone (Setting C).
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW ( , , ) |
| Weight decay | |
| Peak learning rate | |
| LR schedule | Cosine decay with 200-step linear warmup (5% of 4000 steps) |
| Gradient clipping | |
| Trainable parameters | Action-head projector + DiT (EAGLE backbone frozen) |
TABLE V: Fine-tuning hyperparameters for the GR00T N1.5 backbone (Setting C).
| Short name | Full name | Scenario | # demos |
| Easier group (three tasks, used in Setting B sim only) | |||
| PutMug | PutMugInCenterOfTable | Breakfast (B) | 294 |
| PlaceAvocado | PlaceAvocadoFromBowlIntoBin | Shelf (S) | 196 |
| PutSpatula | PutSpatulaOnPlateFromUtensilCrock | DryingRack (D) | 196 |
| Harder group (five tasks, used in Settings A and B) | |||
| DumpVeg | DumpVegetablesFromSmallToLargeContainer | Kitchen (K) | 392 |
TABLE VI: Simulation tasks used in Settings A and B sim (LBM Eval benchmark). Demonstration counts follow LBM 1.0 [ 3 ] . Fractional data experiments in Setting B sim randomly subsample this full pool per task without replacement (e.g., 50% of DumpVeg uses 196 randomly selected demonstrations out of 392).
| Task name | One-line description | # demos |
|---|---|---|
| FoodBank | Pack the food box with at least one of each type of food item on the table | 354 |
| GatheringIngredientsForCookies | Gather the ingredients for making cookies | 199 |
| BusBin | Place plates and silverware into the separate bus bin, and place compost and recycling into the correct bins | 202 |
| ClearKitchenCounter [ 3 ] | Place tools into a tray, clean a cutting board with a sponge, and sweep the remaining waste into a trash bin | 305 |
| BikeRotorInstall [ 3 ] | Seat a rotor on a bicycle wheel, place a lockring, and tighten it at least one revolution with a tool | 533 |
TABLE VII: Hardware tasks used in Setting B real (bimanual Franka FR3).
Fig. 12: Setting B sim : easier (top, three tasks) vs. harder (bottom, five tasks) breakdown.
Fig. 13: Raw LogME scores across data fractions (Setting B sim , eight tasks). Each violin shows the distribution of scores across tasks. The sharp drop from 5% to 25% is a small-sample artifact of the LogME computation, not necessarily a real change in feature quality. All priors share CLD group “a” at every fraction.
Fig. 14: Setting B FP : all methods, including Gaussian, use the flow-matching objective in both pretraining and fine-tuning (five tasks).
Fig. 15: LogME feature quality in Setting B FP (five tasks, 100% data). Each violin shows the distribution of scores across tasks. All four priors are indistinguishable.
Fig. 16: Action-space similarity in Setting B FP (five tasks, 100% data). Top: . Bottom: . Violins show per-observation distributions with CLD letters from Welch’s -test.
Fig. 17: Representation change in Setting B FP ( view, five tasks, 100% data). Top: action change ; all priors cluster within a narrow band. Bottom: encoder change ; restructures the encoder most and the least, with the mixed priors in between.
Fig. 18: Representation change in Setting B FP , cosine-similarity view (companion to Fig. 17 ). Higher values indicate less directional rotation from the pretrained representation.
Fig. 19: Noise ablation (Setting B FP , five tasks). Performance is robust to noise scale for variants; variants degrade with higher noise.
Fig. 20: Setting B real prior-selection pilot (three tasks, 25 rollouts per method). Left: success rate. Right: partial task progress. All three priors are statistically tied on both metrics.
Fig. 21: Zero-shot prior-to-target distance for on RoboCasa. Violins show the distribution of per-step prior-to-target distances pooled across tasks ( atomic, composite); dots mark the mean and CLD letters are from Welch’s -test. remains substantially closer to the target than the Gaussian on both splits.
Fig. 22: Setting B sim per-task success rates at 100% data (LBM 1.0, eight tasks, 200 rollouts per method–task). CLD letters are assigned independently within each task. The vertical dotted line separates the 3 easier tasks (left) from the 5 harder Kitchen-K tasks (right).
Fig. 23: Setting B real per-task performance (LBM 1.0 on hardware, five tasks, 50 A/B-test rollouts per method–task). Top: success rate violins with CLD letters computed within each task. Bottom: partial-task-progress violins with Welch CLD.
Fig. 24: Zero-shot rollouts of all seven priors on DumpVeg (columns: priors as in Table I ; rows: progression over a 20 s rollout). Colored dots show both arms’ predicted end-effector positions over the action chunk horizon (color gradient indicates temporal ordering). Frames without visible dots occur when predicted positions project outside the camera’s field of view (e.g., at – s, where arms moved off-screen after task progress) or cluster into a single indistinguishable point due to near-stationary predictions (e.g., Cocos at s). The Gaussian baseline ( ) is erratic throughout, as expected from pure noise. Several learned priors (e.g., , Cocos, Retrieval) occasionally produce rollouts that make legitimate task progress; the rest sit between these extremes.