PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions

Yiqi Tang; Diyuan Shi; Runze Li; Donglin Wang

cs.ROSep 29, 2026

PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions

Authors: Yiqi Tang, Diyuan Shi, Runze Li, Donglin Wang

Organizations: Westlake University

Abstract

Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation. We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.

Figures & tables

Fig. 1: Overview of PreferenceFlow. Human interventions provide action-chunk comparisons sharing a window-start state, while expert demonstrations provide anchors for zero-gradient regularization. Separate intervention and demonstration sub-batches train a state-conditioned preference model. Its action gradient guides the frozen $\pi_{0.5}$ flow sampler at inference. Only the preference-model parameters are updated during adaptation.

Fig. 2: State-conditioned preference-model architecture. Three camera images, the language instruction, and robot-state information form the frozen $\pi_{0.5}$ conditioning representation, used as keys and values. Candidate action tokens form the queries and cross-attend to the prefix tokens to produce the preference score $F_{\phi}(s,a)$ . Its action gradient provides the guidance signal for flow sampling. The schematic omits robot-state inputs and the per-step score heads followed by mean aggregation.

Fig. 3: Real-world precision insertion tasks. Each row shows one task and progresses from left to right, from the initial approach to insertion.

Method	Ethernet	USB	SATA	Charger	Mean $\uparrow$	$\Delta$ vs. base (pp) $\uparrow$
Frozen $\pi_{0.5}$	$66$	$70$	$68$	$72$	$69.0$	—
Intervention BC	$82$	$84$	$88$	$84$	$84.5$	$+15.5$
QGF	$74$	$74$	$80$	$76$	$76.0$	$+7.0$
FlowDAgger	$78$	$80$	$76$	$80$	$78.5$	$+9.5$
PreferenceFlow (ours)	90	92	92	88	90.5	+21.5

TABLE I: Task success rates (%) over 50 real-robot trials per task. Mean denotes the arithmetic mean across the four tasks.

\Delta

denotes the absolute improvement over the frozen policy in percentage points (pp). Bold indicates the best observed result in each column.

Training objective	Mean success (%) $\uparrow$
	Fixed guidance	Adjusted guidance
Full objective	90.5	90.5
Without gradient-cap loss	$15.0$	$81.0$
Without zero-gradient loss	$5.0$	$74.5$
Preference loss only	Unstable	$69.0$

TABLE II: Gradient-objective ablation: mean success (%) across four tasks. Fixed guidance uses the full model’s weight. For adjusted guidance, each ablated variant’s weight is progressively reduced until deployment is stable. Unstable indicates that stable deployment was not possible at the tested weight; no success rate is assigned. Bold indicates the best observed rates.

Architecture	Mean success (%) $\uparrow$
Cross-attention + temporal self-attention	$77.0$
Per-step cross-attention (ours)	90.5

TABLE III: Preference-model architecture ablation: mean success (%) across four tasks. Both architectures use state-conditioned cross-attention; the ablated variant additionally uses temporal self-attention among action tokens. Bold indicates the best observed rate.

Variant	Mean success (%) $\uparrow$
Preference reranking (5 samples)	92.0
PreferenceFlow	$90.5$
Randomized preference labels	$63.5$

TABLE IV: Score-based selection and negative controls: mean success (%) across four tasks. Bold indicates the best observed rate.

Explore similar work

Freeform Preference Learning for Robotic Manipulation
Jun 30, 2026Marcel Torne, Anubha Mahajan, Abhijnya Bhat +1Robot Policies Robotic Manipulation
Stanford University
FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space
Jul 9, 2026Michael Murray, Daphne Chen, Simran Bagaria +7Robot Policies Human-In-The-Loop
1Microsoft Research · University of Washington · 2Microsoft +1
Steering Generative Robot Policies with Lexicographic Preferences
Sep 14, 2026Yixuan Jia, Jonathan P. HowRobot Policies Robot Systems
Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA 02319 USA

Jun 30, 2026, cs.ROJ/K move · Enter open · S save

Freeform Preference Learning for Robotic Manipulation

Marcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn

Stanford University

Robot Policies Robotic Manipulation Preference Learning

Open paper AskPDF arXiv

Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/

Abstract

Explore similar work

Freeform Preference Learning for Robotic Manipulation

FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space

Steering Generative Robot Policies with Lexicographic Preferences