Organizations: University of Illinois Urbana-Champaign · University of California, Berkeley · Harvard University · Stanford University · University of Illinois Chicago
Vision-language-action (VLA) policies can learn to follow execution preferences, such as where to grasp an object or how to move around an obstacle, from demonstrations annotated with preference labels. Providing these annotations for every new task, however, is costly. We study whether learned preference control can transfer to new tasks whose demonstrations contain the desired behaviors but lack preference annotations. We find that a policy trained with preference annotations can follow preferences on a new task before adaptation but achieves limited task success. Fine-tuning on target demonstrations without these annotations improves task success while weakening preference control. Motivated by this finding, we introduce Self-Labeled Preference Transfer (SPT). SPT jointly trains a shared vision-language backbone for preference-conditioned action prediction and preference recognition on annotated source demonstrations. It then freezes the recognizer to preserve its cross-task preference recognition capability and uses it to self-label target demonstrations for policy fine-tuning. Across five types of execution preferences in simulation, the recognizer achieves 96.4% accuracy, and SPT reaches 84.8% preference following and 89.2% task success, compared with 59.4% and 88.0% for naive finetuning. We further extend to multiple preferences, achieving 93.8% preference following compared with 42.5% of baselines. In real-world experiments, SPT increases preference following from 52.5% to 92.5% relative to naive finetuning across two task transfers. These results show that SPT can support preference adaptation across tasks and objects while maintaining success, without additional manual preference annotations.
Figures & tables
Fig. 1: Motivation and overview. A task can be completed in several valid ways. SPT learns to follow and recognize execution preferences on labeled source tasks. It then labels target demonstrations so that the adapted policy can respond to the same preference commands. Target demonstrations are required; their preference labels are not. The examples illustrate the intended interface, not additional evaluated tasks.
Fig. 2: Self-Labeled Preference Transfer. Stage A trains a shared vision-language backbone for preference-conditioned action prediction and preference recognition. The recognition branch answers a preference question from a short demonstration clip and proprioceptive observations. Stage B has two steps: a frozen copy of the source recognizer labels each target demonstration, and a trainable policy copy adapts to the resulting preference-conditioned prompts. At deployment the user gives a preference command and only the action policy runs.
Fig. 3: Illustration of preferences in simulation. The base instruction is “Pick up A and place it on B” with different objects and receptacles in the source and target sets. Columns show the five preference axes and rows show their two values. The single-axis experiments evaluate each preference type separately; Section V-E combines release height and placement preferences.
Fig. 4: Real-robot setup. Left: pick-and-place (source) and pushing (target) with two obstacle-clearance preferences. Right: air hockey (source) and billiards (target) with direct and bank-shot strategies. Dashed paths show the requested behaviors.
Axis
Demonstration control and rollout statistic
Contact height
Grasp at 20–30% or 70–80% of object height. EE vertical offset from the object origin at grasp; threshold 6.83 cm.
Release height
Low or high release. Object vertical displacement from grasp to immediately before release; threshold 12.45 cm.
Clearance
Direct path or 20–25 cm detour. Maximum XY deviation from the 25%–85% trajectory chord; threshold 15.01 cm.
Orientation
0∘ or 90∘ grasp. Left-wrist orientation feature at grasp; threshold 41.22∘ .
Placement
Center or corner target. EE XY distance to the receptacle’s initial actor position at release; threshold 5.36 cm.
TABLE I: Simulation preference metrics and fixed source-calibrated thresholds.
Preference Recognition Acc.
Task Success
Preference Following
Preference axis
Zero-shot
SPT
Naive FT
SPT
Naive FT
SPT
Contact height
51
100
92
95
59
90
Release height
46
100
89
91
53
81
Clearance
53
92
85
82
57
75
Orientation
50
100
94
92
61
90
Placement
59
90
80
86
67
83
TABLE II: RoboTwin simulation results (%). Preference recognition accuracy is evaluated on target demonstrations; task success and preference following are evaluated on policy rollouts.
Task Success
Preference Following
Target task
Naive FT
SPT
Naive FT
SPT
Pushing
95
100
55
100
Billiards
85
90
50
85
TABLE III: Real-robot results (%), aggregated over both preference commands for each target task (20 trials per task).
Task Success
Preference Following
Method
Placement
Contact
Placement
Contact
Naive FT
80
92
67
59
Source-only
67
85
80
88
Same-init plain FT
69
95
52
50
random labels
66
92
50
55
Zero-shot VLM
70
93
60
50
TABLE IV: Ablations of the two-stage design (%) on placement location and grasp contact height. Bold marks the best result among methods without target preference annotations; ties are also bold. Supervised references are shown below the midrule.
Preference Following
Task Success
Joint Success
Method
Release
Place
Both
Source-only
78.8
71.9
62.5
33.8
29.4
Random labels
43.8
45.6
24.4
91.9
21.9
Same-init plain FT
70.6
61.3
42.5
90.6
40.0
SPT (Ours)
100.0
93.8
93.8
90.6
88.8
TABLE V: Two-preference target adaptation (%). Preference following is reported for each preference and for both on the same rollout.