One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?

Ziqian Huang; Yuzhuo Chen; Shimin Zhang; Junyang He; Xinyue Annie Yang; Animesh Garg

cs.ROOct 20, 2026

One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?

Authors: Ziqian Huang, Yuzhuo Chen, Shimin Zhang, Junyang He, Xinyue Annie Yang, Animesh Garg

Organizations: College of Computing Georgia Institute of Technology

Abstract

Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.

Figures & tables

	Responsiveness			Monotonicity	Selectivity		Task preservation	Failure behavior
Method	Hypervolume $\uparrow$	Exp. Utility $\uparrow$	Self-PRR $\uparrow$	Rank acc. $\uparrow$	Selectivity $\uparrow$	Cross-PRR $\downarrow$	$J_{\mathrm{task}}\uparrow$	Fall rate $\downarrow$
Reward-level	0.316 $\pm$ .016	0.763 $\pm$ .007	0.454 $\pm$ .004	0.847 $\pm$ .018	1.61 $\pm$ .11	0.285 $\pm$ .019	0.761 $\pm$ .014	7.4 $\pm$ .1%
Advantage-level	0.109 $\pm$ .011	0.568 $\pm$ .014	0.232 $\pm$ .045	0.811 $\pm$ .001	2.16 $\pm$ .15	0.108 $\pm$ .013	0.409 $\pm$ .019	8.0 $\pm$ .3%
Loss-level	0.103 $\pm$ .007	0.572 $\pm$ .010	0.225 $\pm$ .004	0.792 $\pm$ .033	2.04 $\pm$ .06	0.112 $\pm$ .006	0.414 $\pm$ .017	7.9 $\pm$ .1%
PCGrad	0.104 $\pm$ .016	0.558 $\pm$ .029	0.233 $\pm$ .004	0.800 $\pm$ .047	2.07 $\pm$ .17	0.114 $\pm$ .008	0.393 $\pm$ .027	8.4 $\pm$ .1%

Table 1: Main comparison on ProtoMotions (G1 humanoid), mean

\pm

std over 3 training seeds at 1.2B environment steps. Column groups correspond to the five evaluation axes in Section 1 : responsiveness, monotonicity, selectivity, task preservation, and failure behavior. Bold marks the best value per column;

\uparrow

\downarrow

indicate the preferred direction.

Figure 1: Per-dimension preference sweep. Each panel shows the normalized reward achieved for one style dimension as its preference weight increases, mean

\pm

std shading over 3 training seeds.

	Responsiveness			Monotonicity	Selectivity		Task preservation	Failure behavior
Method	Hypervolume $\uparrow$	Exp. Utility $\uparrow$	Self-PRR $\uparrow$	Rank acc. $\uparrow$	Selectivity $\uparrow$	Cross-PRR $\downarrow$	$J_{\mathrm{task}}\uparrow$	Fall rate $\downarrow$
Reward-level	0.238 $\pm$ .029	0.769 $\pm$ .009	0.510 $\pm$ .021	0.924 $\pm$ .003	1.272 $\pm$ .055	0.402 $\pm$ .022	0.870 $\pm$ .004	0.34 $\pm$ .08%
Advantage-level	0.298 $\pm$ .015	0.776 $\pm$ .007	0.485 $\pm$ .022	0.937 $\pm$ .010	1.387 $\pm$ .101	0.350 $\pm$ .010	0.865 $\pm$ .006	0.25 $\pm$ .06%
Loss-level	0.290 $\pm$ .012	0.771 $\pm$ .001	0.500 $\pm$ .030	0.937 $\pm$ .022	1.422 $\pm$ .082	0.352 $\pm$ .006	0.858 $\pm$ .005	0.32 $\pm$ .09%

Table 2: Fixed-direction preference-conditioned control in MimicKit, mean

\pm

std over 3 training seeds at 1.2B environment steps. Column groups correspond to the five evaluation axes in Section 1 . Bold marks the best displayed mean, including ties;

\uparrow

\downarrow

indicate the preferred direction.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Description	Bounds
energy	negative raw power (W)	$(-800.0,\ 0.0)$
arm_swing	shoulder-pitch angular velocity (rad/s)	$(0.0,\ 3.0)$
foot_clearance	swing-phase foot height (m)	$(0.0,\ 0.20)$
step_width	lateral foot separation (m)	$(0.0,\ 0.40)$
smoothness	negative $\\|\Delta\text{action}\\|_{2}$	$(-2.5,\ 0.0)$

Appendix

Table 3: Style reward dimensions and normalization bounds.

Horizon $H$	300
Axis-sweep points per dimension $M$	15
Axis-sweep range $[\alpha_{\min},\alpha_{\max}]$	$[0.05,0.95]$
Full-simplex samples $N$	50
Dirichlet concentration $\alpha$	0.5
Hypervolume Monte Carlo samples	$2\times 10^{5}$
Eval repeats	3 (seeds $1000/1001/1002$ )

Appendix

Table 4: Evaluation hyperparameters, shared across all four methods and three training seeds.

Method	Dimension	Self-PRR $\uparrow$	Cross-PRR $\downarrow$	Rank acc. $\uparrow$
Reward-level	energy	0.295 $\pm$ .051	0.441 $\pm$ .042	0.791 $\pm$ .059
	arm_swing	0.703 $\pm$ .080	0.152 $\pm$ .044	0.895 $\pm$ .054
	foot_clearance	0.790 $\pm$ .003	0.290 $\pm$ .035	0.876 $\pm$ .026
	step_width	0.315 $\pm$ .043	0.167 $\pm$ .040	0.778 $\pm$ .003
	smoothness	0.165 $\pm$ .010	0.375 $\pm$ .059	0.893 $\pm$ .028
Loss-level	energy	0.048 $\pm$ .010	0.067 $\pm$ .018	0.678 $\pm$ .035

Appendix

Table 5: Per-dimension Self-PRR, Cross-PRR, and Rank Accuracy, mean

\pm

std over 3 training seeds.

	Hypervolume		Expected Utility
Method	Fixed	Alive	Fixed	Alive
Reward-level	0.316 $\pm$ .016	0.435 $\pm$ .021	0.763 $\pm$ .007	0.818 $\pm$ .008
Advantage-level	0.109 $\pm$ .011	0.158 $\pm$ .012	0.568 $\pm$ .014	0.610 $\pm$ .015
PCGrad	0.104 $\pm$ .016	0.154 $\pm$ .024	0.558 $\pm$ .029	0.601 $\pm$ .031
Loss-level	0.103 $\pm$ .007	0.149 $\pm$ .010	0.572 $\pm$ .010	0.615 $\pm$ .011

Appendix

Table 6: Hypervolume and Expected Utility under the Fixed and Alive conventions, mean

\pm

std over 3 training seeds.

Setting	Value
Parallel training environments	4096
PPO rollout length	32 steps
Actor / critic / discriminator hidden widths	1024 / 1024 / 1024
Gaussian action standard deviation	0.05
Discount $\gamma$ / GAE $\lambda$	$0.99$ / $0.95$
PPO clipping coefficient	$0.2$

Appendix

Table 7: MimicKit optimization hyperparameters, shared across all methods and seeds.

	Responsiveness			Monotonicity	Selectivity		Task preservation	Failure behavior
Method	Hypervolume $\uparrow$	Exp. Utility $\uparrow$	Self-PRR $\uparrow$	Rank acc. $\uparrow$	Selectivity $\uparrow$	Cross-PRR $\downarrow$	$J_{\mathrm{task}}\uparrow$	Fall rate $\downarrow$
Reward-level	0.178 $\pm$ .005	0.728 $\pm$ .006	0.479 $\pm$ .020	0.966 $\pm$ .007	1.315 $\pm$ .034	0.365 $\pm$ .023	0.813 $\pm$ .010	0.26 $\pm$ .04%
Advantage-level	0.199 $\pm$ .004	0.732 $\pm$ .003	0.480 $\pm$ .004	0.971 $\pm$ .005	1.503 $\pm$ .015	0.319 $\pm$ .002	0.809 $\pm$ .003	0.22 $\pm$ .03%
Loss-level	0.201 $\pm$ .005	0.731 $\pm$ .003	0.471 $\pm$ .013	0.975 $\pm$ .008	1.498 $\pm$ .067	0.315 $\pm$ .007	0.807 $\pm$ .010	0.17 $\pm$ .03%

Appendix

Table 8: Reactive-steering preference-conditioned control in MimicKit, mean

\pm

std over 3 training seeds at 1.2B environment steps. Bold marks the best mean;

\uparrow

\downarrow

indicate the preferred direction.

Method	Paired EU $\uparrow$	Mean Self-PRR $\uparrow$	Mean Cross-PRR $\downarrow$
Reward-level	0.608 $\pm$ .014	0.0028 $\pm$ .0027	0.0216 $\pm$ .0012
Advantage-level	0.634 $\pm$ .015	0.0263 $\pm$ .0017	0.0246 $\pm$ .0015
Loss-level	0.586 $\pm$ .020	0.0015 $\pm$ .0003	0.0112 $\pm$ .0005

Appendix

Table 9: MetaDrive environment ablation, mean

\pm

std over three evaluation seeds. Bold marks the best value;

\uparrow

\downarrow

indicate the preferred direction.

Preference-Based RL

Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.

See all 127 papers

Freeform Preference Learning for Robotic Manipulation
Jun 30, 2026Marcel Torne, Anubha Mahajan, Abhijnya Bhat +1Long-Horizon Robotic Manipulation Robot Skill Learning
Stanford University
Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
Jul 14, 2026Taehyung Kim, Gwangmo Lee, Minjun Chang +2Clustering Preference Alignment
Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea
Controllability in preference-conditioned multi-objective reinforcement learning
May 11, 2026Pau de las Heras Molins, Beyazit Yalcinkaya, Lasse Peters +2Multi-Objective Reinforcement Learning Preference-Based RL
LTCI, Télécom Paris, Institut Polytechnique de Paris · University of California, Berkeley · TU Delft +1

Jun 30, 2026J/K move · Enter open · S save

Freeform Preference Learning for Robotic Manipulation

Marcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn

Stanford University

Long-Horizon Robotic Manipulation Robot Skill Learning Robotic Manipulation Policy Learning Preference-Based RL

Open paper AskPDF arXiv

Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/

Abstract

Preference-Based RL

Explore similar work

Freeform Preference Learning for Robotic Manipulation

Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences

Controllability in preference-conditioned multi-objective reinforcement learning