Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG

Nathaniel Dennler; Zhonghao Shi; Yiran Tao; Andreea Bobu; Stefanos Nikolaidis; Maja Matarić

cs.ROMar 9, 2026

Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG

Authors: Nathaniel Dennler, Zhonghao Shi, Yiran Tao, Andreea Bobu, Stefanos Nikolaidis, Maja Matarić

Organizations: CSAIL/AeroAstro, Massachusetts Institute of Technology. · Department of Computer Science, University of Southern California.

Abstract

Robots that interact with humans must adapt to individual users' preferences to operate effectively in human-centered environments. An intuitive and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, e.g., trajectories, gestures, or voices. Existing techniques primarily focus on generating queries that optimize preference learning outcomes, such as sample efficiency or final preference estimation accuracy. However, the focus on outcome overlooks key user expectations in the process of providing these rankings, which can negatively impact users' adoption of robotic systems. This work proposes the Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) algorithm. CMA-ES-IG explicitly incorporates user experience considerations into the preference learning process by suggesting perceptually distinct and informative trajectories for users to rank. We demonstrate these benefits through both simulated studies and real-robot experiments. CMA-ES-IG, compared to state-of-the-art alternatives, (1) scales more effectively to higher-dimensional preference spaces, (2) maintains computational tractability for high-dimensional problems, (3) is robust to noisy or inconsistent user feedback, and (4) is preferred by non-expert users in identifying their preferred robot behaviors. This project's code is available at github.com/interaction-lab/CMA-ES-IG

Figures & tables

Figure 1: Query Generation Techniques. The spheres on the top row represent a 3-dimensional trajectory feature space, with a blue arrow indicating the user’s true preference and the red arrow indicating the current estimated preference. The red circle denotes the equator of the sphere defined by the red preference estimate. The red cloud of points denote the sampling space for CMA-ES. Optimizing for information gain results in trajectories that spaced evenly across the red circle which are easy to distinguish for the user, but do not achieve high rewards (a). Using Covariance matrix adaptation evolution strategies (CMA-ES) results in trajectories with higher rewards, but they are not easy for the user to easily distinguish (b). CMA-ES with information gain (CMA-ES-IG) generates trajectories that are both easy to distinguish for the user and achieve high reward.

Algorithm	AUC Alignment ( $\uparrow$ )				AUC Regret ( $\downarrow$ )
Algorithm	$d=4$	$d=8$	$d=16$	$d=32$	$d=4$	$d=8$	$d=16$	$d=32$
CMA-ES-IG	$.882\pm{\scriptstyle.01}$	$.810\pm{\scriptstyle.01}$	$\mathbf{.718}\pm{\scriptstyle.01}$	$\bm{.517}\pm{\scriptstyle.01}$	$.265\pm{\scriptstyle.05}$	$.492\pm{\scriptstyle.04}$	$\bm{.759}\pm{\scriptstyle.08}$	$\mathbf{1.453}\pm{\scriptstyle.11}$
CMA-ES	$.908\pm{\scriptstyle.01}$	$.847\pm{\scriptstyle.01}$	$.692\pm{\scriptstyle.01}$	$.457\pm{\scriptstyle.01}$	$.157\pm{\scriptstyle.03}$	$.413\pm{\scriptstyle.04}$	$.995\pm{\scriptstyle.07}$	$1.876\pm{\scriptstyle.08}$
Infogain	$\bm{.928}\pm{\scriptstyle.00}$	$\bm{.849}\pm{\scriptstyle.01}$	$.606\pm{\scriptstyle.01}$	$.374\pm{\scriptstyle.02}$	$\bm{.116}\pm{\scriptstyle.01}$	$\bm{.332}\pm{\scriptstyle.04}$	$1.243\pm{\scriptstyle.10}$	$2.115\pm{\scriptstyle.11}$

Table 1: Marginal means for accuracy metrics in the parameter estimation task. We report the mean (

\pm

standard error) of the area under the curve (AUC) of alignment and regret. These metrics compare the estimated user preference to the ground-truth simulated user preferences, with arrows denoting better values. Bolded values denote the highest-performing method for each experiment.

Figure 2: Quality of suggested trajectories over time. Across all dimensions, CMA-ES-IG (orange) generates higher-quality trajectories (i.e., trajectories that receive higher reward on average) for users to rank compared to CMA-ES (blue) and Infogain (gray).

Algorithm	$d=4$	$d=8$	$d=16$	$d=32$
CMA-ES-IG	4.1 ms	3.9 ms	4.4 ms	5.3 ms
CMA-ES	0.1 ms	0.1 ms	0.1 ms	0.2 ms
Infogain	418 ms	805 ms	2287 ms	6256 ms

Table 2: Mean runtime (in milliseconds) for query generation methods. Values are averaged across 100 trials for each representation space dimensions (

d

Figure 3: Simulated Domains. We evaluated our algorithms across four simulated domains that represent physical (a,b) and social (c,d) preference-based robot tasks. In Lunar Lander domain (a), preference determines the path of the spaceship takes to land within the flags. In the Driving domain (b), preference determines how the autonomous vehicle merges. In the Face Design domain (c), preference determines the appearance of a screen-based robot face. In the Voice Design domain (d), preference determines the sound of a text-to-speech voice.

Algorithm	Lunar Lander		Driving		Robot Face Design		Robot Voice Design
Algorithm	Alignment $\uparrow$	Regret $\downarrow$	Alignment $\uparrow$	Regret $\downarrow$	Alignment $\uparrow$	Regret $\downarrow$	Alignment $\uparrow$	Regret $\downarrow$
CMA-ES-IG	$.921\pm{\scriptstyle.004}$	$.023\pm{\scriptstyle.003}$	$.933\pm{\scriptstyle.003}$	$.028\pm{\scriptstyle.004}$	$.946\pm{\scriptstyle.004}$	$.015\pm{\scriptstyle.005}$	$.801\pm{\scriptstyle.015}$	$.082\pm{\scriptstyle.026}$
CMA-ES	$.903\pm{\scriptstyle.006}$	$.037\pm{\scriptstyle.005}$	$.928\pm{\scriptstyle.003}$	$.032\pm{\scriptstyle.004}$	$.943\pm{\scriptstyle.002}$	$.016\pm{\scriptstyle.004}$	$.815\pm{\scriptstyle.015}$	$.033\pm{\scriptstyle.006}$
Infogain	$.933\pm{\scriptstyle.003}$	$.019\pm{\scriptstyle.002}$	$.948\pm{\scriptstyle.001}$	$.016\pm{\scriptstyle.002}$	$.960\pm{\scriptstyle.000}$	$.005\pm{\scriptstyle.001}$	$.852\pm{\scriptstyle.014}$	$.017\pm{\scriptstyle.004}$

Table 3: Mean accuracy metrics across simulated robot tasks. Alignment (

\uparrow

) and Regret (

\downarrow

). All algorithms accurately learned user preferences across all tasks, and CMA-ES-IG demonstrates non-inferiority to the baselines within a 3% margin across all tasks.

Figure 4: Quality of suggested trajectories over time for simulated robotic environments. For each simulated robot domain, CMA-ES-IG significantly outperforms the baseline algorithms by producing higher-quality trajectory queries in earlier iterations. This result demonstrates that CMA-ES-IG consistently suggests high-quality trajectories under a variety of representation spaces.

Figure 5: The two domains that users taught robots their preferences for the robot’s behaviors. In the physical domain (a), users ranked a JACO arm’s movement trajectories to hand them a marker, a cup, and a spoon. In the social domain (b), users ranked a Blossom robot’s gestures to portray happiness, sadness, and anger.

Figure 6: User study interface. Users interacted with the robots through the ranking interface to specify their preferences for how the Blossom robot used gestures to signal different affective states and how the JACO robot arm handed them different items.

Algorithm	Behavioral Adaptation ( $\uparrow$ )	Ease of Use( $\uparrow$ )	Rank ( $\uparrow$ )
CMA-ES-IG	5.18 $\pm{\scriptstyle.001}$	5.50 $\pm{\scriptstyle.28}$	1.48 $\pm{\scriptstyle.15}$
CMA-ES	4.69 $\pm{\scriptstyle.42}$	4.87 $\pm{\scriptstyle.37}$	.89 $\pm{\scriptstyle.15}$
Infogain	4.48 $\pm{\scriptstyle.39}$	5.13 $\pm{\scriptstyle.24}$	.63 $\pm{\scriptstyle.13}$

Table 4: Means of Self-reported results. The reported values are aggregated across all physical and social robot tasks.

Figure 7: Users’ preferences for query generation algorithms. CMA-ES-IG was consistently ranked as the most preferred algorithm for users teaching robots preferences in our user study.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension 4
Means: CMA-ES-IG = .88, CMA-ES = .91, Infogain = .93
Alg. 1	Alg. 2	$t$	$p$ -value	Cohen’s $d$
CMA-ES-IG	CMA-ES	-1.60	$.118$	-.41
CMA-ES-IG	Infogain	-3.03	$.004$	-.78
CMA-ES	Infogain	-2.42	$.020$	-.63
Dimension 8

Appendix

Table 5: Alignment results across dimensions. Higher means indicate better policy alignment.

Dimension 4
Means: CMA-ES-IG = .27, CMA-ES = .16, Infogain = .12
Alg. 1	Alg. 2	$t$	$p$ -value	Cohen’s $d$
CMA-ES-IG	CMA-ES	1.92	$.061$	.50
CMA-ES-IG	Infogain	2.94	$.005$	.76
CMA-ES	Infogain	1.37	$.178$	.35
Dimension 8

Appendix

Table 6: Regret results across dimensions. Lower means indicate better performance.

Dimension 4
Means: CMA-ES-IG = .82, CMA-ES = .74, Infogain = .00
Alg. 1	Alg. 2	$t$	$p$ -value	Cohen’s $d$
CMA-ES-IG	CMA-ES	3.35	$.002$	.86
CMA-ES-IG	Infogain	53.21	$<.001$	13.74
CMA-ES	Infogain	39.00	$<.001$	10.07
Dimension 8

Appendix

Table 7: Quality results across dimensions. Higher means indicate better performance.

Lunar Lander
Means: CMA-ES-IG = .92, CMA-ES = .90, Infogain = .93
Alg. 1	Alg. 2	Diff.	$p$ -value	Cohen’s $d$
CMA-ES-IG	CMA-ES	.018	$<.001$	.37
CMA-ES-IG	Infogain	-.012	$<.001$	-.35
Driving
Means: CMA-ES-IG = .93, CMA-ES = .93, Infogain = .95

Appendix

Table 8: Alignment results across tasks. Higher means indicate better policy alignment.

Lunar Lander
Means: CMA-ES-IG = .02, CMA-ES = .04, Infogain = .02
Alg. 1	Alg. 2	Diff.	$p$ -value	Cohen’s $d$
CMA-ES-IG	CMA-ES	-.014	$.002$	-.35
CMA-ES-IG	Infogain	.004	$<.001$	.16
Driving
Means: CMA-ES-IG = .03, CMA-ES = .03, Infogain = .02

Appendix

Table 9: Regret results across tasks. Lower means indicate better performance.

Lunar Lander
Means: CMA-ES-IG = .63, CMA-ES = .55, Infogain = .01
Alg. 1	Alg. 2	$t$	$p$ -value	Cohen’s $d$
CMA-ES-IG	CMA-ES	4.42	$<.001$	.62
CMA-ES-IG	Infogain	50.54	$<.001$	7.15
CMA-ES	Infogain	39.86	$<.001$	5.64
Driving

Appendix

Table 10: Quality results across tasks. Higher means indicate better performance.

EOU — Perceived Ease of Use [ 112 ] $\alpha=.89$
ID	Item Content
1	Teaching the robot is clear and understandable.
2	Teaching the robot does not require a lot of mental effort.
3	I find the robot easy to teach.
4	I find it easy to get the robot to do what I want it to do.
BA — Perceived Behavioral Adaptation [ 61 ] $\alpha=.97$

Appendix

Table 11: Likert scale items for our two metrics. Items are rated on a 7-point scale (1: Strongly Disagree, 7: Strongly Agree).

Behavioral Alignment (BA)
Means: CMA-ES-IG = 5.18, CMA-ES = 4.69, Infogain = 4.48
Alg. 1	Alg. 2	$W$	$p$ -value	Hedges’ $g$
CMA-ES-IG	CMA-ES	15.0	$.032$	.38
CMA-ES-IG	Infogain	5.5	$.006$	.41
CMA-ES	Infogain	41.5	$.797$	.04
Ease of Use (EOU)

Appendix

Table 12: User study subjective metrics and rankings. Higher means indicate better performance/preference.

Figure 8: Initial step-size sensitivity. We show the AUC of the labeled metric (corresponding to the values in Table 1 and Figure 2 ), across 10 different settings of $\sigma$ ranging from 0.01 to 1.5. We find that the choice of hyperparameter does not largely influence any of the three metrics of interest. Across most hyperparameter values, CMA-ES-IG outperforms CMA-ES on Alignment and Regret metrics. Across all hyperparameter values, CMA-ES-IG outperforms CMA-ES on the Quality metric.

Active Preference Learning

See all 16 papers

Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
Jul 14, 2026Taehyung Kim, Gwangmo Lee, Minjun Chang +2Clustering Preference Alignment
Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea
Beyond expert users: agents should help users construct preferences, not just elicit them
Jun 29, 2026Irena Saracay, Ludwig Schmidt, Carlos GuestrinAI Agent Benchmarks Recommender Systems
Stanford University
Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration
Sep 12, 2026Qiping Zhang, Kate Candon, Debasmita Ghose +1Human Preference Modeling Preference Learning
Department of Computer Science Yale University, New Haven, CT, United States

Jul 14, 2026J/K move · Enter open · S save

Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences

Taehyung Kim, Gwangmo Lee, Minjun Chang, Sunghyun Lim+1

Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea

Clustering Preference Alignment User Preference Modeling Preference-Based RL

Open paper AskPDF arXiv

Aligning robot policies with human preferences is essential for deployment to diverse end users. In per-user alignment approach, preference feedback is often sparse, so learning becomes unstable and vulnerable to human preference noise, and a growing number of individualized policies makes validation difficult before deployment. A single shared policy approach to user alignment avoids this cost but fails to capture heterogeneous preferences and often neglects minority preferences. To address these challenges, we introduce Preference-based REward Clustering (PREC), a novel framework that learns a compact set of policies from binary preference labels provided by diverse users. From a dataset of user trajectories and their preference labels, PREC first sets the labels aside and aggregates trajectories across users to learn a population-level shared trajectory encoder, alleviating limited per-user coverage and avoiding label noise during representation learning. Using this representation, PREC jointly assigns users to preference-coherent clusters and learns a representative reward model per cluster using preference labels, from which a policy is optimized for each cluster. Clustering similar users compensates for the limited number of labels available from each user and mitigates the effect of label noise. At the same time, maintaining a manageable number of reward models reduces the validation burden at deployment. Experiments across diverse simulated locomotion environments show that PREC groups users who label different trajectory subsets into preference-coherent clusters more accurately than baseline methods. Under sparse and noisy feedback, policies trained with PREC improve all three social welfare metrics over an existing single shared-policy user-alignment approach and even outperform per-user alignment approaches.

Abstract

Active Preference Learning

Explore similar work

Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences

Beyond expert users: agents should help users construct preferences, not just elicit them

Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration