Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

Haoyang Jiang; Dana Kulic; Michael Burke; Pamela Carreno-Medrano

cs.ROOct 26, 2026

Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

Authors: Haoyang Jiang, Dana Kulic, Michael Burke, Pamela Carreno-Medrano

Organizations: Department of Electrical and Computer Systems Engineering, Monash University, Clayton, Australia

Abstract

Human behaviour models are increasingly used to provide simulated human interactions for training robots via reinforcement learning (RL). Existing approaches primarily evaluate these models according to how closely their generated behaviours resemble real human data, while their usefulness as training partners is less commonly assessed. This creates a potential objective mismatch: a model that performs well according to behavioural evaluation metrics may not necessarily provide useful training experiences for downstream robot policy learning. Across two human-robot interaction (HRI) domains, highway driving and object handover, we evaluate different types of human behaviour models using common generative-model evaluation metrics. We consider multiple human trajectory representations and compare the resulting evaluations with the downstream performance of RL policies trained using the corresponding models. We find that existing behavioural evaluation metrics do not consistently predict downstream policy performance across domains and trajectory representations, while their evaluations are sensitive to the choice of trajectory representation. We also show that even when the true testing behaviour distribution is known, training against a different distribution may yield better downstream performance than training directly against the true distribution.

Figures & tables

Domain	Type	Model
Highway driving	Rule-based	MOBIL+Gipps
		MOBIL+Krauß
		MOBIL+IDM
	Data-driven	MLP
		MDN
		Attention

TABLE I: Human behaviour models considered in the highway driving and object handover domains.

Model	Crash Rate (%)	Drive Off Rate (%)
Gipps	0.00	0.00
Krauß	0.00	0.00
IDM	0.06	0.00
MLP	63.13	75.36
MDN	49.35	41.17
Attention	44.60	4.99

TABLE II: Crash rate and drive-off rate of the human driving models.

Panel of Figure 3 — Fig. 1: 50 randomly sampled handover trajectories from human demonstrations and trajectories generated by the human behaviour models.

Domain	Repr.	Metrics
Highway driving (8 models)	PCA	FD, density, coverage, $APD_{\mathrm{gap}}$ , $MMS_{\mathrm{gap}}$
	Latent
	Manual	4 feature KL + averaged KL
Object handover (6 models)	PCA	FD, density, coverage, $APD_{\mathrm{gap}}$ , $MMS_{\mathrm{gap}}$
	Latent
	Manual	6 feature KL + averaged KL

TABLE III: Evaluation representations (Repr.) and metrics used for the human behaviour models in each domain.

Domain	Model	Repr.	Fidelity			Diversity
Domain	Model	Repr.	FD $\downarrow$	Density $\uparrow$	KL $\downarrow$	Coverage $\uparrow$	$APD_{\mathrm{gap}}$ $\downarrow$	$MMS_{\mathrm{gap}}$ $\downarrow$
Highway	Gipps	Latent	$25.24$ $\pm 20.61$	$1.73\mathrm{e}{-3}$ $\pm 3.28\mathrm{e}{-3}$		$2.81\mathrm{e}{-3}$ $\pm 4.74\mathrm{e}{-3}$	$2.65$ $\pm 1.95$	$2.80$ $\pm 1.47$
		PCA	$88.63$ $\pm 77.15$	$0.15$ $\pm 0.18$		$0.08$ $\pm 0.12$	$5.67$ $\pm 1.66$	$4.74$ $\pm 3.42$
		Manual			$0.23$ $\pm 0.01$
	Krauss	Latent	$25.35$ $\pm 20.70$	$1.96\mathrm{e}{-3}$ $\pm 3.87\mathrm{e}{-3}$		$2.93\mathrm{e}{-3}$ $\pm 4.55\mathrm{e}{-3}$	$2.66$ $\pm 1.96$	$2.80$ $\pm 1.47$
		PCA	$90.22$ $\pm 77.74$	$0.14$ $\pm 0.18$		$0.07$ $\pm 0.12$	$5.64$ $\pm 1.63$	$4.82$ $\pm 3.41$

TABLE IV: Evaluation metrics (mean

\pm

std over configurations) by domain, model, and representation. Within each domain, the best Latent and Manual values are in bold and the best PCA values are underlined. Empty cells mark metrics that do not apply to a representation.

Domain	Representation	Configuration	Fidelity			Diversity
Domain	Representation	Configuration	FD	Density	KL	Coverage	APD gap	MMS gap
Highway	Latent	Latent = 16, Hidden = 64	0.00	0.61 ×		0.00	0.39	0.00
		Latent = 16, Hidden = 128	0.45 ×	0.45 ×		0.00	0.01	0.39
		Latent = 16, Hidden = 256	0.61 ×	0.61 ×		0.61 ×	0.01	0.61 ×
		Latent = 32, Hidden = 64	0.01	0.01		0.01	0.85 ×	0.85 ×
		Latent = 32, Hidden = 128	0.01	0.01		0.01	0.85 ×	0.85 ×

TABLE V: Normalised regret of metric-selected models by domain, representation, and configuration. A superscript

\times

marks a metric whose

\mathrm{Reg}_{j}^{d}\geq\mathrm{Reg}_{\mathrm{avg}}^{d}

Panel of Figure 7 — Fig. 2: Cross-evaluation results for policies trained and evaluated with different human behaviour models. Each entry represents the mean reward $R_{ij}$ and corresponding reward standard deviation of a policy trained with model $H_{i}^{\mathrm{train}}$ (row) when evaluated against model $H_{j}^{\mathrm{eval}}$ (column). (a) Highway domain: $R_{ij}^{\mathrm{HW}}$ , averaged over 100 evaluation episodes and three seeds. (b) Handover domain: $R_{ij}^{\mathrm{HO}}$ , averaged over 200 evaluation episodes and three seeds. The last column in each subfigure shows the average cross-evaluation reward $R_{i}$ .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{L}$	2.486	0.068	2.358	2.610	0.002	0.001	1906.0	1931.0
$\sigma_{L}$	1.332	0.069	1.211	1.466	0.002	0.001	2030.0	2001.0
$\sigma_{\text{obs}}$	1.466	0.021	1.430	1.508	0.001	0.000	1793.0	1865.0

Appendix

TABLE VI: Gipps’ ADVI report for

L

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{v}^{\max}$	32.336	0.216	31.926	32.734	0.005	0.003	2033.0	1882.0
$\sigma_{v}^{\max}$	5.541	0.136	5.292	5.805	0.003	0.002	1914.0	1810.0
$\sigma_{\text{obs}}$	0.243	0.002	0.240	0.246	0.000	0.000	1975.0	1886.0

Appendix

TABLE VII: Gipps’ ADVI report for

v^{\max}

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{\tau}$	0.087	0.051	-0.006	0.184	0.001	0.001	1695.0	1766.0
$\sigma_{\tau}$	1.341	0.045	1.256	1.423	0.001	0.001	1930.0	1743.0
$\sigma_{\text{obs}}$	0.995	0.020	0.959	1.034	0.000	0.000	1896.0	1885.0

Appendix

TABLE VIII: Krauß ADVI report for

\tau

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{v}^{\max}$	32.671	0.096	32.491	32.848	0.002	0.002	1993.0	1961.0
$\sigma_{v}^{\max}$	1.249	0.085	1.102	1.408	0.002	0.001	2148.0	2004.0
$\mu_{d_{0}}$	0.371	0.088	0.210	0.540	0.002	0.001	2005.0	1985.0
$\sigma_{d_{0}}$	1.338	0.085	1.183	1.504	0.002	0.001	1981.0	1984.0
$\mu_{T}$	-0.812	0.086	-0.979	-0.664	0.002	0.001	1996.0	1960.0
$\sigma_{T}$	1.258	0.084	1.110	1.427	0.002	0.001	2065.0	1698.0

Appendix

TABLE IX: IDM ADVI Report for

v_{n}^{\max}

d_{0}

and

T

Fig. 3: An illustration of the attention-based model architecture. At each time step

t

, the current ego state

\mathbf{s}_{\text{ego}}^{t}\in\mathbb{R}^{6}

(where bold symbols denote vectors) queries its past

h

ego states

\mathbf{s}_{\text{ego}}^{t-h:t-1}\in\mathbb{R}^{h\times 6}

to produce the ego embedding

\mathbf{z}_{\text{ego}}\in\mathbb{R}^{H\cdot d_{z}}

. This embedding then queries the states of the

N

surrounding vehicles (

N=7

in our case)

\mathbf{s}_{\text{surr}}\in\mathbb{R}^{N\times(h+1)\times 6}

to generate the interaction embedding

\mathbf{z}_{\text{inter}}\in\mathbb{R}^{H\cdot d_{z}}

. The ego and interaction embeddings are concatenated into a single scene representation

\mathbf{z}=\text{Concat}(\mathbf{z}_{\text{ego}},\mathbf{z}_{\text{inter}})\in\mathbb{R}^{2H\cdot d_{z}}

, which is used as the conditioning input to a Conditional Variational Autoencoder (CVAE) [ 69 ] . During training, the encoder takes both

\mathbf{z}

and the ground-truth control action

\mathbf{a}_{\text{ego}}^{t}\in\mathbb{R}^{2}

to infer a latent variable

\mathbf{z}_{\text{lat}}

. The decoder then generates the control action

\hat{\mathbf{a}}_{\text{ego}}^{t}\in\mathbb{R}^{2}

conditioned on the concatenation of

\mathbf{z}

and a sampled latent vector. During inference,

\mathbf{z}_{\text{lat}}

is sampled from a prior distribution.

Fig. 4: Kernel density estimates of four driving feature distributions generated by each calibrated car-following model, overlaid with the corresponding distributions from highD data.

Feature	Model	Mean	Variance	Std. Dev.	Median	Q 25	Q 75
Velocity	highD data	31.182	27.570	5.251	31.450	27.960	34.380
	Gipps	28.642	15.168	3.895	28.796	26.219	31.289
	Krauss	29.936	20.549	4.533	31.045	27.201	33.648
	IDM	28.125	6.866	2.620	28.508	26.713	29.885
	MLP	26.619	83.163	9.119	28.578	23.183	32.446
	MDN	31.333	26.904	5.187	31.536	27.971	34.650

Appendix

TABLE X: Summary statistics of driving features across different car-following models and highD data.

Model	Velocity	Relative preceding velocity	Distance headway	Acceleration
Gipps	0.024	0.510	0.214	0.148
Krauss	0.067	0.515	0.316	0.188
IDM	0.142	0.687	0.219	0.087
MLP	0.010	0.071	0.013	0.029
MDN	0.002	0.338	0.031	0.024
Attention	0.001	0.340	0.053	0.267

Appendix

TABLE XI: KL divergence

D_{\mathrm{KL}}(P_{\text{highD}}\,\|\,Q_{\text{model}})

of driving feature distributions between highD data and each car-following model (lower is better).

Fig. 5: Comparison of handover giver trajectory feature distributions between calibrated giver models and human data. Kernel density estimates are shown for end-to-end length, path length, average speed, average acceleration, movement smoothness, and trajectory curvature.

Feature	Model	Mean	Variance	Std. Dev.	Median	Q 25	Q 75
End-to-end length	Human	0.529	0.005	0.071	0.541	0.490	0.581
	Minijerk	0.537	0.005	0.072	0.536	0.489	0.583
	Elliptical	0.531	0.005	0.069	0.529	0.485	0.579
	DMP	0.528	0.005	0.072	0.524	0.483	0.575
	MLP	0.556	0.007	0.082	0.561	0.506	0.608
	MDN	0.531	0.006	0.076	0.539	0.483	0.587

Appendix

TABLE XII: Summary statistics of handover giver trajectory features across giver models and human data.

Model	End-to-end length	Path length	Average speed	Average acceleration	Movement smoothness	Trajectory curvature
Minijerk	0.054	0.064	0.084	0.696	11.743	1.602
Elliptical	0.060	0.060	0.069	0.399	2.298	1.109
DMP	0.067	0.057	0.068	0.250	2.299	0.288
MLP	0.118	0.336	1.059	0.951	2.397	4.083
MDN	0.037	0.093	4.088	0.644	7.449	0.270
Attention	0.140	0.211	1.824	1.619	1.871	3.770

Appendix

TABLE XIII: KL divergence

D_{\mathrm{KL}}(P_{\text{human}}\,\|\,Q_{\text{model}})

of handover giver trajectory feature distributions between human data and each giver model (lower is better).

Fig. 6: Architecture of the autoencoder used for latent feature extraction. A BiLSTM encoder maps the input state sequence to a latent vector

z

, which is decoded by a repeated-vector LSTM and an MLP to reconstruct the original sequence.

Human Behavior Modeling

See all 35 papers

Position: Good Embodied Reward Models Need Bad Behavior Data
May 31, 2026Ran Tian, Yilin Wu, Andrea BajcsyReward Modeling Preference Alignment
UC Berkeley · NVIDIA Research · Carnegie Mellon University
Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence
Aug 30, 2026Gaytri Jena, Kapil Wanaskar, Vinija Jain +3World Model Evaluation AI Agent Benchmarks
UC Berkeley, USA · San Jose State University, USA · Meta, USA +3
WorldFailBench: Do World Models Understand Policy Failures?
Oct 24, 2026Zheng Zhang, Xiaoyang Tang, Shuaiqi Cheng +11World Model Evaluation World Models for Robotics

May 31, 2026J/K move · Enter open · S save

Position: Good Embodied Reward Models Need Bad Behavior Data

Ran Tian, Yilin Wu, Andrea Bajcsy

UC Berkeley · NVIDIA Research · Carnegie Mellon University

Reward Modeling Preference Alignment Reward Model Evaluation

Open paper AskPDF arXiv

This position paper argues that to obtain reliable embodied reward models, the community must invest in ``bad'' robot data: failed, suboptimal, error-prone, and even hazardous behaviors. While reward models are central to any foundation model's lifecycle, today's embodied reward models are trained primarily on successful behaviors. We analyze three state-of-the-art embodied reward models and find that they systematically over-reward behaviors that real human evaluators would penalize, including unsafe interactions, poor execution, and shortcut strategies that only superficially satisfy tasks. We attribute these failures to a key data gap: the scarcity of negative embodied data which is costly to collect and often filtered out or withheld in existing robotics datasets. Furthermore, we show that even modest exposure to real bad behavior data can improve alignment with human preferences and reduce costly false positives. We therefore call on the embodied AI community to curate and release their bad robot data, build synthetic bad data generation engines, develop more decentralized physical evaluation systems, and design benchmarks for fine-grained embodied reward model evaluations.

Abstract

Human Behavior Modeling

Explore similar work

Position: Good Embodied Reward Models Need Bad Behavior Data

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

WorldFailBench: Do World Models Understand Policy Failures?