Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

Haoyang Jiang; Dana Kulic; Michael Burke; Pamela Carreno-Medrano

cs.ROOct 26, 2026

Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

Authors: Haoyang Jiang, Dana Kulic, Michael Burke, Pamela Carreno-Medrano

Organizations: Department of Electrical and Computer Systems Engineering, Monash University, Clayton, Australia

Abstract

Human behaviour models are increasingly used to provide simulated human interactions for training robots via reinforcement learning (RL). Existing approaches primarily evaluate these models according to how closely their generated behaviours resemble real human data, while their usefulness as training partners is less commonly assessed. This creates a potential objective mismatch: a model that performs well according to behavioural evaluation metrics may not necessarily provide useful training experiences for downstream robot policy learning. Across two human-robot interaction (HRI) domains, highway driving and object handover, we evaluate different types of human behaviour models using common generative-model evaluation metrics. We consider multiple human trajectory representations and compare the resulting evaluations with the downstream performance of RL policies trained using the corresponding models. We find that existing behavioural evaluation metrics do not consistently predict downstream policy performance across domains and trajectory representations, while their evaluations are sensitive to the choice of trajectory representation. We also show that even when the true testing behaviour distribution is known, training against a different distribution may yield better downstream performance than training directly against the true distribution.

Figures & tables

Domain	Type	Model
Highway driving	Rule-based	MOBIL+Gipps
		MOBIL+Krauß
		MOBIL+IDM
	Data-driven	MLP
		MDN
		Attention

TABLE I: Human behaviour models considered in the highway driving and object handover domains.

Model	Crash Rate (%)	Drive Off Rate (%)
Gipps	0.00	0.00
Krauß	0.00	0.00
IDM	0.06	0.00
MLP	63.13	75.36
MDN	49.35	41.17
Attention	44.60	4.99

TABLE II: Crash rate and drive-off rate of the human driving models.

Panel of Figure 3 — Fig. 1: 50 randomly sampled handover trajectories from human demonstrations and trajectories generated by the human behaviour models.

Domain	Repr.	Metrics
Highway driving (8 models)	PCA	FD, density, coverage, $APD_{\mathrm{gap}}$ , $MMS_{\mathrm{gap}}$
	Latent
	Manual	4 feature KL + averaged KL
Object handover (6 models)	PCA	FD, density, coverage, $APD_{\mathrm{gap}}$ , $MMS_{\mathrm{gap}}$
	Latent
	Manual	6 feature KL + averaged KL

TABLE III: Evaluation representations (Repr.) and metrics used for the human behaviour models in each domain.

Domain	Model	Repr.	Fidelity			Diversity
Domain	Model	Repr.	FD $\downarrow$	Density $\uparrow$	KL $\downarrow$	Coverage $\uparrow$	$APD_{\mathrm{gap}}$ $\downarrow$	$MMS_{\mathrm{gap}}$ $\downarrow$
Highway	Gipps	Latent	$25.24$ $\pm 20.61$	$1.73\mathrm{e}{-3}$ $\pm 3.28\mathrm{e}{-3}$		$2.81\mathrm{e}{-3}$ $\pm 4.74\mathrm{e}{-3}$	$2.65$ $\pm 1.95$	$2.80$ $\pm 1.47$
		PCA	$88.63$ $\pm 77.15$	$0.15$ $\pm 0.18$		$0.08$ $\pm 0.12$	$5.67$ $\pm 1.66$	$4.74$ $\pm 3.42$
		Manual			$0.23$ $\pm 0.01$
	Krauss	Latent	$25.35$ $\pm 20.70$	$1.96\mathrm{e}{-3}$ $\pm 3.87\mathrm{e}{-3}$		$2.93\mathrm{e}{-3}$ $\pm 4.55\mathrm{e}{-3}$	$2.66$ $\pm 1.96$	$2.80$ $\pm 1.47$
		PCA	$90.22$ $\pm 77.74$	$0.14$ $\pm 0.18$		$0.07$ $\pm 0.12$	$5.64$ $\pm 1.63$	$4.82$ $\pm 3.41$

TABLE IV: Evaluation metrics (mean

\pm

std over configurations) by domain, model, and representation. Within each domain, the best Latent and Manual values are in bold and the best PCA values are underlined. Empty cells mark metrics that do not apply to a representation.

Domain	Representation	Configuration	Fidelity			Diversity
Domain	Representation	Configuration	FD	Density	KL	Coverage	APD gap	MMS gap
Highway	Latent	Latent = 16, Hidden = 64	0.00	0.61 ×		0.00	0.39	0.00
		Latent = 16, Hidden = 128	0.45 ×	0.45 ×		0.00	0.01	0.39
		Latent = 16, Hidden = 256	0.61 ×	0.61 ×		0.61 ×	0.01	0.61 ×
		Latent = 32, Hidden = 64	0.01	0.01		0.01	0.85 ×	0.85 ×
		Latent = 32, Hidden = 128	0.01	0.01		0.01	0.85 ×	0.85 ×

TABLE V: Normalised regret of metric-selected models by domain, representation, and configuration. A superscript

\times

marks a metric whose

\mathrm{Reg}_{j}^{d}\geq\mathrm{Reg}_{\mathrm{avg}}^{d}

Panel of Figure 7 — Fig. 2: Cross-evaluation results for policies trained and evaluated with different human behaviour models. Each entry represents the mean reward $R_{ij}$ and corresponding reward standard deviation of a policy trained with model $H_{i}^{\mathrm{train}}$ (row) when evaluated against model $H_{j}^{\mathrm{eval}}$ (column). (a) Highway domain: $R_{ij}^{\mathrm{HW}}$ , averaged over 100 evaluation episodes and three seeds. (b) Handover domain: $R_{ij}^{\mathrm{HO}}$ , averaged over 200 evaluation episodes and three seeds. The last column in each subfigure shows the average cross-evaluation reward $R_{i}$ .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{L}$	2.486	0.068	2.358	2.610	0.002	0.001	1906.0	1931.0
$\sigma_{L}$	1.332	0.069	1.211	1.466	0.002	0.001	2030.0	2001.0
$\sigma_{\text{obs}}$	1.466	0.021	1.430	1.508	0.001	0.000	1793.0	1865.0

Appendix

TABLE VI: Gipps’ ADVI report for

L

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{v}^{\max}$	32.336	0.216	31.926	32.734	0.005	0.003	2033.0	1882.0
$\sigma_{v}^{\max}$	5.541	0.136	5.292	5.805	0.003	0.002	1914.0	1810.0
$\sigma_{\text{obs}}$	0.243	0.002	0.240	0.246	0.000	0.000	1975.0	1886.0

Appendix

TABLE VII: Gipps’ ADVI report for

v^{\max}

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{\tau}$	0.087	0.051	-0.006	0.184	0.001	0.001	1695.0	1766.0
$\sigma_{\tau}$	1.341	0.045	1.256	1.423	0.001	0.001	1930.0	1743.0
$\sigma_{\text{obs}}$	0.995	0.020	0.959	1.034	0.000	0.000	1896.0	1885.0

Appendix

TABLE VIII: Krauß ADVI report for

\tau

	mean	sd	hdi_3%	hdi_97%	mcse_mean	mcse_sd	ess_bulk	ess_tail
$\mu_{v}^{\max}$	32.671	0.096	32.491	32.848	0.002	0.002	1993.0	1961.0
$\sigma_{v}^{\max}$	1.249	0.085	1.102	1.408	0.002	0.001	2148.0	2004.0
$\mu_{d_{0}}$	0.371	0.088	0.210	0.540	0.002	0.001	2005.0	1985.0
$\sigma_{d_{0}}$	1.338	0.085	1.183	1.504	0.002	0.001	1981.0	1984.0
$\mu_{T}$	-0.812	0.086	-0.979	-0.664	0.002	0.001	1996.0	1960.0
$\sigma_{T}$	1.258	0.084	1.110	1.427	0.002	0.001	2065.0	1698.0

Appendix

TABLE IX: IDM ADVI Report for

v_{n}^{\max}

d_{0}

and

T

Fig. 3: An illustration of the attention-based model architecture. At each time step

t

, the current ego state

\mathbf{s}_{\text{ego}}^{t}\in\mathbb{R}^{6}

(where bold symbols denote vectors) queries its past

h

ego states

\mathbf{s}_{\text{ego}}^{t-h:t-1}\in\mathbb{R}^{h\times 6}

to produce the ego embedding

\mathbf{z}_{\text{ego}}\in\mathbb{R}^{H\cdot d_{z}}

. This embedding then queries the states of the

N

surrounding vehicles (

N=7

in our case)

\mathbf{s}_{\text{surr}}\in\mathbb{R}^{N\times(h+1)\times 6}

to generate the interaction embedding

\mathbf{z}_{\text{inter}}\in\mathbb{R}^{H\cdot d_{z}}

. The ego and interaction embeddings are concatenated into a single scene representation

\mathbf{z}=\text{Concat}(\mathbf{z}_{\text{ego}},\mathbf{z}_{\text{inter}})\in\mathbb{R}^{2H\cdot d_{z}}

, which is used as the conditioning input to a Conditional Variational Autoencoder (CVAE) [ 69 ] . During training, the encoder takes both

\mathbf{z}

and the ground-truth control action

\mathbf{a}_{\text{ego}}^{t}\in\mathbb{R}^{2}

to infer a latent variable

\mathbf{z}_{\text{lat}}

. The decoder then generates the control action

\hat{\mathbf{a}}_{\text{ego}}^{t}\in\mathbb{R}^{2}

conditioned on the concatenation of

\mathbf{z}

and a sampled latent vector. During inference,

\mathbf{z}_{\text{lat}}

is sampled from a prior distribution.

Fig. 4: Kernel density estimates of four driving feature distributions generated by each calibrated car-following model, overlaid with the corresponding distributions from highD data.

Feature	Model	Mean	Variance	Std. Dev.	Median	Q 25	Q 75
Velocity	highD data	31.182	27.570	5.251	31.450	27.960	34.380
	Gipps	28.642	15.168	3.895	28.796	26.219	31.289
	Krauss	29.936	20.549	4.533	31.045	27.201	33.648
	IDM	28.125	6.866	2.620	28.508	26.713	29.885
	MLP	26.619	83.163	9.119	28.578	23.183	32.446
	MDN	31.333	26.904	5.187	31.536	27.971	34.650

Appendix

TABLE X: Summary statistics of driving features across different car-following models and highD data.

Model	Velocity	Relative preceding velocity	Distance headway	Acceleration
Gipps	0.024	0.510	0.214	0.148
Krauss	0.067	0.515	0.316	0.188
IDM	0.142	0.687	0.219	0.087
MLP	0.010	0.071	0.013	0.029
MDN	0.002	0.338	0.031	0.024
Attention	0.001	0.340	0.053	0.267

Appendix

TABLE XI: KL divergence

D_{\mathrm{KL}}(P_{\text{highD}}\,\|\,Q_{\text{model}})

of driving feature distributions between highD data and each car-following model (lower is better).

Fig. 5: Comparison of handover giver trajectory feature distributions between calibrated giver models and human data. Kernel density estimates are shown for end-to-end length, path length, average speed, average acceleration, movement smoothness, and trajectory curvature.

Feature	Model	Mean	Variance	Std. Dev.	Median	Q 25	Q 75
End-to-end length	Human	0.529	0.005	0.071	0.541	0.490	0.581
	Minijerk	0.537	0.005	0.072	0.536	0.489	0.583
	Elliptical	0.531	0.005	0.069	0.529	0.485	0.579
	DMP	0.528	0.005	0.072	0.524	0.483	0.575
	MLP	0.556	0.007	0.082	0.561	0.506	0.608
	MDN	0.531	0.006	0.076	0.539	0.483	0.587

Appendix

TABLE XII: Summary statistics of handover giver trajectory features across giver models and human data.

Model	End-to-end length	Path length	Average speed	Average acceleration	Movement smoothness	Trajectory curvature
Minijerk	0.054	0.064	0.084	0.696	11.743	1.602
Elliptical	0.060	0.060	0.069	0.399	2.298	1.109
DMP	0.067	0.057	0.068	0.250	2.299	0.288
MLP	0.118	0.336	1.059	0.951	2.397	4.083
MDN	0.037	0.093	4.088	0.644	7.449	0.270
Attention	0.140	0.211	1.824	1.619	1.871	3.770

Appendix

TABLE XIII: KL divergence

D_{\mathrm{KL}}(P_{\text{human}}\,\|\,Q_{\text{model}})

of handover giver trajectory feature distributions between human data and each giver model (lower is better).

Fig. 6: Architecture of the autoencoder used for latent feature extraction. A BiLSTM encoder maps the input state sequence to a latent vector

z

, which is decoded by a repeated-vector LSTM and an MLP to reconstruct the original sequence.

Human Behavior Modeling

See all 35 papers

May 31, 2026

Position: Good Embodied Reward Models Need Bad Behavior Data

This position paper argues that to obtain reliable embodied reward models, the community must invest in ``bad'' robot data: failed, suboptimal, error-prone, and even hazardous behaviors. While reward models are central to any foundation model's lifecycle, today's embodied reward models are trained primarily on successful behaviors. We analyze three state-of-the-art embodied reward models and find that they systematically over-reward behaviors that real human evaluators would penalize, including unsafe interactions, poor execution, and shortcut strategies that only superficially satisfy tasks. We attribute these failures to a key data gap: the scarcity of negative embodied data which is costly to collect and often filtered out or withheld in existing robotics datasets. Furthermore, we show that even modest exposure to real bad behavior data can improve alignment with human preferences and reduce costly false positives. We therefore call on the embodied AI community to curate and release their bad robot data, build synthetic bad data generation engines, develop more decentralized physical evaluation systems, and design benchmarks for fine-grained embodied reward model evaluations.

Ran Tian, Yilin Wu, Andrea Bajcsy

UC Berkeley · NVIDIA Research · Carnegie Mellon University

Reward Modeling Preference Alignment Reward Model Evaluation

Ask PDF

Aug 30, 2026

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

Robot learning now advances along two tracks that rarely meet. On one side, direct Vision-Language-Action (VLA) policies map observations to actions and are scored by closed-loop task success. On the other, predictive and generative world models forecast future observations and are scored by open-loop prediction or generation quality. A natural question sits between them: does world modelling earn a measurable, closed-loop advantage over a direct policy, and for which robotic capabilities? We argue that the field cannot yet answer this question, and that the reason is a gap in how it is measured, not in the models themselves. World-model benchmarks score prediction without ever executing it, while task-success suites host a single policy and never build a world-model versus VLA contrast. This survey maps the evaluation landscape around that gap. We catalogue 160 web-verified benchmarks spanning 2017 to 2026 and organise them by evaluation mode, robotic capability, and model family into four lanes: policy suites, embodied agents, world model evaluation, and prediction-to-action bridges. Across the corpus, 138 of 160 benchmarks are model-agnostic and only 11 (7%) build an explicit VLA-versus-world-model contrast; counterfactual capability is almost entirely unmeasured, and only four benchmarks turn prediction into executed action. We contribute an operational taxonomy, a coverage comparison against the eight closest surveys (ours is the only one to cross capability with model family), an evaluation loop that isolates the advantage of prediction, and an actionable protocol of four advantage-aware metrics anchored on named testbeds. The organising claim is not that world models help or do not help, but that answering the question requires benchmarks built to ask it.

Gaytri Jena, Kapil Wanaskar, Vinija Jain +3

UC Berkeley, USA · San Jose State University, USA · Meta, USA +3

World Model Evaluation AI Agent Benchmarks Embodied World Models Robotic Policy Evaluation

Ask PDF

Oct 24, 2026

WorldFailBench: Do World Models Understand Policy Failures?

Robot policy evaluation informs deployment decisions and policy improvement, but physical testing and simulator preparation require substantial effort. World models offer a complementary route through closed-loop interaction in generated environments. Recent benchmarks have demonstrated their potential as policy evaluators through outcome-based evaluation (i.e., assessing task success or failure). However, such evaluation does not establish whether world models faithfully simulate how policies fail. Misrepresenting failures can lead to incorrect diagnoses of policy weaknesses and make policies appear ready for deployment even when small scene changes cause them to fail. We therefore ask: do world models faithfully simulate how and at which stages policies fail, and under what conditions they transition from success to failure? To answer this question, we introduce WorldFailBench with three evaluation tasks. Failure reason tests whether models reproduce the observable errors policies make. Failure stage assesses whether failures concentrate at the same stages of full-task execution. Failure boundary examines whether models reproduce the conditions under which policies transition from success to failure under controlled scene changes in simulation. The benchmark covers manipulation tasks in the LIBERO and RoboTwin simulation environments and real-world manipulation tasks performed by the G1 humanoid robot. Experiments across seven world models reveal substantial gaps in their understanding of policy failures, providing guidance for developing more reliable world models for policy evaluation.

Zheng Zhang, Xiaoyang Tang, Shuaiqi Cheng +11

World Model Evaluation World Models for Robotics Robotic Policy Evaluation Robot Manipulation Benchmarks

Ask PDF

Abstract

Human Behavior Modeling

Explore similar work

Position: Good Embodied Reward Models Need Bad Behavior Data

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

WorldFailBench: Do World Models Understand Policy Failures?