Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation

Bowen Yang; Xinliang Xiao; Wenjing Zhang; Li Yang; Wei Zhou

cs.ROOct 8, 2026

Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation

Authors: Bowen Yang, Xinliang Xiao, Wenjing Zhang, Li Yang, Wei Zhou

Organizations: School of Automation, Nanjing University of Science and Technology, Nanjing, 210094, China.

Abstract

Social and service robots in public spaces need to anticipate which nearby person is about to approach and touch them, so that a response can be prepared before contact. It is largely unknown which cues support this anticipation when a model trained with one robot is used on another robot at a different site. We study this question with a fixed-reference pose residual (FRPR) model: a geometry predictor built from the person's bounding box and mask is trained and frozen, and a temporal network then learns from body pose an additive correction to its logit, so that every prediction splits exactly into a geometry term and a pose term. Between two public egocentric datasets recorded by different robots, HUI360 and SSUP-A, with every choice made on source data, the pose correction raised average precision (AP) from 0.277 to 0.321 from SSUP-A to HUI360 and gave no measurable gain in the opposite direction; the same asymmetry held over a stronger, source-selected geometry reference. Freezing gave no AP advantage over joint training, and simple geometric baselines and tree ensembles remained competitive or better, so the construction serves measurement rather than prediction accuracy. A head-orientation residual added small gains in both directions. Post hoc, whether a person faces the camera kept its discriminative direction across datasets, whereas head pitch reversed. With thresholds chosen on source data, the neural models that use geometry detected at most 17% of target interactions. Code and processed data are available at https://github.com/WeiZhou96/FRPR-interaction-anticipation.

Figures & tables

Figure 1: Fixed-reference pose residual (FRPR) and its head-orientation extension (FRPR-H). (a) A 32-frame track at 15 fps yields a geometry summary $a$ , a box-normalised pose sequence $q$ and, for FRPR-H only, a head-orientation summary $h$ from dense face keypoints; all inputs are standardised with source-training statistics. (b1) Stage 1 trains the geometry multilayer perceptron (MLP), whose logit $g=g_{\theta^{*}}(a)$ gives the stand-alone reference score $p_{G}=\sigma(g)$ ; it remains frozen in Stages 2 and 3. (b2) Stage 2 trains a causal temporal encoder with a zero-initialised linear head, producing the pose residual $r=r_{\phi}(q)$ . (b3) Stage 3, used by FRPR-H only, trains a small MLP with a zero-initialised output on $h$ while $g$ and $r$ stay frozen. (c) The logits are summed before the sigmoid, so each score splits exactly into the terms shown. Frames and crops are taken from the public HUI360 demonstration material; the coloured feature cells are schematic

		HUI360 $\rightarrow$ SSUP-A				SSUP-A $\rightarrow$ HUI360
Method	Input	AP $\uparrow$	AUROC $\uparrow$	Recall $\uparrow$	FPR $\downarrow$	AP $\uparrow$	AUROC $\uparrow$	Recall $\uparrow$	FPR $\downarrow$
Last-frame box area a	$a$	0.178	0.858	0.61	0.133	0.341	0.732	0.01	0.012
Geometry reference	$a$	0.129 $\pm$ 0.014	0.841	0.03	0.004	0.277 $\pm$ 0.022	0.733	0.00	0.006
Random forest	$a$	0.146 $\pm$ 0.006	0.860	0.19	0.028	0.369 $\pm$ 0.011	0.738	0.15	0.031
HistGB b	$a$	0.161	0.841	0.19	0.022	0.311	0.688	0.19	0.047
Pose TCN	$q$	0.048 $\pm$ 0.002	0.652	0.21	0.119	0.334 $\pm$ 0.009	0.757	0.21	0.080

Table 1: Target-domain results under source-only model selection. AP and AUROC are means over five training seeds (

\pm

standard deviation of AP); recall and false-positive rate (FPR) are taken at the threshold that maximises

F_{1}

on source validation data.

\bar{q}

: summary statistics of the pose sequence;

h

k

: head-orientation and hand summaries. The last block contains the orientation extension and HistGB with

h

appended, each scored once after its own lock. Bold and underline: highest and second-highest AP per direction; shading: FRPR and FRPR-H. AP is not comparable between directions because prevalence differs (0.030 and 0.167)

Figure 2: Paired AP differences between FRPR and each comparator on the target domain. Thick bars are 95% percentile intervals from a bootstrap over recording dates and training seeds. Squares mark the four primary comparisons; for these only, thin bars give the 98.75% intervals of the Bonferroni family. Circles mark exploratory comparisons with unadjusted 95% intervals; filled markers indicate intervals that exclude zero. Ticks under the first row are the differences obtained when one target date is removed. Horizontal scales differ between panels; all values are listed in Online Resource 1 (Table S3)

Figure 3: The two terms of FRPR on target data. (a, b) Reference logit against pose correction for every target sample (means over five seeds; 1500 randomly drawn non-interacting samples are shown in (a)), with the marginal distribution of the correction by class; dotted lines and printed values are class medians. (c) AUROC of each term scored alone, on source validation dates and on the target. In HUI360 $\rightarrow$ SSUP-A the reference keeps most of its ranking quality and the correction does not; in SSUP-A $\rightarrow$ HUI360 the correction keeps its level while the reference drops and has the larger logit spread (note the different axis ranges in (b))

Figure 4: Transfer of the head-orientation cues and of the terms of FRPR-H. (a, b) AUROC of each entry of

h

used alone as a score, on the source pool (open circles) and on the target set (filled circles); values below 0.5 mean that lower values indicate interaction. Colour and line style mark whether source and target lie on the same side of 0.5 (blue, solid), on opposite sides (orange, dashed), or both within 0.45–0.55 (grey, dotted); target values at least 0.15 from 0.5 are printed. Computed after the locked target scoring; post hoc and descriptive. (c) Target AUROC of the reference logit

g

, the pose term

r_{q}

and the orientation term

r_{h}

of FRPR-H, each scored alone; circles denote HUI360

\rightarrow

SSUP-A and squares SSUP-A

\rightarrow

HUI360, small markers are seeds and large markers their mean

Figure 5: Observed pose sequences and their exact score decomposition under the frozen seed-11 models. Four frames are sampled from each 32-frame input; time zero is the observation cutoff, not the interaction onset. Skeletons use the recorded keypoints, are centred on the bounding box and share a spatial scale within each sequence. Open blue dots mark the frozen reference logit

g

, filled green dots the final logit

z=g+r

; dashed lines mark the source-selected FRPR threshold. Scores are sigmoid outputs, not calibrated probabilities. Counts give the number of target tracks of the indicated class with the displayed residual sign at seed 11; “sign kept” counts the seeds, of five, under which the displayed track has the same residual sign. Cases (a, b, d) move towards their labels and (c) away from it; neither positive case crosses its threshold

Explore similar work

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities
Sep 7, 2026Raphael Lorenzo-Louis, Bertrand Luvison, Serena IvaldiHuman Behavior Prediction Human-Robot Interaction
Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France
HUI360: A 360° Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation
Aug 11, 2026Raphael Lorenzo-Louis, Fabio Amadio, Bertrand Luvison +1Robotic Data Collection Human-Robot Interaction
Inria, CNRS, UL, Loria, HUCEBOT, F-54600 Villers-les-Nancy, France · Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France
HumAIN: Human-Aware Implicit Social Robot Navigation
Jul 8, 2026Daeun Song, Nhat Le, Jeffrey Chen +7Social Robot Navigation Robot Motion Planning
Ewha Womans University, Korea. · George Mason University, USA. · University of Virginia, USA. +3

Sep 7, 2026, cs.CVJ/K move · Enter open · S save

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

Raphael Lorenzo-Louis, Bertrand Luvison, Serena Ivaldi

Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France

Human Behavior Prediction Human-Robot Interaction VLMs for Robotics

Open paper AskPDF arXiv

Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.

Abstract

Explore similar work

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

HUI360: A 360° Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation

HumAIN: Human-Aware Implicit Social Robot Navigation