Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation

Bowen Yang; Xinliang Xiao; Wenjing Zhang; Li Yang; Wei Zhou

cs.ROOct 8, 2026

Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation

Authors: Bowen Yang, Xinliang Xiao, Wenjing Zhang, Li Yang, Wei Zhou

Organizations: School of Automation, Nanjing University of Science and Technology, Nanjing, 210094, China.

Abstract

Social and service robots in public spaces need to anticipate which nearby person is about to approach and touch them, so that a response can be prepared before contact. It is largely unknown which cues support this anticipation when a model trained with one robot is used on another robot at a different site. We study this question with a fixed-reference pose residual (FRPR) model: a geometry predictor built from the person's bounding box and mask is trained and frozen, and a temporal network then learns from body pose an additive correction to its logit, so that every prediction splits exactly into a geometry term and a pose term. Between two public egocentric datasets recorded by different robots, HUI360 and SSUP-A, with every choice made on source data, the pose correction raised average precision (AP) from 0.277 to 0.321 from SSUP-A to HUI360 and gave no measurable gain in the opposite direction; the same asymmetry held over a stronger, source-selected geometry reference. Freezing gave no AP advantage over joint training, and simple geometric baselines and tree ensembles remained competitive or better, so the construction serves measurement rather than prediction accuracy. A head-orientation residual added small gains in both directions. Post hoc, whether a person faces the camera kept its discriminative direction across datasets, whereas head pitch reversed. With thresholds chosen on source data, the neural models that use geometry detected at most 17% of target interactions. Code and processed data are available at https://github.com/WeiZhou96/FRPR-interaction-anticipation.

Figures & tables

Figure 1: Fixed-reference pose residual (FRPR) and its head-orientation extension (FRPR-H). (a) A 32-frame track at 15 fps yields a geometry summary $a$ , a box-normalised pose sequence $q$ and, for FRPR-H only, a head-orientation summary $h$ from dense face keypoints; all inputs are standardised with source-training statistics. (b1) Stage 1 trains the geometry multilayer perceptron (MLP), whose logit $g=g_{\theta^{*}}(a)$ gives the stand-alone reference score $p_{G}=\sigma(g)$ ; it remains frozen in Stages 2 and 3. (b2) Stage 2 trains a causal temporal encoder with a zero-initialised linear head, producing the pose residual $r=r_{\phi}(q)$ . (b3) Stage 3, used by FRPR-H only, trains a small MLP with a zero-initialised output on $h$ while $g$ and $r$ stay frozen. (c) The logits are summed before the sigmoid, so each score splits exactly into the terms shown. Frames and crops are taken from the public HUI360 demonstration material; the coloured feature cells are schematic

		HUI360 $\rightarrow$ SSUP-A				SSUP-A $\rightarrow$ HUI360
Method	Input	AP $\uparrow$	AUROC $\uparrow$	Recall $\uparrow$	FPR $\downarrow$	AP $\uparrow$	AUROC $\uparrow$	Recall $\uparrow$	FPR $\downarrow$
Last-frame box area a	$a$	0.178	0.858	0.61	0.133	0.341	0.732	0.01	0.012
Geometry reference	$a$	0.129 $\pm$ 0.014	0.841	0.03	0.004	0.277 $\pm$ 0.022	0.733	0.00	0.006
Random forest	$a$	0.146 $\pm$ 0.006	0.860	0.19	0.028	0.369 $\pm$ 0.011	0.738	0.15	0.031
HistGB b	$a$	0.161	0.841	0.19	0.022	0.311	0.688	0.19	0.047
Pose TCN	$q$	0.048 $\pm$ 0.002	0.652	0.21	0.119	0.334 $\pm$ 0.009	0.757	0.21	0.080

Table 1: Target-domain results under source-only model selection. AP and AUROC are means over five training seeds (

\pm

standard deviation of AP); recall and false-positive rate (FPR) are taken at the threshold that maximises

F_{1}

on source validation data.

\bar{q}

: summary statistics of the pose sequence;

h

k

: head-orientation and hand summaries. The last block contains the orientation extension and HistGB with

h

appended, each scored once after its own lock. Bold and underline: highest and second-highest AP per direction; shading: FRPR and FRPR-H. AP is not comparable between directions because prevalence differs (0.030 and 0.167)

Figure 2: Paired AP differences between FRPR and each comparator on the target domain. Thick bars are 95% percentile intervals from a bootstrap over recording dates and training seeds. Squares mark the four primary comparisons; for these only, thin bars give the 98.75% intervals of the Bonferroni family. Circles mark exploratory comparisons with unadjusted 95% intervals; filled markers indicate intervals that exclude zero. Ticks under the first row are the differences obtained when one target date is removed. Horizontal scales differ between panels; all values are listed in Online Resource 1 (Table S3)

Figure 3: The two terms of FRPR on target data. (a, b) Reference logit against pose correction for every target sample (means over five seeds; 1500 randomly drawn non-interacting samples are shown in (a)), with the marginal distribution of the correction by class; dotted lines and printed values are class medians. (c) AUROC of each term scored alone, on source validation dates and on the target. In HUI360 $\rightarrow$ SSUP-A the reference keeps most of its ranking quality and the correction does not; in SSUP-A $\rightarrow$ HUI360 the correction keeps its level while the reference drops and has the larger logit spread (note the different axis ranges in (b))

Figure 4: Transfer of the head-orientation cues and of the terms of FRPR-H. (a, b) AUROC of each entry of

h

used alone as a score, on the source pool (open circles) and on the target set (filled circles); values below 0.5 mean that lower values indicate interaction. Colour and line style mark whether source and target lie on the same side of 0.5 (blue, solid), on opposite sides (orange, dashed), or both within 0.45–0.55 (grey, dotted); target values at least 0.15 from 0.5 are printed. Computed after the locked target scoring; post hoc and descriptive. (c) Target AUROC of the reference logit

g

, the pose term

r_{q}

and the orientation term

r_{h}

of FRPR-H, each scored alone; circles denote HUI360

\rightarrow

SSUP-A and squares SSUP-A

\rightarrow

HUI360, small markers are seeds and large markers their mean

Figure 5: Observed pose sequences and their exact score decomposition under the frozen seed-11 models. Four frames are sampled from each 32-frame input; time zero is the observation cutoff, not the interaction onset. Skeletons use the recorded keypoints, are centred on the bounding box and share a spatial scale within each sequence. Open blue dots mark the frozen reference logit

g

, filled green dots the final logit

z=g+r

; dashed lines mark the source-selected FRPR threshold. Scores are sigmoid outputs, not calibrated probabilities. Counts give the number of target tracks of the indicated class with the displayed residual sign at seed 11; “sign kept” counts the seeds, of five, under which the displayed track has the same residual sign. Cases (a, b, d) move towards their labels and (c) away from it; neither positive case crosses its threshold

Explore similar work

Sep 7, 2026cs.CV

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.

Raphael Lorenzo-Louis, Bertrand Luvison, Serena Ivaldi

Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France

Human Behavior Prediction Human-Robot Interaction VLMs for Robotics

Ask PDF

Aug 11, 2026cs.CV

HUI360: A 360° Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation

As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.

Raphael Lorenzo-Louis, Fabio Amadio, Bertrand Luvison +1

Inria, CNRS, UL, Loria, HUCEBOT, F-54600 Villers-les-Nancy, France · Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France

Robotic Data Collection Human-Robot Interaction

Ask PDF

Jul 8, 2026cs.RO

HumAIN: Human-Aware Implicit Social Robot Navigation

Effective social robot navigation requires sensitivity to human behavior, often revealed through subtle skeletal cues like gait and orientation. We present Human-Aware Implicit Social Robot Navigation (HumAIN), a novel framework that fuses implicit social cues directly into the planning loop via knowledge distillation. We first employ a transformer-based teacher model that fuses rich multi-modal inputs, including historic images, skeletal keypoints, robot state, and a robot's target goal, to learn robust, human-aware representations for the robot's future trajectory planning. To enable real-time deployment, we then distill this knowledge into a lightweight student model. By optimizing for both trajectory reconstruction and latent feature alignment with the teacher, the student learns to infer complex social dynamics from minimal inputs. Bridging the prediction-planning gap with an efficient distilled architecture, our method enables robots to reason about human behavior in a manner that is adaptive, robust, and socially compliant. We validate HumAIN through extensive experiments, where it improves trajectory prediction metrics by an average of 29.8% across all metrics compared to state-of-the-art baselines. These results highlight the benefit of using implicit, whole-body cues to achieve human-like navigation awareness on resource-constrained platforms.

Daeun Song, Nhat Le, Jeffrey Chen +7

Ewha Womans University, Korea. · George Mason University, USA. · University of Virginia, USA. +3

Social Robot Navigation Robot Motion Planning Social Robotics

Ask PDF