HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI

Chengwen Zhang; Chun Yu; Borong Zhuang; Haopeng Jin; Qingyang Wan; Zhuojun Li; Zhe He; Zhoutong Ye

Figure 1. Demonstration of HiSync . Multiple people perform similar gestures at a distance of 34 m. The figure illustrates the application scenario: Person 1 (blue link) controls the quadruped, while Person 2 (orange link) simultaneously controls the drone. Other bystanders act as visual distractors. Each robot receives an inertial stream from its paired device. HiSync enables the robot to identify its bound command issuer, effectively rejecting distractors regardless of their kinematic similarity. HiSync scenario: two robots identify their bound human controllers among multiple people at 34 meters. On the left, a drone and a quadruped robot each connect via a color-coded inertial measurement unit data link (blue to Person_1, orange to Person_2) to a group of four people on the right who all raise one hand in similar gestures. Insets above each robot show likelihood tables from its inertial measurement unit over three candidate users; the quadruped’s table gives 99 percent to Person_1, while the drone’s gives 99 percent to Person_2, and the selected person in each table is outlined and marked with a check. Greyed human silhouettes behind the four foreground people represent visual distractors, and thought bubbles above Person_1 and Person_2 contain icons of the quadruped and drone, indicating which robot each user intends to control.

Figure 2. Visual Ambiguity at a Distance of 34 m. Figure (A) shows a real sample from our dataset (1920

\times

1080 resolution). The inset highlights that the hand region occupies fewer than 10

\times

10 pixels. Even with a manually zoomed-in view of the detection result like (B), YOLOv11x ( Jocher et al., 2023 ) fails to identify the hand. This demonstrates the inherent visual ambiguity in long-range interactions. This figure shows two images demonstrating visual ambiguity in recognizing hand gestures at long range. (A) A real sample from our dataset, captured at a resolution of 1920x1080, showing a hand gesture at a distance of 34 meters. (B) An example of incorrect hand recognition despite manual zoom-in, highlighting the difficulty of accurate gesture identification over long distances.

Work	Sensors	Data Modalities	Supported Distance
PTGAN ( Wei et al., 2018 )	Camera	RGB images	5–20 m
OSNet ( Zhou et al., 2019 )	Camera	RGB images	5–20 m
CASE-Net ( Li et al., 2019 )	Camera	Video	5–20 m
ABD-Net ( Chen et al., 2019 )	Camera	RGB images	5–20 m
AprilTag ( Olson, 2011 )	Camera + Fiducial Tag	Visual Markers	3–10 m
DeepArUco++ ( Berral-Soler et al., 2024 )	Camera + Fiducial Tag	Learned Fiducial Detection	3–10 m

Table 1. Representative works related to long-range gesture CSI. A table comparing related sensing works against the proposed HiSync method across sensors, data modalities, and distance. The columns list the Work name, Sensors used, Data Modalities, and Supported Distance. The rows are grouped into three categories: vision-only methods, hardware-specific approaches (such as Fiducial Tags, Ultra-Wideband, Global Positioning System, and Radar), and sensor fusion. While vision methods typically range from 5 to 20 meters and hardware solutions have specific constraints like needing base stations, the proposed HiSync method is highlighted in the final row. Utilizing Camera plus Inertial Measurement Unit (IMU) sensors with Optical Flow, HiSync demonstrates the widest supported distance of 3 to 34 meters, exceeding prior Camera-IMU works which are limited to under 10 meters.

Figure 3. Apparatus of Formative Study. (A) Three robot forms used in the study. (B) Example of a participant performing a gesture toward a quadruped robot. Apparatus of Formative Study. (A) Three types of robots used in the study: a drone, a quadruped robot, and a dual-arm robotic system. (B) An example of a participant performing a gesture toward the quadruped robot, demonstrating the setup used in the study for gesture-based interaction.

Figure 4. Command Vocabulary Proposed by Participants. This figure shows a Venn diagram representing the overlap of gesture commands proposed by participants. Each circle corresponds to a specific command (e.g., "Approach," "Retreat," "Left," "Right," "Ascend," "Descend," etc.), with areas of overlap indicating shared vocabulary across different users. The diagram visually depicts the commands related to drone control and general navigation, highlighting how different participants proposed similar or distinct gestures for specific actions.

Figure 5. Illustration of the User-defined Gesture Set. This figure shows the set of gestures proposed by participants for interacting with robots. The gestures are organized into rows by their corresponding actions: "approach," "summon," "ascend," "descend," "retreat," "left," and "right." Each action includes multiple variants (e.g., "approach-1," "ascend-1," etc.), demonstrating the diversity of user-defined hand movements for controlling robot behavior.

Figure 6. Gesture Amplitude vs. Interaction Distance. The color intensity represents the average amplitude. A general trend of motion amplification is observed as distance increases. Note: Drone-specific gestures (Ascend, Descend) were not evaluated at close ranges (3-5 m) due to safety constraints. Heatmap of gesture amplitude across interaction distances. A blue heatmap plots average gesture amplitude (color bar 34–45 centimeters) versus interaction distance (3–30 meters) for seven gestures on the vertical axis. Darker cells, especially for the Summon gesture, show that users increase motion amplitude at longer distances. Cells for Ascend and Descend at 3–5 meters are crossed out, indicating that these drone-specific gestures were not evaluated at close range.

Figure 7. Correlation between 3D Radial and 2D Tangential Motion. This figure includes two parts: Panel (a) is a line graph titled "Illustration of 3D Radial and 2D Tangential Motion" plotting motion magnitude over frame indices. An orange line representing tangential motion and a blue line representing radial motion follow nearly identical trajectories and peak simultaneously, illustrating their synchronization. A gray line plots the fluctuating ratio between them on a secondary y-axis. Panel (b) is a bar chart titled "Pearson Correlation Coefficient" displaying average correlation values for seven gesture types: Approach, Retreat, Left, Right, Summon, Ascend, and Descend. All categories exhibit high coefficients exceeding 0.92, with error bars indicating standard deviation, confirming the strong relationship between the two motion types across different gestures.

Figure 8. Visualization of data perspectives. The three camera angles—Upward, Eye-level, and Downward—are designed to emulate the perceptual perspectives of a quadruped, service robot, and drone, respectively. This figure shows three perspectives used in the study: (1) "Upward" view, showing participants walking in a hallway from below; (2) "Eye-level" view, capturing participants from a standard standing perspective in a corridor; (3) "Downward" view, showing participants from a high vantage point, looking down at a lower floor.

Figure 9. Distribution of Dataset. The chart visualizes the composition of 9,465 sequences, demonstrating a relatively uniform coverage across all dimensions to minimize bias. The rings, from outer to inner, represent gesture categories, camera perspectives, and interaction distances. This figure presents a circular diagram showing the distribution of the dataset across three factors: gesture, perspective, and distance. The outer ring represents gesture distribution, with categories like "Right," "Left," "Approach," "Retreat," and "No-Gesture." The middle ring shows perspective distribution, including "Upward," "Eye-level," and "Downward." The inner ring illustrates the distance distribution, with ranges from 3-5 meters to 25-34 meters. The total number of data points is 9,465, and each section’s proportion is indicated.

Figure 10. Dataset Characteristics. (Left) Recording apparatus with cameras and laser ranging, paired with visualizations of their data. The orange-framed inset displays the main industrial camera view used for experiments. (Right) Detailed sensor placement and biometrics. IMUs are positioned to simulate smart ring, handheld phone, and smartwatch form factors, alongside anthropometric measurements collected. Recording setup and sensor placement for the HiSync gesture dataset. The left panel shows a multi-sensor rig on a tripod with an industrial camera, laser rangefinder, and smartphone, plus an inset of the captured hallway view and a distant participant at the target location. The right panel zooms in on a participant’s extended arm and hand, with multiple inertial measurement units attached to the index finger, wrist, and hand to emulate different wearable form factors, alongside annotated body segment lengths for palm, index finger, forearm, and upper arm.

Figure 11. Overview of HiSync. The framework fuses robot-mounted RGB video and wearable IMU signals in the frequency domain for robust Command Source Identification (CSI). It consists of two main stages: (1) a Motion Feature Extractor that estimates and transforms visual velocity and inertial acceleration into spectral representations; and (2) CSINet, which aligns and matches these cross-modal features using Quality-Aware Feature Modulation, IMU-Anchored Attention, and Scale-Aware Multi-Window Fusion to identify the target operator. Block diagram of HiSync pipeline from camera and IMU signals to command source identification. Camera video and wearable inertial data enter a Motion Feature Extractor, where hand regions are tracked with dense optical flow and inertial accelerations are estimated, then both streams are transformed into spectral motion features. These spectral features, organized over multiple temporal windows for each hand and inertial unit, are processed by the Command Source Identification Network—combining scale-aware multi-window fusion, quality-aware feature modulation, IMU-anchored cross-modal attention, and a similarity head—to predict the robot’s target operator.

Figure 12. Architecture of Command Source Identification Network. Taking spectral features as input, for each window size, the network first applies Quality-Aware Feature Modulation to suppress noise in IMU signals. It then utilizes IMU-Anchored Cross-Modal Attention to align visual flow sequences with the inertial reference. Finally, the Similarity Head computes cosine scores, which are dynamically aggregated by Scale-Aware Multi-Window Fusion across multiple window sizes to output the final similarity. Architecture diagram of the Command Source Identification Network detailing the multi-stage feature processing pipeline. On the left, stacked input cards represent parallel processing for different window sizes (e.g., 5 and 20), containing Inertial Measurement Unit (IMU) and Flow Power Spectral Densities (PSD) and features. The central processing flow consists of three modules: first, "Quality-Aware Feature Modulation" passes IMU PSD through linear layers and a Leaky Rectified Linear Unit to generate affine parameters (

\gamma

and

\beta

) for signal denoising. Second, "IMU-Anchored Cross-Modal Attention" uses the modulated IMU data as Queries and Flow PSD as Keys and Values within a Multi-Head Attention mechanism to align the modalities. Third, a "Similarity Head" applies

l_2

normalization to the features and computes a cosine similarity score. Finally, a "Scale-Aware Multi-Window Fusion" module on the bottom left aggregates these scores using a Multi-Layer Perceptron and softmax weighting to produce the final target.

Figure 13. Results of Experiments. This figure shows the CSI Accuracy vs. Distance. The dashed line represents the data obtained directly from the referenced paper. The results show that HiSync outperform baselines, especially at long distances. Error bar denotes standard deviation. This figure presents the accuracy of Command Source Identification (CSI) across different distances from 3 to 34 meters, comparing several models. The curves show the performance of different methods: DNN (red), URGR (green), VIPL (blue), and the proposed method (purple). The dashed lines indicate results from the referenced papers, with “DNN*,” “URGR*,” and “VIPL*” marked as directly taken from previous works. The "Random" baseline is also shown as a horizontal reference line. The CSI accuracy decreases as distance increases, with the proposed method (OURS) performing better than others at longer ranges.

Figure 14. Failure Case. This is a failure sequence where the two persons move in near-perfect sync. However, because the HiSync can capture subtle differences in movements, the failure rate is very low even in such a rare case. This figure shows a failure scenario where two participants (Person 1 and Person 2) move in near-perfect synchronization. The frames capture different moments (Frames 089, 104, 121, 146), with corresponding similarity scores between the two individuals. Despite the near-identical movement, the system struggles to distinguish between them, as indicated by the similarity values (ranging from 0.41 to 0.94). The overall similarity score is shown at the top right. However, because the HiSync can capture subtle differences in movements, the failure rate is very low even in such a rare case.

Figure 15. Ablation Results. We compare the full system against ablated variants across distances. The significant performance drop in incomplete models highlights the necessity of each component for maintaining robust CSI. Error bar denotes standard deviation. This figure shows the performance of different models for Command Source Identification (CSI) across various distances (3-34 meters). The curves represent four models: "Full" (purple), "Skeleton" (gray), "-FiLM" (blue), and "-Multi-window" (green). CSI accuracy decreases as distance increases, with the "Full" model maintaining the highest accuracy. The figure also illustrates how removing certain components (e.g., FiLM or Multi-window) reduces performance, particularly at longer distances.

Figure 16. Impact of Window Size on Accuracy. This figure shows a clear trend where optimal window size increases with distance. This validates our design choice to fuse the different temporal window sizes. This figure displays the Command Source Identification (CSI) accuracy across various window sizes (1-16 frames) and distances (3-34 meters). The curves show how CSI accuracy changes with different window sizes for each distance range, with the peak points marked by triangles. The results indicate that CSI accuracy increases with larger window sizes, with optimal performance achieved at specific window sizes for each distance range. The accuracy decreases as the distance increases, with the 3-5m range showing the highest accuracy.

Methods	Accuracy(%) $\pm$ SD
Methods	$T_{clean}$	$T_{1}$	$T_{2}$	$T_{3}$
HiSync	97.01( $\pm 0.26$ )	96.47( $\pm 0.43$ )	95.70( $\pm 0.95$ )	95.14( $\pm 0.97$ )
w/o Spectral	76.83( $\pm 0.60$ )	65.19( $\pm 0.64$ )	59.16( $\pm 3.11$ )	50.38( $\pm 6.82$ )
Linear ( Furgale et al., 2013 )	78.18( $\pm 1.17$ )	73.38( $\pm 6.37$ )	65.81( $\pm 3.32$ )	51.82( $\pm 5.98$ )
XCorr ( Knapp and Carter, 2003 )	79.11( $\pm 0.58$ )	74.95( $\pm 6.79$ )	63.82( $\pm 3.36$ )	53.27( $\pm 5.91$ )
DTW ( Berndt and Clifford, 1994 )	77.98( $\pm 1.04$ )	74.91( $\pm 1.32$ )	72.09( $\pm 1.62$ )	64.74( $\pm 4.32$ )

Table 2. Evaluation Results on Temporal Misalignment. We compare HiSync against the time-domain ablation ("w/o Spectral"), pre-alignment methods (Linear, XCorr, DTW) applied to the time-domain backbone, and the VIPL baseline. Temporal misalignment increases progressively from

\mathcal{T}_{clean}

to

\mathcal{T}_{3}

. Results demonstrate that HiSync maintains high robustness even under severe synchronization noise (

\mathcal{T}_{3}

) where time-domain methods degrade sharply. Table comparing command source identification accuracy of HiSync and baselines under increasing temporal misalignment. Across all conditions T_\text{clean} to T_3, HiSync achieves the highest accuracy, starting near 97% and remaining above 95% even at T_3. The ablation without spectral features and time-domain baselines (Linear, XCorr, DTW) begin around 77–79% at T_\text{clean} and drop to roughly 50–65% at T_3. The VIPL baseline performs worst overall, with accuracies consistently below the other methods.

Figure 17. Real Robot Apparatus. The real-robot experiment is conducted on a Unitree Go2 quadruped robot, equipped with a Realsense D435 camera. The onboard microphone and official joystick are used in the Voice Control and Remote Controller baseline respectively. Overview of the real-robot setup with Unitree Go2, camera, microphone, and handheld remote controller joystick. A Unitree Go2 quadruped robot stands indoors on a flat floor, facing slightly left. A depth camera is mounted at the front of the robot’s head, and a small microphone is positioned on the front side of the body. In front of the robot on the ground lies a handheld joystick-style remote controller. Text labels point to the camera, microphone, and remote controller joystick to highlight the hardware used for sensing and teleoperation baselines.

Table 3. Quantitative Results of Real-World Experiments. We report the mean CSI accuracy and response time (from gesture onset to robot reaction) under adversarial settings with strong mimics. Data are presented as Mean

\pm

Standard Error (SE). Real-world command source identification accuracy and response time across distance and gesture. Across interaction distances from 3 to 30 meters, Command Source Identification accuracy stays high (about 91–96 percent) with only a slight drop at 30 meters, while response time remains around 3.1–3.9 seconds. Across gesture categories, accuracy is similarly stable (about 91–96 percent), with Left and Summon slightly higher than Right, Approach, and Retreat, and response times again clustered near 3.2–3.5 seconds, indicating robust and consistent performance over distance and gesture type.

Figure 18. Objective Results of Real Robot Experiment. Mean subjective scores with standard error for the three interaction methods. Statistical significance is marked based on the Wilcoxon Signed-Rank Test. Note: Scores for NASA-TLX subscales are inverted, meaning higher values consistently represent a better user experience across all indicators. Overall, HiSync consistently outperforms both baselines across all metrics, demonstrating superior usability and significantly reduced user workload. Aggregated subjective user scores for HiSync and a remote controller across distance. A line chart plots score versus distance from 3 to 30 meters, with HiSync as a higher curve above the remote controller at all distances. Both curves gently decline as distance increases. Each line is surrounded by a shaded band indicating standard error, showing that HiSync’s advantage is consistent over distance.

Figure 19. Aggregated subjective scores as a function of distance. This figure illustrates the average of SUS and NASA-TLX results. The higher the score, the better the system. HiSync maintains a significant (

p<0.0007

) advantage over the Remote Controller. Shaded areas denote standard error. Subjective user ratings for three interaction methods in the real robot experiment. A grouped bar chart compares Voice, HiSync, and Remote Controller across nine metrics (Willing to Use, Easy to Use, Easy to Learn, Confidence, Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration). HiSync bars are consistently the tallest on all metrics, indicating better usability and lower workload than the two baselines. Voice scores lowest overall, while the Remote Controller lies between Voice and HiSync. Error bars show standard error, and significance markers above the groups indicate that many of HiSync’s improvements over the baselines are statistically significant.

Figure 20. Applications. (A) Service-robot interactions at a mall. (B) Summoning a robotaxi. (C) “Hi and Sync” paradigm. This figure illustrates various real-world applications of the system. (A) Service-robot interactions at a mall, where the customer uses a gesture to summon a service robot from 34 meters away. The CSI (Command Source Identification) system matches the customer’s gesture to the robot. (B) Summoning a RoboTaxi at a taxi pickup area using a gesture ("Approach"), with the system matching the customer’s gesture and confirming the selection. (C) The "Hi and Sync" paradigm, where an operator uses a gesture to select Robot A (from multiple robots), demonstrating the system's ability to identify and engage specific robots based on the operator’s movement.

Figure 21. Illustration of VideoFlow Failure Case. At this extreme distance, human movements have become so minute that they are obscured by noise. This figure demonstrates a failure scenario in the VideoFlow. The left image shows a normal scene captured from a camera, while the right image illustrates the result after processing, where the output is heavily pixelated.

Figure 22. CSI Accuracy vs. IMU Position. While the Palm and Wrist provide slightly stronger motion cues than the Ring, the system achieves high accuracy (

>93\%

), validating the system’s usability across diverse wearable form factors. Error bar denotes standard deviation. This figure shows the Command Source Identification (CSI) accuracy for different IMU positions: Ring, Palm, and Wrist. The bar chart illustrates the accuracy percentages, with the Palm position yielding the highest CSI accuracy (~98%) and the Ring and Wrist positions showing slightly lower but similar results.

Figure 23. Gesture Amplitude vs. Time. Under non-response conditions, participants gradually amplify their gesture amplitude, suggesting a natural "graceful escalation" strategy for interaction recovery. Error bar shows standard deviation. This figure shows a boxplot of gesture amplitude (in centimeters) over time (1-5 seconds). The x-axis represents time, while the y-axis shows the amplitude, with the boxes displaying the interquartile range, and the lines indicating the range of values. The dotted line illustrates the general trend, showing an increase in amplitude with time. Each time point (1 to 5 seconds) shows a distinct distribution of gesture amplitudes, suggesting that gesture strength varies with duration.

HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI

Abstract

Figures & tables

Appendix

Human-Robot Interaction

IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusion

RoSHI: A Versatile Robot-oriented Suit for Human Data In-the-Wild

World2HOSI: A Multi-Expert Harness for Long-Horizon Human-Object-Scene Interaction Generation

Abstract

Human-Robot Interaction

Explore similar work

IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusion

RoSHI: A Versatile Robot-oriented Suit for Human Data In-the-Wild

World2HOSI: A Multi-Expert Harness for Long-Horizon Human-Object-Scene Interaction Generation