Frame-Synchronous Hand Gesture Detection by Projected Winding Order

Amey Thakur

cs.CVSep 7, 2026

Frame-Synchronous Hand Gesture Detection by Projected Winding Order

Authors: Amey Thakur

Organizations: Independent Researcher Toronto, Canada

Abstract

Gesture recognition on video is normally posed as classification: label each frame, then act on the label. That is adequate for control, where a command may be obeyed several frames late without a user noticing, and inadequate for synchronisation, where an output must be aligned to the frame on which the gesture physically occurred. We take the synchronisation problem for one common movement, the rotation of an open hand about its long axis, and show that it admits an exact solution needing no classifier, no training data and no calibration. Let s be the normalised two-dimensional cross product of the two palm edges at the wrist and the outer knuckles, under the projection the camera already performs. We prove that s factorises as k(theta)cos(theta) with |k| > 0 everywhere, so s vanishes exactly when the palm is edge-on and its sign tracks the face presented to the camera. Detection is therefore a zero crossing of one scalar, which yields an instant rather than an interval, and we prove the criterion invariant to image mirroring, hand scale and handedness, all from its algebraic form rather than from the landmark estimator. A two-handed interaction uses four fingertips as a window onto a restyled version of the same scene. We give a coverage predicate that stays correct when the hands cross, where the usual triangulation does not, and derive each free parameter of the three stylisation operators from a noise statistic rather than by inspection. Against a corpus with exact ground truth the criterion detects 95% of flips with no false positive in 240 near-miss sequences, and places each within 6.7 ms on average of the instant it occurred, a sixth of the interval at which the hand is observed, against 23.0 ms for reporting the bracketing sample. Reporting an event more finely than one samples follows from treating a gesture as the zero of a continuous quantity rather than a label.

Figures & tables

Figure 1: The projected palm triangle at three rotations, computed rather than drawn. Left: palm toward the camera, positive winding. Centre: edge-on, the projected triangle has no area and $s=0$ , which is the trigger. Right: back toward the camera, winding reversed. The printed values are those the implementation reads for the model hand used in the figure.

Panel of Figure 2 — Figure 2: The pipeline. Inference (cyan) is rate limited and feeds a timeline anchored to the causal instant; rendering runs at the display rate. The encoder reads the composited surface (magenta), so the effect is present in the output as pixels and no editing stage exists.

Figure 3: Verification of the temporal buffer. The synthetic subject renders the elapsed time at which each frame was drawn, and moves, so a recalled frame is distinguishable from a recoloured current one. The strip runs left to right: one live frame at $4.3$ s, three recalled frames reading $2.5$ , $2.8$ and $2.9$ s, and one live frame after the effect at $5.6$ s. The live clock at each recalled frame is not printed on it, but lies between those two bounds and advances with the strip, so each recalled frame is about the requested $2.2$ s behind the moment it was shown.

Class	Requirement	Fired	Correct
Palm flip, quick, $180$ – $520$ ms	must fire	$54/60$	$90.0\%$
Palm flip, deliberate, $0.9$ – $1.6$ s	must fire	$60/60$	$100.0\%$
Held edge-on, $0.8$ – $1.4$ s	must not fire	$0/60$	$100.0\%$
Wobble to $54$ – $79$ degrees	must not fire	$0/60$	$100.0\%$
Rotating closed hand	must not fire	$0/60$	$100.0\%$
Entering frame mid-rotation	must not fire	$0/60$	$100.0\%$

Table 1: Classification over

60

generated sequences per class, at

f_{\Lambda}=24

Hz with the shipped thresholds. The two flip classes differ only in how quickly the hand is turned.

	Reported at $t_{a}$		Interpolated
	ms	frames	ms	frames
Mean absolute error	$23.0$	$0.55$	$6.7$	$0.16$
Median absolute error	$22.2$	$0.53$	$4.0$	$0.10$
Ninetieth percentile	$41.7$	$1.00$	$18.1$	$0.43$
Worst case	$66.3$	$1.59$	$26.7$	$0.64$
Mean signed error	$-22.1$	—	$+0.8$	—

Table 2: Localisation error on the

114

detected flips, reported at the bracketing sample and after the interpolation of Section 4.5 . The sampling interval is

41.7

ms. Both columns are computed by the published evaluation.

Explore similar work

May 22, 2026

Online Hand Gesture Recognition Using 3D Convolutional Neural Networks

In human computer interaction, real-time detection and classification of dynamic hand gestures is challenging as: 1) the system must run in a real-time video stream and there is no noticeable lag in response after performing a gesture; 2) there is a large difference in how people perform gestures, making recognition more difficult. In this paper, an online hand gesture recognition system is proposed, which is able to localize gestures in real-time video stream and recognize what these gestures are. To improve the robustness of the system, the sliding window approach is used to refine results from multiple windows. All of the models in my project are trained on Jester database, achieving 98+% accuracy for detector and 90+% accuracy for classifier. For the overall performance of the system, the best group can respond within three seconds and reach 37.5% Levenshtein accuracy on the homemade dataset. The project codes used in this work are publicly available.

Yinghao Qin, Tijana Timotijevic

School of Electronic Engineering and Computer Science Queen Mary, University of London London, UK

Gesture Recognition 3D CNNs

Ask PDF

Apr 14, 2026

Detecting Precise Hand Touch Moments in Egocentric Video

We address the challenging task of detecting the precise moment when hands make contact with objects in egocentric videos. This frame-level detection is crucial for augmented reality, human-computer interaction, assistive technologies, and robot learning applications, where contact onset signals action initiation or completion. Temporally precise detection is particularly challenging due to subtle hand motion variations near contact, frequent occlusions, fine-grained manipulation patterns, and the inherent motion dynamics of first-person perspectives. To tackle these challenges, we propose a Hand-informed Context Enhanced module (HiCE; pronounced `high-see') that leverages spatiotemporal features from hand regions and their surrounding context through cross-attention mechanisms, learning to identify potential contact patterns. Our approach is further refined with a grasp-aware loss and soft label that emphasizes hand pose patterns and movement dynamics characteristic of touch events, enabling the model to distinguish between near-contact and actual contact frames. We also introduce TouchMoment, an egocentric dataset containing 4,021 videos and 8,456 annotated contact moments spanning over one million frames. Experiments on TouchMoment show that, under a strict evaluation criterion that counts a prediction as correct only if it falls within a two-frame tolerance of the ground-truth moment, our method achieves substantial gains and outperforms state-of-the-art event-spotting baselines by 16.91% average precision.

Huy Anh Nguyen, Feras Dayoub, Minh Hoai

Australian Institute for Machine Learning, Adelaide University, Australia

Egocentric Video Understanding

Ask PDF

May 12, 2026

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

Luming Wang, Hao Shi, Jiajun Zhai +2

National Research Center for Optical Instrumentation, Zhejiang University, Hangzhou 310027, China · Ant Group Company Ltd., Hangzhou 310013, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China +1

Event-Based Vision 3D Hand Pose Estimation Gesture Recognition Stereo Vision Egocentric Vision

Ask PDF