Scale-Gest: Scalable Model-Space Synthesis and Runtime Selection for On-Device Gesture Detection

Abdul Basit; Saim Rehman; Muhammad Shafique

cs.CVMar 16, 2026

Scale-Gest: Scalable Model-Space Synthesis and Runtime Selection for On-Device Gesture Detection

Authors: Abdul Basit, Saim Rehman, Muhammad Shafique

Organizations: New York University (NYU) Abu Dhabi, Abu Dhabi, UAE

Abstract

Realizing on-device ML-based gesture detection under tight real-time performance, energy and memory constraints is challenging, especially when considering mobile devices with varying battery-power levels. Existing EdgeAI deployments typically rely on a single fixed detector, limiting optimization opportunities. We present Scale-Gest, a novel run-time adaptive gesture detection framework that expands the detector space into a dense family of tiny-YOLO architectures. We introduce multiple novel device-calibrated ACE (Accuracy-Complexity-Energy) profiles by analyzing different model-resolution-stride operating points. A lightweight run-time controller selects an appropriate ACE mode under user-defined and battery constraints, while a motion-aware hand-gesture-tracking ROI gate crops the input for reduced complexity detection. To evaluate performance of our system in real-world car driving scenarios, we introduce a temporally-annotated Driver Simulated Gesture (DSG-18) dataset. Scale-Gest maintains event-level F1 while significantly reducing energy and latency compared to single-detector approaches. On a battery-powered laptop running gesture streams, our ACE controller reduces per-frame energy by 4x (from 6.9 mJ to 1.6 mJ) while maintaining high gesture-detection performance (event-level F1 = 0.8-0.9) and low mean latency (6 ms).

Figures & tables

Framework	Dyn. Res.	Dyn. Stride	Model	Energy-	Event-	Gesture
		(FPS)	complex.	aware	temp.	task
BranchyNet ( Teerapittayanon et al., 2016 )	✗	✗	✗	✗	✗	✗
SkipNet ( Wang et al., 2018 )	✗	✗	✗	✗	✗	✗
Deep Feature Flow ( Zhu et al., 2016 )	✗	✓	✗	✗	✗	✗
AdaScale ( Chin et al., 2019 )	✓	✗	✗	✗	✗	✗
DyRA ( Seo et al., 2024 )	✓	✗	✗	✗	✗	✗

Table 1. Comparison of adaptive frameworks with Scale-Gest.

Figure 1. Temporal and spatial sparsity of driver gestures. (A) A typical DSG-18 gesture episode lasting only 10 frames, surrounded by long background periods, continuous high-fidelity inference is therefore wasteful. (B) A Kalman-based hand tracker crops the detector input to a small ROI, reducing effective input size.

Figure 2. Device-calibrated Accuracy–Energy–Complexity (ACE) trade-offs. (A) Baseline YOLO m/s/n variants on a Jetson-class device show diminishing mAP@0.5 gains as average power increases on HaGRID. (B) Shows ACE profiles for YOLOv12 across scales on DSG-18 dataset by profiling different resolutions and strides, exposing several practical operating points. — Figure 2. Device-calibrated Accuracy–Energy–Complexity (ACE) trade-offs. (A) Baseline YOLO m/s/n variants on a Jetson-class device show diminishing [email protected] gains as average power increases on HaGRID. (B) Shows ACE profiles for YOLOv12 across scales on DSG-18 dataset by profiling different resolutions and strides, exposing several practical operating points.

Figure 3. Scale-Gest Methodology: Outlines Components to realize the run-time adaptive framework. After Model Space Sampling, the training of each model family is performed on the HaGRID dataset, and similarly for run-time evaluation the DSG-18 dataset is curated and benchmarked to generate the model ACE profiles, which are used by the run-time selector for dynamically changing models during run time.

Panel of Figure 4 — Figure 4. Model synthesis flowchart

Figure 5. Accuracy–efficiency comparison of YOLOv8, v10 synthesized families.

Figure 6. DSG-18 dataset generation & post-processing pipeline.

Figure 7. Dataset Class Distribution: Comparison of sample counts between HaGRID (Train/Valid) and total DSG-18 frames per gesture class.

Figure 8. Concatenated gesture timeline for DSG-18 ( $\approx$ 3555 s). Color encodes gesture class. The sequence reveals temporally sparse and irregular gesture bursts.

Figure 9. ACE surfaces for micro-scale YOLOv8–12, showing the trade-off between normalized accuracy, complexity, and energy across resolutions and temporal strides; surfaces close to the bottom edge correspond to configurations with lower compute complexity, energy, and high accuracy ACE profiles.

Model	Rec.	Prec.	AP 50	Infer (ms)	Energy (J)
YOLOv12m, full-frame (SOTA)	0.996	0.963	0.993	65.3	703
YOLOv12m, KF-ROI (x1.8)	0.995	0.981	0.992	49.2	355
YOLOv12m, KF-ROI (x1.5)	0.989	0.979	0.988	48.0	351
YOLOv12-solar, full-frame	0.980	0.975	0.978	34.7	205
YOLOv12-solar, KF-ROI (x1.8)	0.980	0.986	0.978	27.7	171
YOLOv12-solar, KF-ROI (x1.5)	0.974	0.986	0.973	26.8	160

Table 2. Runtime and energy–accuracy comparison of full-frame vs. KF-ROI across YOLOv12m and micro-scale variants solar and mercury (v12 solar:

(\alpha,\beta,c_{\max},\text{heads})=(0.25,0.125,320,\text{P3})

, v8 mercury:

(0.18,0.15,192,\text{P3})

Figure 10. Real-time ACE validation on a battery-powered laptop.

Figure 11. Dynamic ACE-aware Pareto strips for the top five tiers selected under four run-time scenarios.

Gesture Recognition

Gesture recognition maps body, hand, or muscle movements to communicative or control actions using cameras and wearable or wireless sensors.

See all 56 papers

TinyGaze: Lightweight Gaze-Gesture Recognition on Commodity Mobile Devices
Mar 30, 2026Yaxiong Lei, Hyochan Cho, Fergus Buchanan +4Gesture Recognition On-Device Inference
University of St Andrews, St Andrews, UK · University of Essex, Colchester, UK · King’s College London, London, UK
Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices
May 13, 2026Pietro Bartoli, Christian Veronesi, Tommaso Bondini +2Human Activity Recognition Tiny ML
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB) Politecnico di Milano, Milan, Italy
LiteEvent-AE: Lightweight Autoencoder for Event-Based Vision on Low-Latency Energy-Constrained Edge Devices
Aug 22, 2026Riadul Islam, Joey Mule, Dhandeep Challagundla +3Event-Based Vision Autoencoders
Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County, MD 21250, USA

Mar 30, 2026J/K move · Enter open · S save

TinyGaze: Lightweight Gaze-Gesture Recognition on Commodity Mobile Devices

Yaxiong Lei, Hyochan Cho, Fergus Buchanan, Shijing He+3

University of St Andrews, St Andrews, UK · University of Essex, Colchester, UK · King’s College London, London, UK

Gesture Recognition On-Device Inference

Open paper AskPDF arXiv

Gaze gestures can provide hands free input on mobile devices, but practical use requires (i) gestures users can learn and recall and (ii) recognition models that are efficient enough for on-device deployment. We present an end-to-end pipeline using commodity ARKit head/eye transforms and a scaffolded guidance-to-recall protocol grounded in learning theory. In a pilot feasibility study (N=4 participants; 240 trials; controlled single-session setting), we benchmark a compact time-series model (TinyHAR) against deeper baselines (DeepConvLSTM, SA-HAR) on 5-way gesture recognition and 4-way user identification. TinyHAR achieves strong performance in this pilot benchmark (Macro F1 = 0.960 for gesture recognition; Macro F1 = 0.997 for user identification) while using only 46k parameters. A modality analysis further indicates that head pose dynamics are highly informative for mobile gaze gestures, highlighting embodied head--eye coordination as a key design consideration. Although the small sample size and controlled setting limit generalizability, these results indicate a potential direction for further investigation into on-device gaze gesture recognition.

Abstract

Gesture Recognition

Explore similar work

TinyGaze: Lightweight Gaze-Gesture Recognition on Commodity Mobile Devices

Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices

LiteEvent-AE: Lightweight Autoencoder for Event-Based Vision on Low-Latency Energy-Constrained Edge Devices