OpenZilo: An Open Smart-Ring Platform for Human-Centric Sensing and Interaction with Robots
Authors: Qianggang Ding, Xingyao Wang, Rui Feng, Chi Qin, Weijian Lin, Lizhi Ding, Jiankai Tang, Yuntao Wang, Xiangke Liu, Bang Liu
Organizations: Université de Montréal · Mila – Quebec Artificial Intelligence Institute · Zilo Lab · Southeast University · Nanjing Medical University · Dalian University of Technology · Tsinghua University · DeepMirror
Wearable robot interfaces must connect user input to timely physical action. We present the proposed platform, an open smart-ring platform for studying this connection across task stages. The ring integrates a microphone, an inertial measurement unit, photoplethysmography, and skin-temperature sensing. Its software architecture separates acquisition, event interpretation, interaction policy, and robot execution. We characterize seven operating modes and evaluate speech and gesture control with 10 participants. Across five commands and 250 trials per modality, speech and gestures achieved task success rates of 90.4% and 88.8%, respectively. Recognition accuracy exceeded task success for both modalities. Gaussian hidden Markov models achieved higher accuracy than a support vector machine across three vocabulary sizes, with longer scoring times. Following a shared spoken task start, gesture-based adjustments had shorter median response times than speech. Whole-task success was 70.0% with gestures and 64.0% with speech. We further examine physiological-event-triggered robot interaction using simulated event notifications. Mean notification-receipt-to-arrival times were 11.71 s for the robot and 9.20 s for an on-site person. These descriptive findings support evaluating wearable robot interfaces through both recognition and physical outcomes, with timing measures appropriate to each task stage.
Figures & tables
Figure 1: Overview of the proposed platform and its robot-interaction scenarios. The upper panels illustrate basic commands, online adjustments, and physiological-event-triggered interaction. The lower panels show (a) the four sensing modalities, (b) the SDK, application agent, and robot adapter interfaces, and (c) interaction scenarios corresponding to RQ1–RQ3. The physiological-event branch illustrates the interaction concept; RQ3 evaluates local-arrival responses to simulated physiological-event notifications.
Figure 2: Design and implementation of OpenZilo. (a) Exploded view showing the button, ring housing, circuit board, curved battery, and housing cover. (b) Component layout on both sides of the circuit board, including the MCU, two microphones, PPG sensor, IMU, temperature sensor, NOR flash, and button. (d) OpenZilo worn on the finger. (e) Inner view of the assembled ring. (f) Photographs of both sides of the fabricated circuit board. The ring integrates speech, inertial, optical physiological, and temperature sensing within a compact wearable form factor.
Component
Part number / Specification
Qty.
MCU
Nordic nRF54L15-CAAA
1
IMU
TDK ICM-42688-P
1
PPG sensor
ADI MAXM86161
1
Microphone
Knowles SPH0641LU4H-1
2
Temperature sensor
TI TMP119
1
NOR flash
128 Mbit
1
Table 1: Component selection for the OpenZilo prototype.
Figure 3: Hardware architecture design schematic. The labeled nRF54L15 controller coordinates microphone, IMU, PPG, and temperature channels with nonvolatile storage, power management, button and LED feedback, and wireless transmission. The PPG inset details photodiode bias, ambient-light cancellation, and a current-mode analog-to-digital converter (ADC). The architecture separates wearable acquisition from host processing and application logic; the downstream AI block may run at the edge or in a cloud service according to the application deployment.
Figure 4: Host-side HMM reference pipeline for gesture training and recognition. (a) Offline training processes labeled IMU recordings into window-level features to train gesture-specific Gaussian hidden Markov models (HMMs), which are exported together with preprocessing and recognition settings. (b) Online recognition segments live IMU signals, extracts features, and applies HMM-based scoring and rejection criteria to generate gesture events for the application agent.
Operating mode
Current
Test
BLE disconnected
59–61 μ A
320–331 h
BLE connected
54–55 μ A
355–361 h
IMU only
1.12 mA
17.41 h
Real-time recording
3.88 mA
5.03 h
Non-real-time recording
4.10 mA
4.76 h
IMU + real-time recording
4.63 mA
4.21 h
Table 2: Reported current draw and idealized estimates for a nominal 19.5 mAh battery; these are not measured depletion lifetimes.
Figure 5: Robot-control setup and interaction examples for the proposed platform. (a) Wheeled quadruped robot. (b) Close-up of the ring-based control setup and robot-mounted hardware. (d) Speech control of the quadruped through the ring microphone. (e) IMU-based gesture control of the quadruped. (c) and (f) Extension of the proposed platform to a humanoid robot.
Speech command
Correct trials
Latency (s)
Wrong actions
Recognition
Task success
Forward
48/50 (96.0%)
46/50 (92.0%)
1.37 [1.13, 1.68]
1
Backward
47/50 (94.0%)
45/50 (90.0%)
1.54 [1.26, 1.87]
2
Turn left
46/50 (92.0%)
43/50 (86.0%)
1.61 [1.32, 1.94]
2
Turn right
47/50 (94.0%)
44/50 (88.0%)
1.48 [1.21, 1.79]
2
Stop
49/50 (98.0%)
48/50 (96.0%)
1.32 [1.09, 1.59]
0
Table 3: Speech-only command recognition and execution over 50 trials per command, pooled across 10 participants. Recognition accuracy and task success are reported as correct recognitions and successful tasks, respectively, divided by total trials (%). Latency starts at speech offset and ends at action onset for movement commands or verified stationary state for STOP. For successful trials, latencies are reported as medians, with the 25th and 75th percentiles in brackets. Wrong actions are unintended executions.
Gesture command
Correct trials
Latency (s)
Wrong actions
Recognition
Task success
Forward
46/50 (92.0%)
44/50 (88.0%)
0.56 [0.41, 0.73]
1
Backward
45/50 (90.0%)
43/50 (86.0%)
0.62 [0.46, 0.82]
2
Turn left
46/50 (92.0%)
44/50 (88.0%)
0.71 [0.54, 0.92]
2
Turn right
47/50 (94.0%)
45/50 (90.0%)
0.67 [0.50, 0.87]
2
Stop
47/50 (94.0%)
46/50 (92.0%)
0.49 [0.36, 0.65]
1
Table 4: Gesture-only command recognition and execution over 50 trials per command, pooled across 10 participants under a fixed wearing and calibration configuration. Recognition accuracy and task success are reported as correct recognitions and successful tasks, respectively, divided by total trials (%). Latency starts at gesture completion and ends at action onset for movement commands or verified stationary state for STOP. For successful trials, latencies are reported as medians, with the 25th and 75th percentiles in brackets. Wrong actions are unintended executions.
K
Algorithm
Recognition
Scoring time (ms)
k/N (%)
Macro-F1
4
Feature-based SVM
183/200 (91.5%)
0.921
1.3
Gaussian HMM
190/200 (95.0%)
0.952
3.6
8
Feature-based SVM
351/400 (87.8%)
0.887
1.6
Gaussian HMM
366/400 (91.5%)
0.923
6.7
12
Feature-based SVM
505/600 (84.2%)
0.853
2.1
Table 5: Gesture recognition across vocabulary sizes with an 80:20 training–test split and 50 held-out test trials per class. Vocabulary size is denoted by K ; k/N reports correct predictions over all test trials, with rejection and no output counted as errors. Macro-F1 is averaged over gesture classes. Scoring time is the median per complete segment on a workstation equipped with an NVIDIA RTX 5090.
Condition
Turn
STOP
Whole-task success
k/NT (%)
Latency (s)
k/NS (%)
Latency (s)
V
181/200 (90.5%)
2.32 [1.98, 2.75]
47/50 (94.0%)
2.18 [1.85, 2.61]
32/50 (64.0%)
G
185/200 (92.5%)
1.74 [1.43, 2.12]
48/50 (96.0%)
1.24 [1.01, 1.52]
35/50 (70.0%)
Table 6: Online-adjustment performance following the same spoken task start, pooled across 10 participants. V uses speech and G uses gestures for online adjustments. Each condition contains 50 task attempts, with NT=200 turn cues and NS=50 terminal STOP cues. Success is reported as successful events/eligible events (%) or successful tasks/task attempts (%). Turn latency runs from cue presentation to rotation onset; STOP latency runs from cue presentation to verified stationary state. For successful events, latencies are reported as medians, with the 25th and 75th percentiles in brackets.
Responder
Valid timing records ( n )
Receipt → arrival (s)
Quadruped robot
28
11.71±2.94
On-site person
28
9.20±2.56
Table 7: Robot and on-site human response times following manually triggered physiological-event notifications in simulated scenarios. Times run from verified notification receipt to arrival at the marked destination and stopping. Values are mean ± SD over n=28 valid timing records per responder under matched route and starting-distance conditions.
Scope
Response interval
Time (min)
Source
US, 2015
911 receipt → first EMS unit on scene
7.9±5.7 (mean ± SD)
Mell et al. [ 35 ]
US, 2023
Dispatch → patient’s side
7.1 (median)
Hayes et al. [ 36 ]
Table 8: Published US EMS response-time statistics. Reported intervals, populations, and summary measures differ across sources and are not directly comparable with each other or with the local-arrival intervals in Table 7 .
This paper describes an initiation of interaction(IoI) detection framework without keywords for human-robot interaction(HRI) based on audio and vision sensor fusion in a domestic environment. In the proposed framework, the robot has its own audio and vision sensors, and can employ external vision sensor for stable human detection and tracking. When the user starts to speak while looking at the robot, the robot can localize his or her position by its sound source localization together with human tracking information. Then the robot can detect the IoI if it perceives the face of the speaker faces the robot. In case that the user does not speak directly, the robot can also detect the IoI if he or she looks at the robot for more than predefined periods of time. A state transition model for the proposed IoI detection framework is designed and verified by experiments with a mobile robot. In order to implement and associate our model in a robot architecture, all the components are implemented and integrated in the Robot Operating System(ROS) environment.
Guhnoo Yun, Juhan Yoo, Kijung Kim +1
Korea Institute of Science and Technology, Seoul, 02792, Republic of Korea · Department of Computer Science, Semyung University, Jecheon-si, 27136, Republic of Korea
In this paper, we investigated how the choice of a Wizard-of-Oz (WoZ) interface affects communication with a robot from both the user's and the wizard's perspective. In a conversational setting, we used three WoZ interfaces with varying levels of dialogue input and output restrictions: a) a restricted perception GUI that showed fixed-view video and ASR transcripts and let the wizard trigger pre-scripted utterances and gestures; b) an unrestricted perception GUI that added real-time audio from the participant and the robot c) a VR telepresence interface that streamed immersive stereo video and audio to the wizard and forwarded the wizard's spontaneous speech, gaze and facial expressions to the robot. We found that the interaction mediated by the VR interface was preferred by users in terms of robot features and perceived social presence. For the wizards, the VR condition turned out to be the most demanding but elicited a higher social connection with the users. VR interface also induced the most connected interaction in terms of inter-speaker gaps and overlaps, while Restricted GUI induced the least connected flow and the largest silences. Given these results, we argue for more WoZ studies using telepresence interfaces. These studies better reflect the robots of tomorrow and offer a promising path to automation based on naturalistic contextualized verbal and non-verbal behavioral data.
Ekaterina Torubarova, Jura Miniota, Andre Pereira
Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden
Social robots must coordinate speech and movement while interpreting multimodal human input. We present a multimodal framework that integrates speech, vision, and touch to generate contextually appropriate robot speech and movement. We evaluated the framework in virtual reality with 18 participants across five social interaction scenarios, comparing its performance against a human-teleoperated baseline. Although participants rated the autonomous system above the neutral midpoint for both naturalness and social appropriateness, the teleoperated baseline received significantly higher ratings on both measures. Behavioral coding and qualitative feedback revealed that conversational timing, clarification of ambiguous requests, and spatial coordination were important factors contributing to this performance gap. These findings highlight key challenges in developing autonomous robots capable of natural, multimodal social interaction.
Richard Xu, Dakarai Crowder, Alexis E. Block +1
University of Illinois at Urbana-Champaign, Champaign, IL, USA · Case Western Reserve University, Cleveland, OH, USA