Sensor-Placement-Agnostic Sonomyography: Toward Continuous High-Dimensional Control by Users with Tetraplegia
Authors: Gavin Sueltz, Vikram Athithan, Emma Ferran, Maria Herrera, Carson J. Wynn, Laura A. Hallock
Abstract
Sonomyography (SMG) enables continuous device control via ultrasound-measured muscle deformation signals, but existing SMG interfaces generally require substantial user- and sensor-location-specific training data and provide only one proportional signal or task-specific classification. We present a real-time, sensor-placement-agnostic SMG control system based on sparse optical flow tracking that enables continuous 1-DOF control after minimal calibration (3 pose definitions). We also present a preliminary expansion of this method that augments this algorithm with a short computer-aided calibration to enable 2-DOF control. We evaluate both 1- and 2-DOF systems' performance for a preliminary cohort of 3 cervical spinal cord injury survivors and 6 uninjured individuals across 6 sensor placements spanning the arm, neck, and upper torso. As assessed by a cursor trajectory tracking task, all participants achieved continuous 1-DOF control at all tested sensor locations (even those that relied on passive tissue motions), with all participants achieving <5.5% tracking error using at least one placement (and many <4% across many). All participants were also able to modulate 2D cursor position via the 2-DOF system, with varying levels of control authority, and several were able to complete a 2D drawing task, constituting the first (to our knowledge) demonstration of location-agnostic multi-DOF continuous SMG-based control. These results highlight the promise of SMG to enable rapidly calibratable, high-dimensional, sensor-placement-agnostic device control by users with tetraplegia, and also illuminate key challenges in both signal processing and practical system deployment. To enable further development by scientific and user communities, developed algorithms have been open-sourced as part of the OpenMyoControl project on SimTK (simtk.org/projects/openmyocontrol).
Powered prosthetic hands are frequently abandoned, largely due to the limited functionality of current devices that rely on surface electromyography (sEMG). Sonomyography (ultrasound) has emerged as a promising alternative, owing to its ability to observe muscle activity in real time and control a greater number of degrees of freedom. Yet, existing ultrasound-based methods require per-user fine-tuning, limiting their commercialization. We propose SonoRank, an important step towards calibration-free finger flexion detection from forearm ultrasound video. SonoRank first learns to rank pairs of ultrasound sequences by their relative motion magnitude for each of the five fingers. The learned representations are then fine-tuned to classify whether each finger is actively flexing, using a rest reference that is captured at the beginning of the operation. Under 12-fold leave-one-subject-out cross-validation on a dataset of twelve subjects with synchronized kinematics, SonoRank achieves a 28% improvement in F1 score over direct classification baselines that skip the ranking stage. These results establish pairwise ranking as an effective pretraining signal for subject-independent control, bringing ultrasound-based prosthetics closer to practical, calibration-free deployment.
Flexible endoscopic continuum manipulators offer high dexterity and access to complex anatomy, but nonlinear hysteresis limits feedforward control accuracy. Closed-loop control can compensate for these errors but requires accurate six-degree-of-freedom (6D) end-effector pose feedback. We present MarUco, a markerless 6D pose estimation framework for closed-loop control using only stereo vision during operation. A photorealistic pseudo-rigid-body simulation pipeline generates large-scale annotated training data without manual labeling. A multifeature fusion network integrates masks, keypoints, heatmaps, and bounding boxes from both stereo views to estimate an initial pose, followed by a learned single-pass render-and-compare module that refines the pose without iterative optimization. Kinematics-free hand-eye calibration estimates camera-to-robot-base extrinsics, and self-supervised adaptation uses unlabeled real stereo pairs to mitigate sim-to-real pose error. Across 1,000 real samples, MarUco achieves translation and rotation errors of 0.78 ± 0.50 mm and 3.07 ± 1.25°, respectively, with a total processing time of 52.4 ms per stereo pair. In closed-loop experiments over eight reference paths, MarUco achieves a mean terminal translation error of 1.8 mm, an 88% reduction relative to uncompensated open-loop control. To the best of our knowledge, this is the first markerless position-based visual servoing framework for continuum manipulators.
Pattern-recognition control promises a myoelectric prosthesis that responds to many intended gestures rather than one or two, but the promise has stayed in the laboratory. A recogniser trained on one person rarely transfers to the next, and useful performance usually demands a fresh round of labelled calibration from the end user. A montage-agnostic encoder is introduced that reads each electrode with shared weights and locates it by its physical coordinate rather than its index, so one architecture ingests any channel count without montage-specific parameters. Trained across users, it exceeds a per-user Hudgins and linear-discriminant classifier by 0.234 macro-F1 on DB1 for every held-out subject and by 0.108 on DB2, and falls below it on the ten-subject DB5. Each of the encoder's three key components individually accounts for more than half of its 3-shot macro F1 in an otherwise budget-matched ablation study. A controlled subject-count sweep shows the margin is close to flat from nine training subjects to thirty-nine, so the training pool binds only as a stability floor below which cross-user training fails to converge; what tracks the direction of the comparison across the three databases is instead the strength of the per-user baseline, which signal fidelity sets. Comparing against an LDA baseline depends on budget spent training models and on how good that baseline is, and self-supervised pretraining had no benefits once a supervised model was adequately trained.