Event-Based Dynamic Lip Region Estimation During Speech
Organizations: Independent Researcher Dubai, UAE · Italian Institute of Technology (CONTACT Unit) Genova, Italy
Abstract
Event cameras offer high temporal resolution and low power consumption, making them promising for speech-related applications. These include automatic lip reading, audiovisual speech analysis, and human-robot interaction, particularly in challenging environments characterized by low illumination or high dynamic range. Event-based lip reading, speaker recognition, and voice activity detection, however, mostly operate on a cropped region of interest (ROI), by hand or by the dataset, and no event-based method supplies that ROI as a spatial region; existing event-based face analysis gives the mouth as landmarks, which presents a mismatch between application and localization. The present work predicts a dense heatmap over the lip region from the full event "frame". Events are encoded as a polarity-separated voxel grid and processed at every location by a two-pathway continuous-time recurrent network (CT-RNN) with learned, log-space bounded time constants acting on the real elapsed time of each bin: a stateless fast pathway and a stateful slow pathway, utilised for refining and anchoring the prediction. The model reaches 69.5% on lips on four unseen speakers and switches target only 2.1 times per second, against 8.9 and 11.0 for convolutional baselines with more parameters. Adding a spatial encoder improves precision but fails on a new face scale. On a real event sensor, trained only on simulated events, the proposed model reaches 58.0% on lips against 43.2% for the CNN, with fewer than half its target switches.
Figures & tables
| Method | Objective | Event representation | Model | Mouth output | Input crop |
|---|---|---|---|---|---|
| [ 27 ] | Lip reading (word classification) | Event frames at two frame rates | Two branches, message flow fusion | None, window given | Mouth, by dataset ( ) |
| [ 31 ] | Speaker recognition | Polarity-separated voxels, learned temporal allocation | Modified ResNet34 | None, window given | Mouth, by hand ( ) |
| [ 25 ] | Voice activity detection | Hand-crafted Gabor filter | MAP estimate over a coarse grid | Lip centre (one point) | None, optional face prior |
| [ 17 ] | Face and five-landmark detection | Locally adaptive decay surface | MobileNetV3, PIPNet head | Mouth corners (2 of 5 points) | Face, by detector |
| [ 18 ] | 98-landmark and blink detection | Histogram or time surface, summed polarity | MobileNetV2, ConvLSTM, PIPNet head | Mouth contour (about 20 of 98 points) | Face, by detector |
| Proposed | Lip region localization | Polarity-separated voxel grid ( bins) | Two-pathway CT-RNN with persistent slow memory | Dense lip region (heatmap) | None |
| On lips (%) | Med. | P90 | 10 px | IoU 0.5 | Eye | Switch | Jitter | Streak | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Params | S10 | S11 | S12 | S13 | All | px | px | % | % | % | /s | cells | ms |
| Ours | 29,107 | 91.7 | 75.7 | 31.9 | 79.3 | 69.5 | 16.2 | 144 | 24.6 | 35.9 | 6.2 | 2.1 | 2.8 | 326 |
| CNN | 164,650 | 96.9 | 82.7 | 47.6 | 74.5 | 75.6 | 11.2 | 81 | 44.9 | 43.2 | 3.9 | 8.9 | 6.0 | 89 |
| U-Net | 165,846 | 87.5 | 91.5 | 40.1 | 31.9 | 62.2 | 13.9 | 140 | 40.3 | 43.1 | 15.7 | 11.0 | 6.6 | 63 |
| Ours + deep encoder | 50,221 | 89.5 | 90.9 | 65.3 | 17.6 | 66.4 | 13.8 | 115 | 31.1 | 22.0 | 8.6 | 2.1 | 1.9 | 281 |
| Term | Definition | |
|---|---|---|
| 1.0 | ||
| Tversky, on missed mass | 1.0 | |
| focal BCE, , | 0.2 | |
| 4.0 | ||
| 0.5 | ||
| balanced BCE between and | 0.5 |
| Setting | Med. px | Eye % | Detected % | Switch /s | Jitter cells | Streak ms |
|---|---|---|---|---|---|---|
| Full model | 16.2 | 6.3 | 98.6 | 2.14 | 2.78 | 326 |
| w/o fast pathway | 17.4 | 5.7 | 95.8 | 0.78 | 1.37 | 712 |
| w/o slow pathway | 25.1 | 11.6 | 99.7 | 10.14 | 7.90 | 59 |
| w/o cross-window state | 23.3 | 11.5 | 99.8 | 9.30 | 7.40 | 67 |
| On lips | Med. | P90 | 10 px | IoU 0.5 | Eye | Switch | Jitter | Streak | |
|---|---|---|---|---|---|---|---|---|---|
| Model | % | px | px | % | % | % | /s | cells | ms |
| Ours | 58.0 | 17.6 | 208 | 26.0 | 27.1 | 2.3 | 5.0 | 5.1 | 126 |
| CNN | 43.2 | 28.4 | 414 | 10.6 | 17.8 | 11.9 | 12.4 | 13.9 | 42 |
Event-Based Vision
Event-based vision uses asynchronous camera measurements to perceive motion and scene changes with high temporal resolution and dynamic range.
See all 250 papers