cs.CVNov 17, 2026

Event-Based Dynamic Lip Region Estimation During Speech

Authors: Hamze Hammami, Omar Eldardeer, Francesco Rea

Organizations: Independent Researcher Dubai, UAE · Italian Institute of Technology (CONTACT Unit) Genova, Italy

Abstract

Event cameras offer high temporal resolution and low power consumption, making them promising for speech-related applications. These include automatic lip reading, audiovisual speech analysis, and human-robot interaction, particularly in challenging environments characterized by low illumination or high dynamic range. Event-based lip reading, speaker recognition, and voice activity detection, however, mostly operate on a cropped region of interest (ROI), by hand or by the dataset, and no event-based method supplies that ROI as a spatial region; existing event-based face analysis gives the mouth as landmarks, which presents a mismatch between application and localization. The present work predicts a dense heatmap over the lip region from the full event "frame". Events are encoded as a polarity-separated voxel grid and processed at every location by a two-pathway continuous-time recurrent network (CT-RNN) with learned, log-space bounded time constants acting on the real elapsed time of each bin: a stateless fast pathway and a stateful slow pathway, utilised for refining and anchoring the prediction. The model reaches 69.5% on lips on four unseen speakers and switches target only 2.1 times per second, against 8.9 and 11.0 for convolutional baselines with more parameters. Adding a spatial encoder improves precision but fails on a new face scale. On a real event sensor, trained only on simulated events, the proposed model reaches 58.0% on lips against 43.2% for the CNN, with fewer than half its target switches.

Figures & tables

Explore similar work

CardsList
  1. TVTA: Trajectory-Aware Viseme-Guided Temporal Aggregation for Event-Based Lip Reading

    Jul 9, 2026Jingrong Zheng, Hongwei Ren, Xiangqian WuEvent-Based VisionVisual Speaker Recognition

  2. NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition

    Apr 17, 2026Junguang Yao, Wenye Liu, Stjepan Picek +1Event-Based VisionDomain Generalization

  3. LipLingo: CNN Model for Lip Reading

    Nov 2, 2026Mahmoud S. Mayaleh, Liban Ali Achkir, Nijat Mehdiyev +1Automatic Speech Recognition