cs.LGNov 2, 2026

LipLingo: CNN Model for Lip Reading

Authors: Mahmoud S. Mayaleh, Liban Ali Achkir, Nijat Mehdiyev, Ahmet Ozdil

Organizations: National Conservatory of Arts and Crafts (CNAM), Paris, France · Ostim Technical University, Ankara, Türkiye

Abstract

Lip reading plays a crucial role in applications such as speech recognition, assistive technologies for the hearing-impaired, and biometric authentication. However, performance often degrades when the speaker varies and when certain speech sounds are visually similar. To address these challenges, we propose LipLingo, a robust lip-reading system based on the LipNet architecture, enhanced with a standardized preprocessing pipeline for consistent mouth-region representation, a rotational validation strategy for improved generalization to unseen speakers, and a flexible similarity evaluation for more reliable sentence-level assessment. The model architecture combines 3D convolutional layers for spatio-temporal feature extraction and bidirectional recurrent layers for sequence modeling, optimized using the Adam optimizer and Connectionist Temporal Classification (CTC) loss. On the GRID dataset, LipLingo achieved 95.0% character-level accuracy (CER 4.99%) and 87.2% sentence-level accuracy. At the corpus level, the system obtained a (WER 13.1%), outperforming the baseline LipNet and demonstrating competitive performance relative to recent 3D-ResNet + Bi-GRU + MixUp approaches. These results provide insights into strategies that enhance lip-reading robustness and highlight directions for future work toward real-world deployment.

Figures & tables

Explore similar work

CardsList
  1. Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

    Oct 14, 2026Jing-Xuan Zhang, Genshun Wan, Jianqing Gao +3Audio-Visual Representation LearningCross-Modal Knowledge Distillation

  2. Event-Based Dynamic Lip Region Estimation During Speech

    Nov 17, 2026Hamze Hammami, Omar Eldardeer, Francesco ReaEvent-Based Vision

  3. TVTA: Trajectory-Aware Viseme-Guided Temporal Aggregation for Event-Based Lip Reading

    Jul 9, 2026Jingrong Zheng, Hongwei Ren, Xiangqian WuEvent-Based VisionVisual Speaker Recognition