Digital pens are widely used to capture handwriting on digital devices, enabling precise trace recording and enhancing human-computer interaction. However, most are bundled with tablets and lack cross-brand compatibility. Recent digital pens equipped with kinematic sensors have emerged, designed for use on any surface. This especially opens significant potential for supporting handwriting acquisition in classrooms. Handwriting reconstruction from such an IMU-equipped pen poses a challenge due to the significant variability in sensor signals between adults and children. Even when producing visually similar traces, variations in writing dynamics, motor control, pen holding, and user confidence introduce substantial discrepancies in the captured signals. Additionally, the high variability in children's handwriting requires collecting large amounts of data, which is not feasible to implement in a school environment at scale. Furthermore, models trained exclusively on adult data fail to generalize to children's handwriting, and conversely, models trained on children's data perform poorly on adult writers. This crosspopulation degradation highlights the need for a unified model that can be deployed directly on the pen, without any user-specific adaptation. To address this issue, we propose a cross-domain learning, using an original neural network architecture based on a Temporal Convolutional Network and multiple prediction heads. The model is designed to be robust across age groups by leveraging shared features while effectively handling variability induced by differences in graphomotor development. This approach aims to improve handwriting trace reconstruction from sensor data, where each domain benefits from additional data provided by the other domain.
Digital pens are commonly used to write on digital devices, providing the handwriting trace and enhancing human-computer interation. This study focuses on a digital pen equipped with kinematic sensors, allowing users to write on any surface while simultaneously preserving a digital trajectory of handwriting. This technology holds significant potential as a valuable educational tool, particularly in classrooms where it can facilitate the process of learning to write. A major issue is based on the difference in captured signals between adults and children. For similar handwriting trace, we have large differences in sensor signals due to differences in speed and confidence in the handwriting gesture of children. To address this, we investigate a domain adaptation approach to build a unified intermediate feature representation aimed at facilitating the trajectory reconstruction. We demonstrate the interest of domain adaptation methods in leveraging existing knowledge for application in different contexts. Specifically, we compare our domain adaptation approach with two other methods: training the model from scratch and fine-tuning the model.
Florent Imbert, Romain Tavenard, Yann Soullard +1
IRISA, Universite de Rennes, INSA Rennes, Rennes, France · IRISA, Universite Rennes 2, Rennes, France
Capturing the digital trace of handwriting usually requires a specific stylus and a compatible substrate, be it a capacitive touchscreen, an ElectroMagnetic Resonance (EMR) tablet as used in Wacom systems or special paper. While writing on regular paper offers rich haptics, no latency and is well known for improving information retention, no low-cost and widely accepted, effective solution exists to digitize such a pen trace. The challenge is to accurately track the pen's trajectory without an external reference system while allowing unrestricted freedom of pen movement across a surface. We propose an innovative solution that combines a digital pen, advanced artificial intelligence algorithms, and adaptive AI techniques to reconstruct the digital trace of handwriting. Our approach integrates hardware development, focusing on a sensor-equipped pen, with software innovations to optimize trajectory reconstruction and processing in real time using an embedded AI. This work aims to advance the state-of-the-art in automated trace reconstruction of handwriting, enabling a seamless connection between traditional handwriting on paper and capturing the trace digitally.
Florent Imbert, Yann Soullard, Eric Anquetil +5
IRISA, Universit´e de Rennes, INSA Rennes, France · LTU, Machine Learning Group, Lule˚a University of Technology, Sweden · IRISA, Universit´e Rennes 2, France +2
The use of digital pens for online handwriting trajectory reconstruction is a prevalent method for human-computer interaction. In this study, we focus on a digital pen equipped with sensors where we aim at reconstructing the online handwriting trajectory. This pen enables writing on any surface and preserving the digital trace of handwriting. This type of pen could be used as an aid to learning to write in classroom. In this paper, we propose a new approach learning to finely reconstruct the touching trajectories while precisely analyzing the hovering part in order to position the next touching trace correctly. This relies on a Mixture-Of-Experts (MOE) approach. The first expert is dedicated for the pencil touch, and is named touching expert model. The second one is dedicated for the hovering pen trajectory, and is named hovering expert model. We improve on the learning of each of these experts based on additional context or specific examples. In addition we introduce a novel public benchmark dataset, to enable future research and comparisons in the field of handwriting reconstruction. The results demonstrates a significant enhancement compared to its primary competitors.