cs.CVFeb 8, 2024

InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

Authors: Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li, Andrii Maksai, Jesse Berent, Claudiu Musat

Organizations: Google DeepMind · EPFL

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.

Figures & tables

Figure 2: Left [Animated visualization] : Illustration of model inference. Inputs include an image and a prompt specifying the task. Outputs may consist of ink alone or a combination of ink and text, as indicated by matching colors in both the input prompt and the output. Ink tokens are detokenized into digital ink using our proposed ink tokenizer explained in Section 3.3 . Right [Animated visualization] : Diagram of the Full Page System.
Figure 3: Illustration of multi-task training mixture. The training mixture comprises five different task types: two derendering tasks (ink output), two recognition tasks (text output), and one mixed task (text-and-ink output). Each type of task utilizes a task-specific input text, enabling the model to distinguish between tasks during both training and inference.
TaskDescription
Vanilla DerenderThe model receives a synthetic ink image with a conditioning task prompt and outputs ink tokens corresponding to the image content.
Derender with TextThe model receives a synthetic ink image and a task conditioning prompt including the target textual content, and outputs ink tokens corresponding to the provided text within the image.
Recognition (Syn/Real)The model receives a handwriting image (either synthetic or real) and a task conditioning prompt and outputs the recognized textual content. This task uses both synthetic ink images and real images from OCR datasets.
Recognize and DerenderThis task combines recognition and derendering. The model receives a synthetic ink image and a prompt and outputs both the recognized text and the corresponding ink tokens.
Table 1: Summary of tasks in our proposed multi-task training.
Figure 4: Ink tokenization for a single-stroke digital ink [Animated visualization] . The dark red ink depicts the normalized ink stroke, with numbered circles marking sampled points after time resampling. The color gradient of the sampled points indicates the point order. Each point is represented with two tokens encoding its xx and yy coordinates. The token sequence for this ink begins with b , signifying the beginning of the stroke, followed by the tokens for coordinates of sampled points.
Figure 5: Comparison between performance of GVS, Small-i , Small-p , and Large-i on IAM (top) and IMGUR5K (bottom), more visualizations in Section N.1 .
Figure 6: Comparison between performance of GVS, Small-i , Small-p , and Large-i on HierText, which includes data with varying styles and occlusions.
Figure 7: Sketch derendering for Small-p , Small-i , Large-i and GVS. Our models are mostly able to derender simple sketches, however with significant artifacts: missing strokes (e.g the cat), creating cycles (e.g the eye of the penguin for Small-i, the paws of the cat for Large-i) or unnatural strokes (e.g. for Small-p).
Figure 8: Human evaluation results for Small-p, Small-i and Large-i on 200 “golden” samples of HierText dataset.
ModelIAMIMGUR5KHierText
F1Acc.F1Acc.F1Acc.
Small-p0.650.600.470.330.530.37
Small-i0.650.590.510.330.610.44
Large-i0.660.580.510.320.610.46
GVS0.690.020.510.010.550.01
Human *––––0.640.74
Table 2: Automated metrics comparison between three variants of our model, GVS, and the “golden” human traced data. We compare both Character Level F1 score of the rendered characters and Accuracy of recognizing the generated digital ink with state-of-the-art recognizers.
Figure 9: Online handwriting recognition results on IAMOnDB test set for Small-i trained on real digital inks (IAMOnDB train set), derendered digital inks (from IAM train set) and their combination.
SetupIAMIMGUR5KHierText
 F1Acc.F1Acc.F1Acc.
Small-i †0.66 ±0.07\pm 0.070.59 ±0.01\pm 0.010.51 ±0.09\pm 0.090.33 ±0.02\pm 0.020.61 ±0.07\pm 0.070.45 ±0.01\pm 0.01
Vanilla0.61\,{\color[rgb]{0.8,0,0}\downarrow}0.53\,{\color[rgb]{0.8,0,0}\downarrow}0.44\,{\color[rgb]{0.8,0,0}\downarrow}0.28\,{\color[rgb]{0.8,0,0}\downarrow}0.53\,{\color[rgb]{0.8,0,0}\downarrow}0.35\,{\color[rgb]{0.8,0,0}\downarrow}
R+D0.62\,{\color[rgb]{0.8,0,0}\downarrow}0.57\,{\color[rgb]{0.8,0,0}\downarrow}0.46\,{\color[rgb]{0.8,0,0}\downarrow}0.320.57\,{\color[rgb]{0.8,0,0}\downarrow}0.42\,{\color[rgb]{0.8,0,0}\downarrow}
Remove  
data aug *0.42\,{\color[rgb]{0.8,0,0}\downarrow}0.33\,{\color[rgb]{0.8,0,0}\downarrow}0.21\,{\color[rgb]{0.8,0,0}\downarrow}0.09\,{\color[rgb]{0.8,0,0}\downarrow}0.23\,{\color[rgb]{0.8,0,0}\downarrow}0.13\,{\color[rgb]{0.8,0,0}\downarrow}
Table 3: Ablation studies on Small-i . The first row shows the best performance setup (with inference mode Derender with Text ). Subsequent rows display results for the Vanilla Derender and Recognize and Derender (R + D) tasks ( Figure 3 ), followed by the impact of various design choices on performance.
Figure 10: Varied digital ink outputs from Small-i inference tasks on samples with ambiguous transcription. Column titles correspond to the ground truth labels. Image titles correspond to either the recognition result from our model or an OCR system.
Appendix figures & tables54 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Large-i full page results of handwritten notes in a real-life scenario, credit: Unsplash .
Figure 12: Small-i full page results of handwritten notes in a real-life scenario, credit: Unsplash .
Figure 13: Small-p full page results of handwritten notes in a real-life scenario, credit: Unsplash .
Figure 14: Large-i full page results of French sample. credit: Unsplash
Figure 15: Small-i full page results of French sample. credit: Unsplash
Figure 16: Small-p full page results of French sample. credit: Unsplash
Figure 17: Large-i full page results of handwritten notes in a real-life scenario with low resolution, credit: Unsplash .
Figure 18: Small-i full page results of handwritten notes in a real-life scenario with low resolution, credit: Unsplash .
Figure 19: Small-p full page results of handwritten notes in a real-life scenario with low resolution, credit: Unsplash .
Figure 20: Large-i full page results of a Post-It note.
Figure 21: Small-i full page results of a Post-It note.
Figure 22: Small-p full page results of a Post-It note.
Figure 23: Large-i full page results of handwritten notes of mass-energy equivalence.
Figure 24: Small-i full page results of handwritten notes of mass-energy equivalence.
Figure 25: Small-p full page results of handwritten notes of mass-energy equivalence.
Figure 26: Large-i full page results of multilingual handwritten notes.
Figure 27: Small-i full page results of multilingual handwritten notes.
Figure 28: Small-p full page results of multilingual handwritten notes.
Figure 29: Large-i full page results of OOD sample.
Figure 30: Small-i full page results of OOD sample.
Figure 31: Small-p full page results of OOD sample.
Figure 32: Large-i full page results of OOD sample, credit: Unsplash .
Figure 33: Small-i full page results of OOD sample, credit: Unsplash .
Figure 34: Small-p full page results of OOD sample, credit: Unsplash .
Figure 35: Large-i full page results of handwritten notes in a real-life scenario, credit: Unsplash .
Figure 36: Small-i full page results of handwritten notes in a real-life scenario, credit: Unsplash .
Figure 37: Small-p full page results of handwritten notes in a real-life scenario, credit: Unsplash .
Figure 38: Large-i full page results of handwritten notes in a real-life scenario in Korean, credit: Unsplash .
Figure 39: Small-i full page results of handwritten notes in a real-life scenario in Korean, credit: Unsplash .
Figure 40: Small-p full page results of handwritten notes in a real-life scenario in Korean, credit: Unsplash .
Figure 41: In-house Datasets Language Distributions. Left: Digital inks, Right: OCR for Recognition. These datasets are used to train both in-house models Small-i and Large-i .
Figure 42: Public Datasets. Language distribution of public Digital ink datasets used to train our public models (on the left) and public OCR datasets used to train our public available model Small-p (on the right).
Task TypeDatasetSamplesLicense / Terms
DerenderingDeepWriting (words)89,565Custom 1
DeepWriting (lines)33,933Custom 1
DeepWriting (characters)359,643Custom 1
VNOnDB66,991Custom 2
SCUT-COUCH (chars)1,998,784Custom 3
SCUT-COUCH (pinyin)15,653Custom 3
Appendix
Table 4: Public training datasets used for training small-p .
StudyCode AvailabilityData AvailabilityModel Weights AvailabilityYear
Mohamed Moussa et al. (2023)✗✓✗2023
Chen et al. (2022b)✗ Repo missing core elements: data preprocessing, incompatible configs✓✗2022
Archibald et al. (2021)✗✓✗2021
Bhunia et al. (2021)✗ Repo missing core elements: training scripts, data preprocessing✓✗2021
Nguyen et al. (2021)✗✓✗2021
Sumi et al. (2019)✗✓✗2019
Appendix
Table 5: Existing pen trajectory recovery approaches and their reproducibility.
Figure 43: Comparison between frozen and unfrozen ViT Small-i trainings. Comparison is executed with 3 random seeds. Left: ratio of empty ink outputs for unfrozen ViT multi-task training setup. Right: ratio of empty ink outputs for the frozen ViT multi-task training setup, both use inference task Derender with Text .
Figure 44: Difference between Small-i-unfrozen which was trained with an unfrozen ViT (seed 46 in Figure 43 ) and Small-i which was trained with a frozen ViT. This shows that Small-i-unfrozen is more sensitive to background noise compared to Small-i .
Figure 45: Difference between inference tasks of Small-i on samples from 3 public evaluation datasets, where the texts on top are the ground truth labels for the images in these datasets. Vanilla stands for inference with Vanilla Derendering , w/Recognition stands for inference with Recognize and Derender , and w/ OCR Input stands for inference with Derender with Text where we resort to an OCR system.
Figure 46: Cases when there is a mismatch between ground-truth textual information and external textual information provided to the model. We evaluated this using Small-i on samples from 3 public evaluation datasets, where the texts on top are the ground truth labels for the images in these datasets. Vanilla stands for inference with Vanilla Derender , w/Recognition stands for inference with Recognize and Derender , and w/ OCR Input stands for inference with Derender with Text where we resort to an OCR system.
AugmentationPossible ValuesExamples
Rotationangle (rad): [−π4,π4][-\frac{\pi}{4},\frac{\pi}{4}] 
Stroke colorRGB: [0,1]3[0,1]^{3} 
Background colorRGB: [0,1]3[0,1]^{3} 
Stroke Widthwidth: [1px,12px][1\text{px},12\text{px}] 
Linesline width: [1px,6px][1\text{px},6\text{px}] line dist: [10px,100px][10\text{px},100\text{px}] line color: RGB: [0,1]3[0,1]^{3} 
Gridsline width: [1px,6px][1\text{px},6\text{px}] line dist: [10px,100px][10\text{px},100\text{px}] line color: RGB: [0,1]3[0,1]^{3} 
Appendix
Table 6: Data Augmentations Overview (sample from Aksan et al. 2018 ).
Figure 47: Demonstration of possible ways to combine all used augmentations. (Sample from ( Aksan et al., 2018 ) )
ModelComponentsParametersTraining Data
Large-iViT-L303MIn-house proprietary datasets
mT5-Large783Mas shown in Appendix C .
Small-iViT-B85MIn-house proprietary datasets
mT5-Base247Mas shown in Appendix C .
Small-pViT-B85MPublicly available datasets
mT5-Base247Mas shown in Appendix C .
Appendix
Table 7: Model overview and components.
Figure 48: Comparison between General Virtual Sketching Framework inference setups.
 IAMOnDBSmall-i IAM derenderedSmall-i IAMOnDB + IAM derenderedSmall-p IAM derenderedSmall-p IAMOnDB + IAM derenderedLarge-i IAM derenderedLarge-i IAMOnDB + IAM derendered
CER6.17.84.68.24.58.44.6
Appendix
Table 8: Online handwriting recognition results on IAMOnDB test set for models trained on real digital inks (IAMOnDB train set), derendered digital inks (from IAM train set) and the combination of the two.
DescriptionInstruction
Task setupYou will be looking at input images of photos of handwriting in a variety of styles and the corresponding digital ink tracing performed either by a human or by one of our derendering models. Each stroke is rendered in a different color and with color gradient (darker →\to whiter) to reflect the direction. You will see the derendered ink as well as the derendered ink overlaid on top of the image and can use either to make the judgment. For each sample answer the two questions on the right and click submit to proceed to the next sample.
Is ReasonableIs the digital ink a reasonable tracing of the input image? • Yes, it’s a good tracing • It’s an okay tracing, but has some small errors • It’s a bad tracing, has some major artifacts One interpretation of the quality of the ink could be like this: If you were to use it in a note taking app, would you • keep it as is: good tracing • need to make some edits: okay tracing • not use it at all: bad tracing If you need some examples to get a better feeling, you can use the ones below, however there is some ambiguity within the task since we don’t want to impose any strict rules but allow for a natural variety of what people find a good derendering (it is designed to be subjective).
Is RealCould this digital ink have been produced by a human? • Yes • No In this question go with your intuition, do you think the ink could have been traced by a [reasonable] human?
Appendix
Table 10: Human evaluation campaign instructions.
Figure 49: A screenshot of the human evaluation interface.
Figure 50: Examples of samples produced by our models and ratings given to them by our evaluators.
Model Summary
Model ArchitectureA multimodal sequence-to-sequence Transformer ( Vaswani et al., 2017 ) model with the mT5 ( Xue et al., 2021 ) encoder-decoder architecture. It takes text tokens and ViT ( Dosovitskiy et al., 2021 ) dense image embeddings as inputs to an encoder and autoregressively predicts discrete text and ink tokens with a decoder.
Input(s)A pair of image and text.
Output(s)Generated digital ink.
Usage 
ApplicationThe model is for research prototype, and the public version Small-p is planned to be released and available for the public.
Known CaveatsNone.
Appendix
Table 11: Model card of publicly available Small-p .
Figure 51: Results of Small-p on randomly selected samples from 3 public evaluation datasets.
Figure 52: Results of Small-i on randomly selected samples from 3 public evaluation datasets.
Figure 53: Results of Large-i on randomly selected samples from 3 public evaluation datasets.
Figure 54: Comparison between Small-i , Small-p , and Large-i on IAM Dataset.
Figure 55: Comparison between Small-i , Small-p , and Large-i on IMGUR5K Dataset.
Figure 56: Comparison between Small-i , Small-p , and Large-i on HierText Dataset.
Figure 57: Inability to perform open-ended generation. Prompting Small-p , Small-i and Large-i with text that is not presented in the image does not produce digital inks that match the textual input.
More in this topic

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 493 papers

Explore similar work

CardsList
Mar 3, 2026

ScribeTokens: Fixed-Vocabulary Tokenization of Digital Ink

Digital ink -- the coordinate stream captured from stylus or touch input -- lacks a unified representation. Continuous vector representations produce long sequences and suffer from training instability, while existing token representations require large vocabularies, face out-of-vocabulary issues, and underperform vectors on recognition. We propose ScribeTokens, a tokenization that decomposes pen movement into unit pixel steps. Together with two pen-state tokens, this fixed 10-token base vocabulary suffices to represent any digital ink and enables aggressive BPE compression. On handwritten text generation, ScribeTokens dramatically outperforms vectors (17.33% vs. 70.29% CER), showing tokens are far more effective for generation. On recognition, ScribeTokens is the only token representation to outperform vectors without pretraining. We further introduce next-ink-token prediction as a self-supervised pretraining strategy, which consistently improves recognition across all token-based models and accelerates convergence by up to 83x. With pretraining, ScribeTokens achieves the best recognition results across all representations on both datasets (8.27% CER on IAM, 9.83% on DeepWriting).

Douglass Wang

Independent Researcher

Online Handwriting RecognitionSelf-Supervised Pre-Training
AskPDF
Apr 2, 2026

BoundInk: Boundary-Aware Online Handwriting Generation

Realistic online handwriting depends not only on individual character shapes, but also on how a writer connects, spaces, and aligns adjacent characters. Existing methods rely primarily on long-range sequence modeling and capture these inter-character behaviors only implicitly. This often produces plausible glyphs accompanied by broken cursive joins, inconsistent spacing, or writer-inconsistent transitions. We introduce BoundInk, a writer-conditioned framework that treats inter-character boundaries as explicit generation units. By jointly modeling local transitions and surrounding text context, BoundInk preserves writer-specific glyph appearance while improving connectivity and spacing across complete text lines. We further introduce a boundary-aware evaluation framework that directly assesses cursive continuity and spatial relationships between characters beyond conventional trajectory similarity. Across three benchmark-matched settings, BoundInk improves all applicable boundary-quality measures and reduces normalized dynamic time warping by 17.6--47.8%. In blind human evaluations, BoundInk outputs are preferred in 78.0--82.6% of valid criterion-wise judgments.

Jinsu Shin, Sungeun Hong, JinYeong Bak

Sungkyunkwan University, Suwon, Korea

Conditional Generative Modeling
AskPDF
Jul 3, 2026

Handwriting Trajectory Recovery with Diffusion Models

Recovering online pen trajectories from offline handwriting images, often referred to as handwriting trajectory recovery (stroke recovery), is an offline-to-online conversion task with applications in stroke-level editing and forensic analysis. We propose, to the best of our knowledge, the first diffusion-model-based framework for this task. Our method formulates trajectory recovery as image-conditioned generation and uses a denoising diffusion model to sample pen trajectories consistent with the observed ink trace. Through extensive quantitative evaluations on CASIA-OLHWDB (1.0-1.1), we verify that the proposed approach enables accurate recovery even for complex multi-stroke characters, substantially improving both temporal similarity (DTW/LDTW) and shape fidelity (AIoU) over representative prior methods such as PEN-Net and Cross-VAE. We further show that the model captures general stroke-order tendencies and generalizes to classes unseen during training, exemplified by cross-script transfer: a model trained on Chinese characters can recover reasonable stroke orders for Latin letters to some extent.

Hiroki Nagamatsu, Shoji Toyota, Seiichi Uchida

Kyushu University, Fukuoka, Japan

Topic pending
AskPDF