ScribeTokens: Fixed-Vocabulary Tokenization of Digital Ink

Douglass Wang

cs.CVMar 3, 2026

ScribeTokens: Fixed-Vocabulary Tokenization of Digital Ink

Authors: Douglass Wang

Organizations: Independent Researcher

Abstract

Digital ink -- the coordinate stream captured from stylus or touch input -- lacks a unified representation. Continuous vector representations produce long sequences and suffer from training instability, while existing token representations require large vocabularies, face out-of-vocabulary issues, and underperform vectors on recognition. We propose ScribeTokens, a tokenization that decomposes pen movement into unit pixel steps. Together with two pen-state tokens, this fixed 10-token base vocabulary suffices to represent any digital ink and enables aggressive BPE compression. On handwritten text generation, ScribeTokens dramatically outperforms vectors (17.33% vs. 70.29% CER), showing tokens are far more effective for generation. On recognition, ScribeTokens is the only token representation to outperform vectors without pretraining. We further introduce next-ink-token prediction as a self-supervised pretraining strategy, which consistently improves recognition across all token-based models and accelerates convergence by up to 83x. With pretraining, ScribeTokens achieves the best recognition results across all representations on both datasets (8.27% CER on IAM, 9.83% on DeepWriting).

Figures & tables

Figure 1: ScribeTokens representation of a handwritten sentence. Pen strokes are decomposed into unit directional steps via Bresenham’s algorithm, then compressed with BPE. Each color denotes a distinct BPE token; faint colors indicate pen-in-air movement between strokes. The zoom shows the sequence of arrows making up an example token.

Representation	Value
Digital Ink	$\color[rgb]{0,0,1}((0,0),(1,0))$ , $\color[rgb]{1,0,0}((2,1),(4,-1))$
Visualization	$x$ $y$ $0$ $1$ $2$ $3$ $4$ $-1$ $0$ $1$
Point-3	$\color[rgb]{0,0,1}(1,0,1)$ , $\color[rgb]{0.5,0.5,0.5}(1,1,0)$ , $\color[rgb]{1,0,0}(2,-2,1)$
Point-5	$\color[rgb]{0,0,1}(1,0,1,0,0)$ , $\color[rgb]{0.5,0.5,0.5}(1,1,0,1,0)$ , $\color[rgb]{1,0,0}(2,-2,0,0,1)$
AbsTokens	[(0, 0)] , [(1, 0)] , [UP] , [(2, 1)] , [(4, -1)] , [UP]
RelTokens	[(1, 0)] , [UP] , [(1, 1)] , [(2, -2)] , [UP]

Table 1: Digital ink representations for a two-stroke example. Colors distinguish the first stroke , pen-in-air , and second stroke . Point-3 encodes offsets with a binary pen flag; Point-5 extends this with a one-hot pen state that additionally signals end of sequence; AbsTokens and RelTokens discretize absolute and relative coordinates into tokens; TextTokens serialize offsets as character sequences; ScribeTokens (Ours) decompose strokes into unit directional steps via Bresenham’s algorithm.

Panel of Figure 2 — Figure 2: Bresenham Decomposition of a line segment between two grid points (start , end ). The segment is rasterized into adjacent grid cells via Bresenham’s algorithm, then encoded as a sequence of Freeman chain code directions.

Figure 3: Average compression ratios (

\uparrow

) of BPE-based digital ink representations on the IAM validation set, across target vocabulary sizes and quantization parameters

\delta

. ScribeTokens consistently achieves the highest compression across nearly all settings.

Figure 4: Average out-of-vocabulary (OOV) rates (

\downarrow

) of BPE-based digital ink representations on the IAM validation set, across target vocabulary sizes and quantization parameters

\delta

. ScribeTokens and TextTokens are OOV-free by construction, while AbsTokens and RelTokens exhibit non-zero OOV rates as their coordinate-based vocabularies inevitably encounter unseen values at test time.

		IAM		DeepWriting
Method		CER (%) $\downarrow$	Acc (%) $\uparrow$	CER (%) $\downarrow$	Acc (%) $\uparrow$
Point-5		$9.43$	$31.38$	$10.94$	$81.97$
	+PT	$13.63$ (+4.19)	$18.92$ (-12.45)	$10.25$ (-0.69)	$83.16$ (+1.19)
RelTokens		$12.69$	$25.11$	$11.22$	$81.60$
	+PT	$9.16$ (-3.53)	$30.75$ (+5.65)	$10.49$ (-0.73)	$82.73$ (+1.13)
TextTokens		$82.00$	$0.00$	$11.65$	$81.17$

Table 2: Handwritten text recognition on IAM and DeepWriting, measured by CER (

\downarrow

) and accuracy (

\uparrow

) under autoregressive decoding (temperature 0). +PT denotes initialization from next-ink-token prediction pretraining prior to task fine-tuning. Deltas show the effect of pretraining: teal indicates improvement, red degradation. Best results are bolded .

		IAM		DeepWriting
Method		CER (%) $\downarrow$	Acc (%) $\uparrow$	CER (%) $\downarrow$	Acc (%) $\uparrow$
Point-5		$70.29$	$0.03$	$14.36$	$71.09$
	+PT	$16.83$ (-53.46)	$15.96$ (+15.93)	$26.84$ (+12.48)	$51.24$ (-19.85)
RelTokens		$25.89$	$4.43$	12.75	73.04
	+PT	$11.93$ (-13.96)	$20.33$ (+15.90)	$12.98$ (+0.23)	$72.48$ (-0.56)
TextTokens		$13.65$	$16.58$	$15.32$	$70.85$

Table 3: Handwritten text generation on IAM and DeepWriting, measured by CER (

\downarrow

) and accuracy (

\uparrow

) under autoregressive decoding (temperature 1). +PT denotes initialization from next-ink-token prediction pretraining prior to task fine-tuning. Generated inks are post-processed (Appendix A ) and evaluated by the best HTR model (ScribeTokens + PT; Table 2 ). Deltas show the effect of pretraining: teal indicates improvement, red degradation. Best results are bolded .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Effect of quantization parameter

\delta

on reconstruction quality. Each row shows the same IAM sample quantized at a different

\delta

, displayed both as raw quantized ink (left) and after Savitzky–Golay post-processing (right). Post-processed inks are visually indistinguishable from the original for

\delta\leq 8

; the row at

\delta=8

(highlighted in green) maximizes compression without sacrificing fidelity and is used in all downstream experiments.

Architecture
Model type	Decoder-only Transformer (LLaMA-style)
Parameters	${\sim}$ 34 $M (token-based) / ${\sim}$21$ M (Point-5)
Layers	$12$
Attention heads	$6$
Hidden dimension	$384$
Positional encoding	RoPE [ 33 ]

Appendix

Table 4: Architecture and training hyperparameters shared across all representations and tasks.

	IAM				DeepWriting
Method	PT Ep.	No PT	+PT	Spd.	PT Ep.	No PT	+PT	Spd.
Point-5	$36$	$171$	–	–	$55$	$13$	–	–
RelTokens	$72$	$193$	$13$	$14.8$ $\times$	$99$	$23$	$9$	$2.6$ $\times$
TextTokens	$99$	$18$	$6$	$3.0$ $\times$	$100$	$30$	$6$	5.0 $\times$
ScribeTokens (Ours)	$90$	$193$	$9$	21.4 $\times$	$99$	$26$	$8$	$3.2$ $\times$

Appendix

Table 5: Convergence speedup from pretraining for HTR. PT Ep. : pretraining epochs run. No PT : epochs to converge without pretraining. +PT : fine-tuning epochs to reach the same converged loss. Spd. : speedup ratio (No PT / +PT). “–” indicates the pretrained model never reached the baseline loss. Best speedups are bolded .

	IAM				DeepWriting
Method	PT Ep.	No PT	+PT	Spd.	PT Ep.	No PT	+PT	Spd.
Point-5	$36$	$41$	$20$	$2.0$ $\times$	$55$	$61$	–	–
RelTokens	$72$	$80$	$1$	$80.0$ $\times$	$99$	$98$	$18$	5.4 $\times$
TextTokens	$99$	$87$	$4$	$21.8$ $\times$	$100$	$99$	$19$	$5.2$ $\times$
ScribeTokens (Ours)	$90$	$83$	$1$	83.0 $\times$	$99$	$96$	$27$	$3.6$ $\times$

Appendix

Table 6: Convergence speedup from pretraining for HTG. PT Ep. : pretraining epochs run. No PT : epochs to converge without pretraining. +PT : fine-tuning epochs to reach the same converged loss. Spd. : speedup ratio (No PT / +PT). “–” indicates the pretrained model never reached the baseline loss. Best speedups are bolded .

Figure 6: Validation cross-entropy loss during HTR training on IAM without pretraining. RelTokens and ScribeTokens exhibit a double descent pattern, recovering after an initial loss increase, while TextTokens diverges and fails to learn.

Figure 7: Attention rollout for the TextTokens HTR model (82.00% CER, 0% accuracy on IAM). Each subplot shows the rolled-out attention from the most recently decoded character token onto the ink input at a different decoding step. The attention is diffuse with no discernable correspondence to the target character. Attention split: 50.7% ink, 49.3% text.

Figure 8: Attention rollout for the ScribeTokens HTR model (13.15% CER, 22.79% accuracy on IAM). Compared to TextTokens (Figure 7 ), the attention exhibits a clear left-to-right pattern, concentrating slightly ahead of the character being recognized, consistent with causal lookahead. Attention split: 66.9% ink, 33.1% text.

Figure 9: Attention rollout for the ScribeTokens HTR model with pretraining (8.27% CER, 32.93% accuracy on IAM). The attention loses all spatial localization, making the heatmaps uninterpretable. Compared to the non-pretrained model (Figure 8 ; 66.9% ink, 33.1% text), pretraining shifts attention almost entirely to ink. Attention split: 91.2% ink, 8.8% text.

Figure 10: Generated ink samples from all four representations trained on IAM, under autoregressive decoding (temperature 1); all generations are post-processed (Appendix A ). Columns correspond to different text prompts; rows alternate between models trained from scratch and models initialized with next-ink-token prediction pretraining (+PT). Without pretraining, generations are frequently incomplete or illegible; pretraining substantially improves legibility across all methods, with ScribeTokens + PT being the only method to correctly generate both prompts.

Online Handwriting Recognition

Online Handwriting Recognition turns time-ordered pen movements or sensor signals into characters and words.

See all 18 papers

InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
Feb 8, 2024Blagoj Mitrevski, Arina Rak, Julian Schnitzler +4Generative Modeling
Google DeepMind · EPFL
BoundInk: Boundary-Aware Online Handwriting Generation
Apr 2, 2026Jinsu Shin, Sungeun Hong, JinYeong BakConditional Generative Modeling
Sungkyunkwan University, Suwon, Korea
VecGlypher: Unified Vector Glyph Generation with Language Models
Feb 25, 2026Xiaoke Huang, Bhavul Gauri, Kam Woh Ng +12Vector Graphics Generation Text-to-SVG Generation
Meta AI · UC, Santa Cruz

Feb 8, 2024J/K move · Enter open · S save

InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li+3

Google DeepMind · EPFL

Generative Modeling

Open paper AskPDF arXiv

Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.

Abstract

Online Handwriting Recognition

Explore similar work

InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

BoundInk: Boundary-Aware Online Handwriting Generation

VecGlypher: Unified Vector Glyph Generation with Language Models