CONSTANT: Towards High-Quality One-Shot Handwriting Generation with Patch Contrastive Enhancement and Style-Aware Quantization
Organizations: Viettel Artificial Intelligence and Data Services Center, Vietnam · Ho Chi Minh City University of Technology, Vietnam
Abstract
One-shot styled handwriting image generation, despite achieving impressive results in recent years, remains challenging due to the difficulty in capturing the intricate and diverse characteristics of human handwriting by using solely a single reference image. Existing methods still struggle to generate visually appealing and realistic handwritten images and adapt to complex, unseen writer styles, struggling to isolate invariant style features (e.g., slant, stroke width, curvature) while ignoring irrelevant noise. To tackle this problem, we introduce Patch Contrastive Enhancement and Style-Aware Quantization via Denoising Diffusion (CONSTANT), a novel one-shot handwriting generation via diffusion model. CONSTANT leverages three key innovations: 1) a Style-Aware Quantization (SAQ) module that models style as discrete visual tokens capturing distinct concepts; 2) a contrastive objective to ensure these tokens are well-separated and meaningful in the embedding style space; 3) a latent patch-based contrastive (LLatentPCE) objective help improving quality and local structures by aligning multiscale spatial patches of generated and real features in latent space. Extensive experiments and analysis on benchmark datasets from multiple languages, including English, Chinese, and our proposed ViHTGen dataset for Vietnamese, demonstrate the superiority of adapting to new reference styles and producing highly detailed images of our method over state-of-the-art approaches. Code is available at GitHub
Figures & tables
| Method | Reference | HWD | FID | WER | |
|---|---|---|---|---|---|
| HWT [ 2 ] | Few-shot | 1.23 | 19.82 | 0.62 | 9.08 |
| VATr [ 36 ] | 1.13 | 16.30 | 0.51 | 49.81 | |
| DiffusionPen [ 33 ] | 1.04 | 18.94 | 0.23 | 38.86 | |
| HiGAN+ [ 11 ] | One-shot | 0.89 | 13.90 | 0.56 | 55.20 |
| HiGAN [ 12 ] | 1.55 | 27.13 | 0.55 | 29.79 | |
| One-DM [ 4 ] | 1.05 | 15.97 | 0.36 | 4.5 |
| IV-S | OOV-S | IV-U | OOV-U | |||||
| Method | HWD | FID | HWD | FID | HWD | FID | HWD | FID |
| HWT [ 2 ] | 2.30 | 135.51 | 2.30 | 146.16 | 2.35 | 138.39 | 2.36 | 148.75 |
| VATr [ 36 ] | 2.50 | 132.87 | 2.51 | 140.56 | 2.58 | 137.34 | 2.60 | 144.02 |
| DiffusionPen [ 33 ] | 1.14 | 91.20 | 1.16 | 97.65 | 1.52 | 112.87 | 1.65 | 122.52 |
| HiGAN [ 12 ] | 2.29 | 118.70 | 2.31 | 128.49 | 2.36 | 119.56 | 2.37 | 128.60 |
| HiGAN+ [ 11 ] | 1.76 | 117.76 | 1.79 | 122.56 | 1.78 | 117.63 | 1.81 | 124.38 |
| Style images | FID | HWD | |||||
| Base | |||||||
| 16.73 | 0.87 | ||||||
| 12.47 | 0.85 | ||||||
| 12.55 | 0.84 | ||||||
| 10.20 | 0.74 | ||||||
| Chinese | Vietnamese | |||
|---|---|---|---|---|
| Method | HWD | FID | HWD | FID |
| One-DM [ 4 ] | 0.48 | 22.97 | 1.08 | 22.53 |
| Our | 0.37 | 22.74 | 0.83 | 18.81 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | ViHTGen (ours) | IAM |
|---|---|---|
| Language | Vietnamese | English |
| # Writers | 223 | 500 |
| # Word Instances | 50000+ | 62857 |
| # Unique words | 4644 | 3332 |
| Style Complexity (Slant, Ink color, Stroke width, character shape) | High | Medium |
| Background complexity | High | Low |
| Method | HWD | FID |
|---|---|---|
| One-DM [ 4 ] | 1.22 | 17.74 |
| Ours | 0.73 | 10.22 |
| Chinese | Vietnamese | |||
|---|---|---|---|---|
| Method | HWD | FID | HWD | FID |
| DiffusionPen [ 33 ] | 0.57 | 54.07 | 1.05 | 23.59 |
| Our | 0.37 | 22.74 | 0.83 | 18.81 |
| Data Source | Accuracy |
|---|---|
| Real data | 81.76 |
| Real data + 100K | 82.13 |
| Real data + 200K | 82.96 |
| IAM | IMGUR5K | IIIT-English-Word | ||||
| Codebook length | HWD | FID | HWD | FID | HWD | FID |
| 512 | 0.86 | 13.80 | 1.02 | 13.37 | 0.75 | 14.02 |
| 1024 | 0.84 | 12.46 | 1.03 | 12.9 | 0.77 | 12.61 |
| 2048 | 0.86 | 14.10 | 0.99 | 11.48 | 0.73 | 10.22 |
| Criteria | CONSTANT | One-DM |
|---|---|---|
| Model parameters | 124 | 185 |
| Training VRAM | 6.82GB | 18.66GB |
| Inference time 7 7 7 Sampling taken under 50 steps | 1.25 s/sample | 1.85 s/sample |
| Inference VRAM | 2.1GB | 2.9GB |
Image Generation
Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.
See all 461 papers