Structure-Level Disentangled Diffusion for Few-Shot Chinese Font Generation
Authors: Jie Li, Suorong Yang, Jian Zhao, Furao Shen
Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Department of Computer Science and Technology, Nanjing University, China · School of Electronic Science and Engineering, Nanjing University, China
Few-shot Chinese font generation aims to synthesize new characters in a target style using only a handful of reference images. Achieving accurate content rendering and faithful style transfer requires effective disentanglement between content and style. However, existing approaches achieve only feature-level disentanglement, allowing the generator to re-entangle these features, leading to content distortion and degraded style fidelity. We propose the Structure-Level Disentangled Diffusion Model (SLD-Font), which receives content and style information from two separate channels. SimSun-style images are used as content templates and concatenated with noisy latent features as the input. Style features extracted by a CLIP model from target-style images are integrated via cross-attention. Additionally, we train a Background Noise Removal module in the pixel space to remove background noise in complex stroke regions. Based on theoretical validation of disentanglement effectiveness, we introduce a parameter-efficient fine-tuning strategy that updates only the style-related modules. This allows the model to better adapt to new styles while avoiding overfitting to the reference images' content. We further introduce the Grey and OCR metrics to evaluate the content quality of generated characters. Experimental results show that SLD-Font achieves significantly higher style fidelity while maintaining comparable content accuracy to existing state-of-the-art methods.
Figures & tables
Figure 1: Existing methods, as illustrated in (a), are typically built upon a framework that achieves feature-level disentanglement. In contrast, our approach achieves structure-level disentanglement as shown in (b), yielding more robust content preservation and style controllability. As demonstrated in (c), our method produces Chinese characters with noticeably higher visual quality.
Figure 2: Both source and target images are encoded into the latent space through a VAE encoder. Gaussian noise is added to the latent target representation during diffusion, producing a noisy latent image that is concatenated with the source and fed into the U-Net. Content and style are disentangled within the U-Net: content components capture structural patterns from the source image, while style components receive CLIP-extracted features from multiple style reference images and inject them via cross-attention. The U-Net output is decoded by the VAE, binarized, and refined in pixel space using the BNR module to produce clean and accurate glyphs.
Figure 3: The VAE output shows background noise in dense stroke areas, which is visible in the grayscale histogram. The BNR module effectively removes the noise.
Figure 4: Results of gradient analysis. Style-related parameters are more sensitive to unseen styles, while content-related parameters are more sensitive to unseen content.
Model
SCUF
UCUF
SSIM ↑
LPIPS ↓
FID ↓
L1 ↓
Grey ↑
OCR ↑
SSIM ↑
LPIPS ↓
FID ↓
L1 ↓
Grey ↑
OCR ↑
LF
0.307
0.275
1.275
0.255
0.381
0.986
0.304
0.283
1.298
0.257
0.380
0.978
MX
0.319
0.256
0.671
0.249
0.285
1.000
0.321
0.259
0.689
0.246
0.283
0.999
NTF
0.359
0.250
0.623
0.229
0.376
0.997
0.352
0.257
0.631
0.232
0.373
0.995
Font-diff (1shot)
0.298
0.270
4.087
0.254
0.518
0.949
-
-
-
-
-
-
FontDiffuser (1shot)
0.347
0.265
2.770
0.237
0.984
1.000
0.341
0.270
2.621
0.240
0.983
0.999
Table 1: Quantitative comparison under SCUF and UCUF settings across diverse evaluation metrics. Since Font-diff only supports SCUF, we do not report the results under UCUF. SCSF : Seen Chars Seen Font. UCUF : Unseen Chars Seen Font.
Figure 5: Visualization of Chinese character generation results under SCUF and UCUF settings, illustrating two fonts for each of six representative style characteristics. Problematic generations are highlighted with red circles.
Figure 6: Visualization of handwritten style test.
Strategy
CLIP
KV
ALL
PEFT
Num. Trainable
0.79M
2.36M
34.67M
7.30M
Time (s)
116.99
120.97
126.9
121.66
Table 2: Comparison of different fine-tuning strategies in terms of parameter size and fine-tuning time for a single font style. Experiments are conducted on a single NVIDIA V100 GPU.
Strategy
BNR
SSIM ↑
LPIPS ↓
Fid ↓
L1 ↓
Grey ↑
OCR ↑
No
✗
0.377
0.222
2.189
0.220
0.896
0.993
CLIP
✗
0.376
0.222
2.187
0.220
0.896
0.992
KV
✗
0.390
0.214
1.781
0.212
0.898
0.994
ALL
✗
0.388
0.210
2.164
0.210
0.510
0.973
PEFT
✗
0.423
0.202
1.719
0.196
0.779
0.986
No
✓
0.394
0.224
1.010
0.219
0.995
0.994
Table 3: Effect of the component in our method. “CLIP” tunes the last CLIP layer, “KV” the K/V projections, “ALL” all parameters, and “PEFT” applies parameter-efficient fine-tuning.