Rethinking Supervision in Document Unwarping: A Self-consistent Flow-free Approach
Authors: Shaokai Liu, Hao Feng, Wengang Zhou
Organizations: MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei, 230027, China.
In recent years, the proliferation of smartphones has led to an upsurge in the digitization of document files via these portable devices. However, images captured by smartphones often suffer from distortions, thereby negatively affecting digital preservation and downstream applications. To address this issue, we introduce DRNet, a novel deep network for document image rectification. Our approach is based on three key designs. Firstly, we exploit the intrinsic geometric consistency inherent in document images to guide the learning process of distortion rectification. Secondly, we design a coarse-to-fine rectification network to leverage the representations derived from the distorted document image, thereby enhancing the rectification result. Thirdly, we propose a unique perspective for supervising the learning of rectification networks, where undistorted document images are employed for supervision, which is free of warping mesh as ground truth in existing methods. Technically, both low-level pixel alignment and high-level semantic alignment jointly contribute to the learning of the mapping relationship between deformed document images and distortion-free ones. We evaluate our method on the challenging DocUNet Benchmark dataset, where it sets a series of state-of-the-art records, demonstrating the superiority of our approach compared to existing learning-based solutions. Additionally, we conduct a comprehensive series of ablation experiments to further validate the effectiveness and merits of our method.
Figures & tables
Fig. 1: An illustration of rectified results of our DRNet. The three rows show the distorted images due to various folds and creases, the predicted warping mesh, and our rectified images, respectively.
Fig. 2: An overview of our proposed DRNet. Given a distorted document image Id , DRNet first extracts its features with a pre-trained transformer-based backbone network. Thereafter, the obtained features are fed into a deformation estimation network that performs distortion rectification via a coarse-to-fine manner. Besides, in DRNet, distortion-free document images are used for supervision without the need for warping mesh as ground truth. The dashed line signifies a process that is exclusive to the training phase, while the solid line indicates a process that occurs during the inference phase.
Fig. 3: An illustration of the proposed consistency loss to facilitate document image rectification.
Methods
Venue
MSSIM ↑
LD ↓
ED ↓
CER ↓
Distorted
/
0.25
20.51
2111.56
0.5352
DocUNet [ 40 ]
CVPR’18
0.41
14.19
1933.66
0.4632
DocProj [ 49 ]
TOG’19
0.29
18.01
1712.48
0.4267
FCN-based [ 42 ]
DAS’20
0.45
7.84
1792.60
0.4213
DewarpNet [ 35 ]
ICCV’19
0.47
8.39
885.90
0.2373
DocTr [ 37 ]
MM’21
0.51
7.76
724.84
0.1832
TABLE I: Quantitative comparisons with existing methods on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite. The best and second-best results are highlighted in bold and underlined , respectively.
Method
MSSIM ↑
LD ↓
ED ↓
CER ↓
Baseline
+ Self-consistency loss Lc
+ Refinement
(a)
✓
0.50
7.63
801.52
0.2019
(b)
✓
✓
0.52
7.57
722.14
0.1808
(c)
✓
✓
✓
0.51
7.42
644.48
0.1649
TABLE II: Ablation experiments of the training strategy of DRNet in terms of image similarity, distortion metrics, and OCR performance on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite.
Method
MSSIM ↑
LD ↓
ED ↓
CER ↓
loss on warping mesh
pixel-wise L1 loss Lp
VGG loss Lv
(a)
✓
0.51
7.58
712.12
0.1682
(b)
✓
0.49
8.81
1135.50
0.2887
(c)
✓
0.50
7.69
804.76
0.2036
(d)
✓
✓
0.51
7.42
644.48
0.1649
TABLE III: Ablation experiments of supervision setting of DRNet in terms of image similarity, distortion metrics, and OCR performance on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite.
Refinement Times
MSSIM ↑
LD ↓
ED ↓
CER ↓
baseline
0.52
7.57
722.14
0.1808
1
0.51
7.42
644.48
0.1649
2
0.52
7.58
715.82
0.1820
3
0.52
7.69
757.24
0.1796
TABLE IV: Ablation experiments of the refinement times in terms of image similarity, distortion metrics, and OCR performance on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite.
Fig. 4: Robustness experiments about the shooting view. The first row shows the input distorted images captured with various shooting views, and the second row presents the rectified images of our DRNet.
Fig. 5: Robustness experiments about the illumination. The first row shows the input distorted images captured under different illumination conditions, and the second row presents the rectified images of our DRNet.
Fig. 6: Qualitative results of the proposed DRNet on various real-world distorted document images. Both printed and handwritten cases are included. The first and third rows present the distorted document images, and the second and fourth rows illustrate the corresponding rectified results.
Fig. 7: Qualitative comparisons with existing learning-based methods, including original distorted images, DocUNet [ 40 ] , DewarpNet [ 35 ] , DocTr [ 37 ] , PWUNet [ 36 ] , DocGeoNet [ 45 ] , Marior [ 48 ] , RDGR [ 38 ] , PaperEdge [ 47 ] , and our DRNet, from left to right.
Fig. 8: Failure cases of DRNet. The two rows display input distorted and corresponding rectified images, respectively. When the input distorted images have only incomplete (left two columns) or no document boundaries (right two columns), the rectified images of DRNet remain partially distorted.
Fig. 9: Qualitative comparisons with the prevalent techniques in smartphones. The two rows show input distorted and corresponding rectified images. Different from the prevalent techniques in smartphones, the rectified images of our method reveal less distortion.
Tianmushan Laboratory, Beihang University, Hangzhou, China · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology, Beijing, China · School of Astronautics, Beihang University, Beijing, China +2