Rethinking Supervision in Document Unwarping: A Self-consistent Flow-free Approach
Authors: Shaokai Liu, Hao Feng, Wengang Zhou
Organizations: MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei, 230027, China.
In recent years, the proliferation of smartphones has led to an upsurge in the digitization of document files via these portable devices. However, images captured by smartphones often suffer from distortions, thereby negatively affecting digital preservation and downstream applications. To address this issue, we introduce DRNet, a novel deep network for document image rectification. Our approach is based on three key designs. Firstly, we exploit the intrinsic geometric consistency inherent in document images to guide the learning process of distortion rectification. Secondly, we design a coarse-to-fine rectification network to leverage the representations derived from the distorted document image, thereby enhancing the rectification result. Thirdly, we propose a unique perspective for supervising the learning of rectification networks, where undistorted document images are employed for supervision, which is free of warping mesh as ground truth in existing methods. Technically, both low-level pixel alignment and high-level semantic alignment jointly contribute to the learning of the mapping relationship between deformed document images and distortion-free ones. We evaluate our method on the challenging DocUNet Benchmark dataset, where it sets a series of state-of-the-art records, demonstrating the superiority of our approach compared to existing learning-based solutions. Additionally, we conduct a comprehensive series of ablation experiments to further validate the effectiveness and merits of our method.
Figures & tables
Fig. 1: An illustration of rectified results of our DRNet. The three rows show the distorted images due to various folds and creases, the predicted warping mesh, and our rectified images, respectively.
Fig. 2: An overview of our proposed DRNet. Given a distorted document image Id , DRNet first extracts its features with a pre-trained transformer-based backbone network. Thereafter, the obtained features are fed into a deformation estimation network that performs distortion rectification via a coarse-to-fine manner. Besides, in DRNet, distortion-free document images are used for supervision without the need for warping mesh as ground truth. The dashed line signifies a process that is exclusive to the training phase, while the solid line indicates a process that occurs during the inference phase.
Fig. 3: An illustration of the proposed consistency loss to facilitate document image rectification.
Methods
Venue
MSSIM ↑
LD ↓
ED ↓
CER ↓
Distorted
/
0.25
20.51
2111.56
0.5352
DocUNet [ 40 ]
CVPR’18
0.41
14.19
1933.66
0.4632
DocProj [ 49 ]
TOG’19
0.29
18.01
1712.48
0.4267
FCN-based [ 42 ]
DAS’20
0.45
7.84
1792.60
0.4213
DewarpNet [ 35 ]
ICCV’19
0.47
8.39
885.90
0.2373
DocTr [ 37 ]
MM’21
0.51
7.76
724.84
0.1832
TABLE I: Quantitative comparisons with existing methods on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite. The best and second-best results are highlighted in bold and underlined , respectively.
Method
MSSIM ↑
LD ↓
ED ↓
CER ↓
Baseline
+ Self-consistency loss Lc
+ Refinement
(a)
✓
0.50
7.63
801.52
0.2019
(b)
✓
✓
0.52
7.57
722.14
0.1808
(c)
✓
✓
✓
0.51
7.42
644.48
0.1649
TABLE II: Ablation experiments of the training strategy of DRNet in terms of image similarity, distortion metrics, and OCR performance on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite.
Method
MSSIM ↑
LD ↓
ED ↓
CER ↓
loss on warping mesh
pixel-wise L1 loss Lp
VGG loss Lv
(a)
✓
0.51
7.58
712.12
0.1682
(b)
✓
0.49
8.81
1135.50
0.2887
(c)
✓
0.50
7.69
804.76
0.2036
(d)
✓
✓
0.51
7.42
644.48
0.1649
TABLE III: Ablation experiments of supervision setting of DRNet in terms of image similarity, distortion metrics, and OCR performance on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite.
Refinement Times
MSSIM ↑
LD ↓
ED ↓
CER ↓
baseline
0.52
7.57
722.14
0.1808
1
0.51
7.42
644.48
0.1649
2
0.52
7.58
715.82
0.1820
3
0.52
7.69
757.24
0.1796
TABLE IV: Ablation experiments of the refinement times in terms of image similarity, distortion metrics, and OCR performance on the DocUNet Benchmark dataset [ 40 ] . “ ↑ ” indicates the higher the better and “ ↓ ” means the opposite.
Fig. 4: Robustness experiments about the shooting view. The first row shows the input distorted images captured with various shooting views, and the second row presents the rectified images of our DRNet.
Fig. 5: Robustness experiments about the illumination. The first row shows the input distorted images captured under different illumination conditions, and the second row presents the rectified images of our DRNet.
Fig. 6: Qualitative results of the proposed DRNet on various real-world distorted document images. Both printed and handwritten cases are included. The first and third rows present the distorted document images, and the second and fourth rows illustrate the corresponding rectified results.
Fig. 7: Qualitative comparisons with existing learning-based methods, including original distorted images, DocUNet [ 40 ] , DewarpNet [ 35 ] , DocTr [ 37 ] , PWUNet [ 36 ] , DocGeoNet [ 45 ] , Marior [ 48 ] , RDGR [ 38 ] , PaperEdge [ 47 ] , and our DRNet, from left to right.
Fig. 8: Failure cases of DRNet. The two rows display input distorted and corresponding rectified images, respectively. When the input distorted images have only incomplete (left two columns) or no document boundaries (right two columns), the rectified images of DRNet remain partially distorted.
Fig. 9: Qualitative comparisons with the prevalent techniques in smartphones. The two rows show input distorted and corresponding rectified images. Different from the prevalent techniques in smartphones, the rectified images of our method reveal less distortion.
Reliable document digitization in uncontrolled capture settings is challenging because real images exhibit multiple interacting degradations rather than a single isolated distortion. Documents thus captured are affected simultaneously by geometric distortions, like page warping, as well as photometric degradations such as non-uniform illumination, and blurring. However, most existing approaches address these factors independently and are evaluated on benchmarks containing only one distortion type, limiting their real-world applicability. We introduce GutenDoc, a large-scale dataset of high-resolution dense-text documents with physically grounded compound degradations. Using physics-based rendering, our dataset jointly models geometric warping and diverse photometric effects, enabling systematic evaluation under realistic capture conditions. We further propose a unified restoration framework that jointly corrects geometric and photometric distortions within a single-network and single-training setup, without the need for degradation-specific retraining or sequential inference passes. Extensive experiments show that our method remains competitive on established single-distortion benchmarks while substantially improving robustness under compound degradations, providing a practical solution for real-world document digitization.
Sujal Burad, Aakanksha, A. N. Rajagopalan +1
Indian Institute of Technology, Madras · Adobe, India
Image correction and rectangling are valuable tasks in practical photography systems such as smartphones. Recent remarkable advancements in deep learning have undeniably brought about substantial performance improvements in these fields. Nevertheless, existing methods mainly rely on task-specific architectures. This significantly restricts their generalization ability and effective application across a wide range of different tasks. In this paper, we introduce the Unified Rectification Framework (UniRect), a comprehensive approach that addresses these practical tasks from a consistent distortion rectification perspective. Our approach incorporates various task-specific inverse problems into a general distortion model by simulating different types of lenses. To handle diverse distortions, UniRect adopts one task-agnostic rectification framework with a dual-component structure: a {Deformation Module}, which utilizes a novel Residual Progressive Thin-Plate Spline (RP-TPS) model to address complex geometric deformations, and a subsequent Restoration Module, which employs Residual Mamba Blocks (RMBs) to counteract the degradation caused by the deformation process and enhance the fidelity of the output image. Moreover, a Sparse Mixture-of-Experts (SMoEs) structure is designed to circumvent heavy task competition in multi-task learning due to varying distortions. Extensive experiments demonstrate that our models have achieved state-of-the-art performance compared with other up-to-date methods.
Linwei Qiu, Gongzhe Li, Xiaozhe Zhang +2
Tianmushan Laboratory, Beihang University, Hangzhou, China · State Key Laboratory of High-Efficiency Reusable Aerospace Transportation Technology, Beijing, China · School of Astronautics, Beihang University, Beijing, China +2
Deep learning models have become the standard tool for document rectification and illumination correction, yet their performance is fundamentally bound by their training data. For nearly a decade, the community has heavily relied on Doc3D, a pioneering but increasingly limited document unwarping dataset in terms of scale and quality. To address this bottleneck, we introduce SyntheticDoc, a massive, high-quality dataset designed to push the boundaries of document unwarping. SyntheticDoc is composed of 1,000,000 high-resolution procedurally generated training samples, alongside extensive validation and test sets. Each sample is paired with rich, pixel-perfect annotations, including UV maps, normal maps, albedo and shading. To ensure physical accuracy and photorealism, the paper geometries are generated via a physics-based simulator and rendered using a path tracer. To demonstrate the benefit of our dataset, we train a simple baseline model on SyntheticDoc and report on its performance in comparison to state-of-the-art methods on both document unwarping and illumination correction tasks. Our dataset is available at https://igl.ethz.ch/projects/SyntheticDoc/ and the code used to generate it at https://github.com/tanguymagne/SyntheticDoc .
Daniel Woortmann, Tanguy Magne, Olga Sorkine-Hornung
ETH Zurich, Department of Computer Science, Universitätstrasse 6, 8092 Zurich, Switzerland