DualTSR: Unified Dual-Diffusion Transformer for Scene Text Image Super-Resolution

Axi Niu; Kang Zhang; Qingsen Yan; Hao Jin; Jinqiu Sun; Yanning Zhang

cs.CVMar 15, 2026

DualTSR: Unified Dual-Diffusion Transformer for Scene Text Image Super-Resolution

Authors: Axi Niu, Kang Zhang, Qingsen Yan, Hao Jin, Jinqiu Sun, Yanning Zhang

Organizations: Northwestern Polytechnical University · Korea Advanced Institute of Science and Technology

Abstract

Scene Text Image Super-Resolution (STISR) aims to restore high-resolution details in low-resolution text images, which is crucial for both human readability and machine recognition. Existing methods, however, often depend on external Optical Character Recognition (OCR) models for textual priors or rely on complex multi-component architectures that are difficult to train and reproduce. In this paper, we introduce DualTSR, a unified end-to-end framework that addresses both issues. DualTSR employs a single multimodal transformer backbone trained with a dual diffusion objective. It simultaneously models the continuous distribution of high-resolution images via Conditional Flow Matching and the discrete distribution of textual content via discrete diffusion. This shared design enables visual and textual information to interact at every layer, allowing the model to infer text priors internally instead of relying on an external OCR module. Compared with prior multi-branch diffusion systems, DualTSR offers a simpler end-to-end formulation with fewer hand-crafted components. Experiments on synthetic Chinese benchmarks and a curated real-world evaluation protocol show that DualTSR achieves strong perceptual quality and text fidelity.

Figures & tables

Figure 1: A comparison of text super-resolution architectures. Our proposed method unifies image generation and text recognition within a single model. (a) Baseline: Directly models the conditional distribution of high-resolution images from low-resolution inputs, without a text-based prior. (b) DiffTSR [ 44 ] : Independently models the image and text priors, subsequently fusing their representations using a Multi-modality Module (MoM). (c) Proposed Method: A unified multimodal transformer jointly optimizes for image generation and text recognition through two distinct but interconnected diffusion processes.

Figure 2: Illustration of the joint attention mechanism. Following the MM-DiT block design in SD3 [ 6 ] , we incorporate a joint attention module that enables the image latent refinement and text reconstruction processes to operate in parallel.

Method	$\times$ 2					$\times$ 4
Method	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ACC $\uparrow$	NED $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ACC $\uparrow$	NED $\uparrow$
ESRGAN [ 38 ]	24.09	0.3046	15.06	67.08%	83.79%	22.18	0.3986	18.25	43.69%	62.15%
MSRResNet [ 42 ]	28.03	0.3030	30.47	68.94%	85.37%	24.52	0.4029	50.60	49.04%	67.96%
SwinIR [ 16 ]	28.39	0.2983	31.54	70.01%	86.10%	24.73	0.3957	50.89	50.09%	68.93%
SRFormer [ 47 ]	28.89	0.2829	26.93	70.93%	86.91%	25.05	0.3801	46.23	51.83%	70.70%
MARCONet [ 13 ]	23.14	0.4518	88.06	57.93%	76.43%	25.05	0.3801	46.23	51.83%	70.70%

Table 1: Quantitative comparison on the synthetic CTR-TSR . Best in bold .

Method	$\times\textbf{2}$					$\times\textbf{4}$
Method	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ACC $\uparrow$	NED $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ACC $\uparrow$	NED $\uparrow$
ESRGAN [ 38 ]	20.08	0.3833	103.50	58.80%	86.30%	19.97	0.3721	87.18	58.50%	87.65%
MSRResNet [ 42 ]	19.25	0.3033	43.82	62.80%	89.08%	19.28	0.3590	60.81	58.00%	86.87%
SwinIR [ 16 ]	20.24	0.3810	87.98	59.30%	86.44%	20.23	0.3271	56.77	62.10%	87.55%
SRFormer [ 47 ]	20.85	0.3081	50.81	62.90%	89.47%	20.29	0.3732	83.47	57.50%	85.79%
MARCONet [ 13 ]	18.93	0.4144	87.54	60.58%	87.11%	20.30	0.3732	83.47	57.50%	85.79%

Table 2: Quantitative comparison on RealCE . Best in bold .

Figure 3: Qualitative comparison for the synthetic dataset CTR-TSR-Test with different methods on $\times$ 4 scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 4: Qualitative comparison for RealCE with different methods on $\times$ 4 scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Setting	FID $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	ACC $\uparrow$	NED $\uparrow$
(a) $\mathcal{L}_{\text{Joint-MG}}$	13.73	19.82	0.3804	49.85%	68.28%
(b) $\mathcal{L}_{\text{Joint-MG}}+\mathcal{L}_{\text{TXT}}$	13.39	19.79	0.3782	52.72%	72.17%
(c) $\mathcal{L}_{\text{Joint-MG}}+\mathcal{L}_{\text{TXT}}+\mathcal{L}_{\text{IMG-MG}}$	9.92	20.12	0.3550	53.71%	73.60%

Table 3: Loss ablation study. All models are trained for 300k iterations with a batch size of 128.

Figure 5: Trade-off between perceptual quality and text fidelity across sampling steps. FID improves with more sampling steps, reaching its best value around

\sim\!40

, while NED worsens as text strokes become over-smoothed. We select

4

steps as a balanced operating point.

Figure 6: Trade-off between perceptual quality and text fidelity across sampling steps.

CFG	1	1.45	2	4	6
FID $\downarrow$	16.10	19.59	24.19	35.35	39.49
NED $\uparrow$	76.66%	76.71%	75.95%	73.09%	72.20%

Table 4: Effect of CFG scale on FID and NED. Increasing CFG worsens FID and eventually harms NED.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Feature	DiffTSR	Ours
Training Stages	3-stage training	End-to-end training
Core Components	3 (IDM, TDM, MoM)	1 (Multimodal Transformer)
Auxiliary Models	Pre-trained VAE + text recognizer	Pre-trained VAE

Appendix

Table 5: Comparison of training framework. Our framework consolidates text recognition and super-resolution into a single, end-to-end module, whereas DiffTSR [ 44 ] requires a multi-stage pipeline with multiple components and auxiliary models.

Figure 7: Visual comparison for the synthetic dataset CTR-TSR-Test with different methods on $\times 2$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 8: Visual comparison for the synthetic dataset CTR-TSR-Test with different methods on $\times 2$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 9: Visual comparison for the synthetic dataset CTR-TSR-Test with different methods on $\times 4$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 10: Visual comparison for the real-world dataset RealCE with different methods on $\times 2$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 11: Visual comparison for the real-world dataset RealCE with different methods on $\times 2$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 12: Visual comparison for the real-world dataset RealCE with different methods on $\times 4$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Figure 13: Visual comparison for the real-world dataset RealCE with different methods on $\times 4$ scale. The comparison methods include ESRGAN [ 38 ] , MSRResNet [ 42 ] , SwinIR [ 16 ] , SRFormer [ 47 ] , MARCONet [ 13 ] , MARCONet++ [ 14 ] , DiffTSR [ 44 ] and our method.

Explore similar work

Coupled Continuous-Discrete Generation for Scene Text Image Super-Resolution
Aug 5, 2026Axi Niu, Knag Zhang, Qingsen Yan +3Image Super-Resolution Diffusion Models
Northwestern Polytechnical University, Xi’an, China · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
TOLA: Text-aware One-Step Latent Adaptation for Diffusion-based Text Image Super-Resolution
Sep 24, 2026Yike Xu, Yue Shi, Yong Guo +1Diffusion Models for Image Restoration Image Super-Resolution
Shanghai Jiao Tong University · University of Chinese Academy of Sciences · Shanghai AI Laboratory
TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution
Oct 9, 2026Zhiming Hu, Ran Zhang, Angela Ye +7Image Super-Resolution Diffusion Models for Image Restoration
AIC-Toronto, Samsung Electronics · AMD · University of Toronto +2

Aug 5, 2026J/K move · Enter open · S save

Coupled Continuous-Discrete Generation for Scene Text Image Super-Resolution

Axi Niu, Knag Zhang, Qingsen Yan, Hao Jin+2

Northwestern Polytechnical University, Xi’an, China · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

Image Super-Resolution Diffusion Models Multimodal Diffusion Models

Open paper AskPDF arXiv

Scene text image super-resolution (STISR) aims to recover visually plausible appearance while preserving character semantics from degraded inputs. Existing STISR systems often rely on externally generated priors or separate image and text models, resulting in error propagation and costly multi-stage inference. We present DualTSR, a unified framework that formulates STISR as coupled continuous-discrete generation. Conditional flow matching restores continuous image latents, while absorbing-state discrete diffusion reconstructs text tokens. Both processes share a multimodal transformer backbone, allowing the evolving image and text states to interact throughout generation without an external OCR prior at inference. On CTR-TSR, DualTSR achieves the best FID, LPIPS, ACC, and NED among the compared methods at both X2 and X4. On an aligned RealCE subset, it obtains the best FID, ACC, and NED with competitive LPIPS. Compared with DiffTSR at X4, DualTSR improves ACC by 12.78 percentage points while reducing the parameter count from 1.23B to 203M and end-to-end latency from 13.3s to 132ms. These results establish DualTSR as an accurate and efficient method for STISR.

Abstract

Explore similar work

Coupled Continuous-Discrete Generation for Scene Text Image Super-Resolution

TOLA: Text-aware One-Step Latent Adaptation for Diffusion-based Text Image Super-Resolution

TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution