Multi-Modal Character Localization and Extraction for Chinese Text Recognition

Qilong Li; Chongsheng Zhang

cs.CVMar 14, 2026

Multi-Modal Character Localization and Extraction for Chinese Text Recognition

Authors: Qilong Li, Chongsheng Zhang

Organizations: School of Computer and Information Engineering and the Henan Provincial Key Laboratory of Big Data Analysis and Processing, Henan University, Kaifeng, P.R. China.

Abstract

Scene text recognition (STR) methods have demonstrated their excellent capability in English text images. However, due to the complex inner structures of Chinese and the extensive character categories, it poses challenges for recognizing Chinese text in images. Recently, studies have shown that the methods designed for English text recognition encounter an accuracy bottleneck when recognizing Chinese text images. This raises the question: Is it appropriate to apply the model developed for English to the Chinese STR task? To explore this issue, we propose a novel method named LER, which explicitly decouples each character and independently recognizes characters while taking into account the complex inner structures of Chinese. LER consists of three modules: Localization, Extraction, and Recognition. Firstly, the localization module utilizes multimodal information to determine the character's position precisely. Then, the extraction module dissociates all characters in parallel. Finally, the recognition module considers the unique inner structures of Chinese to provide the text prediction results. Extensive experiments conducted on large-scale Chinese benchmarks indicate that our method significantly outperforms existing methods. Furthermore, extensive experiments conducted on six English benchmarks and the Union14M benchmark show impressive results in English text recognition by LER. Code is available at https://github.com/Pandarenlql/LER.

Figures & tables

Fig. 1: (a) Autoregressive decoder and parallel decoder. (b) The forward propagation of our LER.

<s>

</s>

<p>

C

, and

P

denote the start of the sentence, the end of the sentence, pad, character, and position, respectively.

T

denotes the text query proposed in this paper,

A

denotes the visual feature obtained by localization module,

F

denotes the independent character feature. Capital letters (e.g.,

T

) denote features, while lowercase letters (e.g.,

t

) represent vectors within the features. Subscripts indicate the index of vectors (e.g.,

t_{i}

). The blue flow denotes the visual feature obtained by the image encoder.

Fig. 2: (a) The spatial structure of Chinese characters. (b) Illustration of character decomposition.

Fig. 3: The proposed LER framework and IDS decoder, $N$ and $M$ denote the number of blocks in the LER network and the number of char cutter blocks in the extraction module, respectively.

Fig. 4: Multimodal Localization Block. (a) The CLIP’s text feature. (b) The structure of the Multimodal Localization Block (MLB).

Vis

T

P

A

, and

C

denote the visual feature, the text query, the position embedding, the localization feature, and character, respectively. Superscript denotes the index of MLB in the localization module.

Model	Operator	Out Size	Depth	Head	Dim
Image Encoder	Conv Mix	(H/4,W/4)	6	-	128
Localization	MLB	(H/4,W/4)	6	4	128
Extraction	Char cutter	( $c_{h}$ , $c_{w}$ )	3	8	256
IDS Decoder	TDB	( $L_{ids}$ , $n_{ids}$ )	6	8	384

TABLE I: Architecture specifications of LER. TDB denotes the transformer decoder block.

Method	Dataset				Average(%)	Parameters(M)
Method	Scene	Web	Document	Handwriting	Average(%)	Parameters(M)
CRNN [ 18 ]	53.41 / 0.712	57.00 / 0.716	96.62 / 0.992	50.83 / 0.814	64.47 / 0.809	12.4
ASTER [ 22 ]	61.34 / 0.815	51.67 / 0.715	96.19 / 0.991	37.00 / 0.683	61.55 / 0.801	27.2
MORAN [ 42 ]	54.61 / 0.684	31.47 / 0.446	86.10 / 0.962	16.24 / 0.305	47.10 / 0.559	28.5
SAR [ 23 ]	59.67 / 0.766	58.03 / 0.716	95.67 / 0.988	36.49 / 0.736	62.48 / 0.802	27.8
SEED [ 43 ]	44.72 / 0.681	28.06 / 0.460	91.38 / 0.980	20.97 / 0.475	46.30 / 0.549	36.1

TABLE II: Comparison with previous methods on the CTR benchmark. The metrics are LACC / NED.

Type	Method	IIIT	IC13	SVT	IC15	SVTP	CUTE	Avg	Paramters(M)
CTC	CRNN [ 18 ]	90.8	91.8	83.8	71.8	70.4	80.9	81.6	8.3
CTC	SVTR [ 19 ]	95.9	95.5	92.4	83.9	85.7	93.1	91.1	24.6
AR	MORAN [ 42 ]	96.7	94.6	91.7	84.6	85.7	90.3	90.6	17.4
	ASTER [ 22 ]	96.1	94.9	93.0	86.1	87.9	92.0	91.7	19.0
	DAN [ 25 ]	97.5	96.5	94.7	87.1	89.1	94.4	93.2	27.7
	RobustScanner [ 48 ]	98.5	97.7	95.8	88.2	90.1	97.6	94.7	48.0

TABLE III: Results on the six common English benchmarks, bond and underline values denote the 1st and 2nd results in each column.

Type	Method	Curve	Multi-	Artistic	Context	Salient	Multi-	General	Avg	Parameters(M)
Type	Method		Oriented		-less		Words		Avg	Parameters(M)
CTC	CRNN [ 18 ]	19.4	4.5	34.2	44.0	16.7	35.7	60.4	30.7	8.3
CTC	SVTR [ 19 ]	72.4	68.2	54.1	68.0	71.4	67.7	77.0	68.4	24.6
AR	MORAN [ 42 ]	51.2	15.5	51.3	61.2	43.2	64.1	69.3	50.8	17.4
	ASTER [ 22 ]	70.9	82.2	56.7	62.9	73.9	58.5	76.3	68.8	19.0
	DAN [ 25 ]	74.9	63.3	63.4	70.6	70.2	71.1	76.8	70.1	27.7

TABLE IV: Results on the Union14M benchmarks, bond and underline values denote the 1st and 2nd results in each column.

Localization	Extraction	Recognition		Handwriting	Time (ms)
Localization	Extraction	Char	IDS	Handwriting	Time (ms)
$\checkmark^{-}$				56.53 / 0.847	6.1
$\checkmark$				57.22 / 0.850	6.1
$\checkmark$		$\checkmark$		57.34 / 0.851	6.3
$\checkmark$	$\checkmark$	$\checkmark$		57.75 / 0.855	6.9
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	59.29 / 0.871	6.9

TABLE V: Effectiveness of each module in the LER network.

\checkmark^{-}

means without CLIP features in the localization module.

$M$	Handwriting	Parameters(M)
1	58.40 / 0.862	21.8
3	59.29 / 0.871	25.2
6	59.32 / 0.865	30.7

TABLE VI: Effectiveness of char cutter depth.

Models	[ $D_{0},D_{1},D_{2}$ ]	Heads	Params(M)	LACC / NED
LER-S	[96,192,256]	[3,6,8]	16.4	57.31 / 0.858
LER-B	[128,256,384]	[4,8,12]	25.2	59.29 / 0.871
LER-L	[192,256,512]	[6,8,16]	32.1	60.34 / 0.886

TABLE VII: Effectiveness of different model sizes. “S”, “B” and “L” denote small, base and large, respectively. All experiments are conducted on the handwriting scenario of the CTR benchmark dataset.

$c_{h}$	$c_{w}$	Handwriting
3	3	59.22 / 0.864
4	3	58.87 / 0.861
4	4	59.29 / 0.871

TABLE VIII: Effectiveness of different character size.

Fig. 5: Visualization of the character localization.

Training strategy	Handwriting
End-to-End	56.36 / 0.852
Two stage	59.29 / 0.871

TABLE IX: Effectiveness of training strategy.

Fig. 6: Visualization of the character localization on a long text instance.

Fig. 7: Visualization of the predictions from different methods. Red character denotes the error prediction.

Fig. 8: The text images with errors predicted by the localization module are correctly recognized by the complete LER model. Red character denotes the error prediction.

Fig. 9: Error predictions from LER, the red box highlights falsely recognized characters, while red text denotes incorrect predictions.

Fig. 10: Some error labels in the CTR benchmark result in LER’s correct predictions being marked as errors. Red character denotes incorrect labels.

Explore similar work

TextVerifier: Seeing Beyond Recognition to Verify Chinese Text Rendering
Oct 14, 2026Yiyang Chen, Guo-Hua Wang, Yaoli Liu +4T2I Generation Evaluation T2I Diffusion Models
State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China · Alibaba Group · State Key Lab of CAD&CG, Zhejiang University
Towards Universal Khmer Text Recognition
Feb 28, 2026Marry Kong, Rina Buoy, Sovisal Chenda +3Optical Character Recognition Low-Resource Language Processing
Techo Startup Center, Ministry of Economy and Finance, Cambodia · Osaka Metropolitan University, Japan
DTRNet: Dual Text-Radical Decoding for Handwritten Chinese Text Recognition with Faked Character Detection
Aug 6, 2026Runrui Li, Lin Zhu, Hua HuangOptical Character Recognition
School of Artificial Intelligence Beijing Normal University Beijing Key Laboratory of Intelligence for Education Engineering Research Center of Intelligent Technology and Educational Application Beijing, China

Oct 14, 2026J/K move · Enter open · S save

TextVerifier: Seeing Beyond Recognition to Verify Chinese Text Rendering

Yiyang Chen, Guo-Hua Wang, Yaoli Liu, Qing-Guo Chen+3

State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China · Alibaba Group · State Key Lab of CAD&CG, Zhejiang University

T2I Generation Evaluation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image models increasingly generate readable Chinese text that still contains local rendering defects, such as a character with an extra or missing stroke or a malformed component. Such errors are invisible to OCR and vision-language models, which are trained for recognition and are therefore invariant to the very structural deviations that define a rendering error. We formalize text rendering verification as a target-conditioned problem: given a target character and an image crop, decide whether the crop is a correct rendering of that character. Our verifier TEXTVERIFIER feeds the text tower a character's Ideographic Description Sequence (IDS) rather than a single-character symbol, and is trained on 12.7M crop-IDS pairs plus label-free "same-character, different-quality" hard pairs sampled from the training trajectory of a text-region inpainting model. To evaluate this ability we build TEXTRENDERBENCH-ZH, the first per-character benchmark for generated Chinese text, annotating position, identity, and correctness for 7,358 characters. TEXTVERIFIER reaches an F1 of 0.844, far above OCR confidence (0.526), a general VLM (0.329), and TextPecker (0.590). Used as a per-character reward for Flow-GRPO post-training, it further reduces rendering errors, showing that its score can both detect and correct errors.

Abstract

Explore similar work

TextVerifier: Seeing Beyond Recognition to Verify Chinese Text Rendering

Towards Universal Khmer Text Recognition

DTRNet: Dual Text-Radical Decoding for Handwritten Chinese Text Recognition with Faked Character Detection