A Hybrid Vision Transformer Approach for Mathematical Expression Recognition
Authors: Anh Duy Le, Van Linh Pham, Vinh Loi Ly, Nam Quan Nguyen, Huu Thang Nguyen, Tuan Anh Tran
Organizations: Viettel Cyberspace Center, Viettel Group, Vietnam. Lot D26 Cau Giay New Urban Area, Yen Hoa Ward, Cau Giay District, Hanoi, Vietnam. · Faculty of Computer Science & Engineering, Ho Chi Minh City-University of Technology (HCMUT), 268 Ly Thuong Kiet Street, District 10, Ho Chi Minh City, Vietnam. · Vietnam National University Ho Chi Minh City, Linh Trung Ward, Thu Duc District, Ho Chi Minh City, Vietnam.
One of the crucial challenges taken in document analysis is mathematical expression recognition. Unlike text recognition which only focuses on one-dimensional structure images, mathematical expression recognition is a much more complicated problem because of its two-dimensional structure and different symbol size. In this paper, we propose using a Hybrid Vision Transformer (HVT) with 2D positional encoding as the encoder to extract the complex relationship between symbols from the image. A coverage attention decoder is used to better track attention's history to handle the under-parsing and over-parsing problems. We also showed the benefit of using the [CLS] token of ViT as the initial embedding of the decoder. Experiments performed on the IM2LATEX-100K dataset have shown the effectiveness of our method by achieving a BLEU score of 89.94 and outperforming current state-of-the-art methods.
Figures & tables
Fig. 1: A general overview of our proposed framework
Fig. 2: The proposed architecture pipeline and illustration of ViT block.
Acc
BLEU-4
TED
params
Component
Experiment
%
%
%
×106
Feat.
Baseline
22.15
79.12
77.32
7.05
V1
37.55
85.90
89.90
45.76
Context.
Baseline
22.15
79.12
77.32
7.05
V2
34.97
87.73
92.23
12.13
V3
43.69
90.61
94.52
9.75
TABLE III: Ablation experiments on different key components of our approach evaluate on IM2LATEX-100K validation set. Particularly, Feat. denote feature extraction, Context. denote context modeling, and Pred. denote prediction.
Positional Encoding
Acc
BLEU-4
IED
EMA (w/o space)
1D
45.78
88.84
89.31
82.93
2D
48.39
89.94
92.23
86.48
TABLE IV: Comparison between two positional encoding method when apply to our proposed method.
Use init
Acc
BLEU-4
IED
EMA (w/o space)
✓
48.39
89.94
92.23
86.48
✗
43.87
81.73
89.36
81.02
TABLE V: Comparison on two initial hidden state settings when apply to our proposed method.
Fig. 3: Comparison between our proposed method and the baseline model at different sequence length.
Fig. 4: Examples of self-attention map of [CLS] token embedding.
Fig. 5: Visualization of a step-by-step decoding process using our method on example math expression image.
Handwritten Mathematical Expression Generation (HMEG) is challenging due to the complex two-dimensional layouts and long-range structural dependencies of mathematical expressions. Existing methods typically rely on explicit spatial supervision, such as symbol-level bounding boxes, which incurs high annotation costs and limits scalability. In this work, we propose DiffMath, a symbol- and graph-aware latent diffusion framework that leverages the hierarchical structure inherent in LaTeX as a structural prior, eliminating the need for positional supervision. First, we design a Relational Abstract Syntax Tree (RelAST), a generation-oriented representation that distills MathML trees into compact triplet sequences [S, R, D], where each token directly encodes a symbol identity, spatial relation, or nesting depth. Second, we introduce MathVAE, which learns structure-preserving latent representations through symbol-aware and relation-aware perceptual regularization, ensuring that the latent space captures both character semantics and spatial topology. Third, MathDiT performs conditional denoising in this structured latent space, further guided by a global symbol-count prior via Adaptive Layer Normalization (AdaLN) to improve structural coherence. Experiments show that DiffMath produces structurally consistent handwritten expressions, achieves superior performance over existing methods, and improves the accuracy of downstream OCR models through synthetic data augmentation.
Wei Pan, Xuhan Zheng, Yilin Shi +5
South China University of Technology · Huawei Technologies Co., Ltd.
Handwritten mathematical expression recognition (HMER) requires reasoning over diverse symbols and structures, yet autoregressive models struggle with exposure bias and syntax inconsistency. We present GryphOne, a discrete diffusion framework which reformulates HMER as iterative symbolic refinement instead of sequential generation. GryphOne progressively refines symbols and relations, removing autoregression and improving consistency. Symbol-aware tokenization and random-masking mutual learning further enhance robustness to handwriting diversity. On the MathWriting benchmark, GryphOne achieves 5.51% CER and 59.9% EM (ExpRate), outperforming all reimplemented models in the matched setting as well as the commercial HMER system. Held-out evaluation on CROHME 2014-2023 further shows strong cross-dataset generalization.
Takaya Kawakatsu, Ryo Ishiyama
Preferred Networks, Inc., Otemachi, Tokyo, Japan · Kyushu University, Fukuoka, Japan
Handwritten mathematical expression recognition (HMER) is conventionally scored by exact-match rates and string-similarity metrics that are blind to where an error occurs: two predictions with identical token-error counts receive identical scores whether they misplace a subscript or swap the operands of a fraction. Render-based character detection matching (CDM) aligns glyphs robustly but remains position-blind---on controlled fraction-operand swaps it scores 0.8595 where position-aware scoring yields 0.6253. Tree-edit metrics exhibit a complementary blind spot: rewrites outside the parser's normalization coverage are penalized as structural errors (0.8552 where render-based metrics score 1.0). We propose PA-CDM, a position-aware metric that couples character detection matching with position-forest encoding and divergence-level weighting; StructPerturb v2.0, a frozen benchmark of 1,340 controlled perturbation pairs across 15 type--intensity cells; and a cross-metric consistency protocol combining a sensitivity matrix, a human study, and LLM-judge calibration. In a six-annotator study, PA-CDM attains the highest correlation with human judgments among seven automatic metrics (Spearman rho=0.9535, n=990). A frontier LLM judge correlates slightly higher (rho=0.9613) but is costly, nondeterministic, and API-dependent; PA-CDM approaches it at zero marginal cost with deterministic, diagnosable behavior.