A High-Accuracy Optical Music Recognition Method Based on Bottleneck Residual Convolutions
Authors: Junwen Ma, Huhu Xue, Xingyuan Zhao, and Weicheng Fu
Organizations: Department of Artificial Intelligence, Tianshui Normal University, Tianshui 741001, Gansu, China. · Department of Physics, Tianshui Normal University, Tianshui 741001, Gansu, China. · School of Music and Dance, Tianshui Normal University, Tianshui 741001, Gansu, China.
Optical Music Recognition (OMR) aims to convert printed or handwritten music score images into editable symbolic representations. This paper presents an end-to-end OMR framework that combines residual bottleneck convolutions with bidirectional gated recurrent unit (BiGRU)-based sequence modeling. A convolutional neural network with ResNet-v2-style residual bottleneck blocks and multi-scale dilated convolutions is used to extract features that encode both fine-grained symbol details and global staff-line structures. The extracted feature sequences are then fed into a BiGRU network to model temporal dependencies among musical symbols. The model is trained using the Connectionist Temporal Classification loss, enabling end-to-end prediction without explicit alignment annotations. Experimental results on the Camera-PrIMuS and PrIMuS datasets demonstrate the effectiveness of the proposed framework. On Camera-PrIMuS, the proposed method achieves a sequence error rate (SeER) of 7.52% and a symbol error rate (SyER) of 0.45%, with pitch, type, and note accuracies of 99.33%, 99.60%, and 99.28%, respectively. The average training time is 1.74~s per epoch, demonstrating high computational efficiency while maintaining strong recognition performance. On PrIMuS, the method achieves a SeER of 8.11% and a SyER of 0.49%, with pitch, type, and note accuracies of 99.27%, 99.58%, and 99.21%, respectively. A fine-grained error analysis further confirms the effectiveness of the proposed model.
Figures & tables
Fig. 1: Architecture of the proposed end-to-end Optical Music Recognition model based on bottleneck residual convolution and BiGRU. The model consists of a bottleneck residual CNN feature extractor, a BiGRU sequence modeling module, and a fully connected output layer for end-to-end symbol sequence recognition.
Augmentation Type
Probability
Brightness/Contrast/Gamma Adjustment
0.9
Blur Perturbation
0.5
JPEG Compression Artifacts
0.5
Morphological Erosion
0.08
Morphological Dilation
0.08
Small-Angle Rotation and Shearing
0.6
TABLE I: Data augmentation operations and activation probabilities
Fig. 2: Comparison between the original music score image and its augmented versions.
Category
Name
Value
Input
Image height
128
Number of channels
1
Batch size
16
Convolutional encoder
Number of residual blocks
5
Convolution channels
[32, 64, 128, 256, 256]
Kernel size
3×3
TABLE II: Main hyperparameter configuration of the model.
Category
Name
Value
Input
Img. height
128
Num. channels
1
Batch size
16
Conv. encoder
Num. residual blocks
5
Conv. channels
[32, 64, 128, 256, 256]
Kernel size
3×3
TABLE III: Main hyperparameter configuration of the model
Data type
Model
Clean
Distortions
Agnostic
Semantic
Agnostic
Semantic
Clean
CRNN
1.10 / 21.7
0.80 / 12.5
44.3 / 95.1
59.7 / 97.9
R2-CRNN
0.59 / 16.2
0.56 / 13.4
7.63 / 20.9
38.19 / 91.0
Ours
0.49 / 8.66
0.49 / 7.80
23.41 / 87.91
39.05 / 93.6
Distortions
CRNN
1.40 / 24.9
3.30 / 44.6
1.60 / 24.7
3.40 / 38.3
R2-CRNN
0.68 / 18.7
1.06 / 25.4
1.01 / 27.8
1.50 / 35.8
TABLE IV: Recognition performance of different models on the Camera-PrIMuS dataset under Clean and Distortions conditions (without data augmentation, in %)
Model
Agnostic
Semantic
CRNN
1.00 / 17.9
0.80 / 12.5
Ours
0.53 / 9.09
0.50 / 8.33
TABLE V: Recognition performance of different models on the PrIMuS dataset under Clean and Distortions conditions (without data augmentation, in %)
Methods
SeER (%)
SyER (%)
Semantic
7.80
0.49
Semantic+Augmentation
7.52
0.45
Agnostic
8.66
0.49
Agnostic+Augmentation
7.64
0.44
TABLE VI: Performance comparison of our model with and without data augmentation on the Camera-PrIMuS dataset.
Methods
SeER (%)
SyER (%)
Semantic
8.33
0.50
Semantic+Augmentation
8.11
0.49
Agnostic
9.09
0.53
Agnostic+Augmentation
8.55
0.48
TABLE VII: Performance comparison of our model with and without data augmentation on the PrIMuS dataset
Methods
SeER
SyER
Pitch accuracy
Type accuracy
Note accuracy
Training time (s/epoch)
ConvNet-STN
27.0298
9.3975
70.0000
72.3987
68.3274
2.79
DWD
25.1475
7.9030
82.1123
80.9635
78.5064
3.50
MCRR
5.1488
1.0612
90.7544
91.5520
88.2094
1.93
Ours
7.52
0.45
99.33
99.60
99.28
1.74
TABLE VIII: Performance comparison of SOTA models on the Camera-PrIMuS dataset
Methods
SeER
SyER
Pitch accuracy
Type accuracy
Note accuracy
Training time (s/epoch)
ConvNet-STN
16.8056
5.0208
89.7107
94.0273
85.5620
0.98
DWD
18.5609
8.7811
95.2074
96.0049
91.1255
1.21
MCRR
1.4571
0.3234
97.0000
97.1298
94.4447
0.56
Ours
8.11
0.49
99.27
99.58
99.21
1.80
TABLE IX: Performance comparison of SOTA models on the PrIMuS dataset
Category
I
D
N1
N2
OMR-NED (%)
Errors (%)
Barlines
76
190
29,516
29,402
0.45
22.77
Clefs
3
3
7,935
7,935
0.04
0.51
GraceNotes
186
172
2,043
2,057
8.73
30.65
KeySignatures
2
0
6,095
6,097
0.02
0.17
MultiRests
4
5
1,990
1,989
0.23
0.77
Notes
202
209
120,423
120,416
0.17
35.19
TABLE X: Fine-grained OMR-NED analysis across different symbol categories (Semantic encoding) on the Camera-PrIMuS dataset
Category
I
D
N1
N2
OMR-NED (%)
Errors (%)
Accidentals
148
77
15,663
15,734
0.72
18.53
Barlines
66
208
29,619
29,477
0.46
22.57
Clefs
1
1
7,957
7,957
0.01
0.16
Fermatas
1
0
394
395
0.13
0.08
GraceNotes
154
152
2,028
2,030
7.54
25.21
MeterSigns
0
1
4,230
4,229
0.01
0.08
TABLE XI: Fine-grained OMR-NED analysis across different symbol categories (Agnostic encoding) on the Camera-PrIMuS dataset.
Category
I
D
N1
N2
OMR-NED (%)
Errors (%)
Barlines
90
219
29,413
29,284
0.53
22.91
Clefs
4
2
7,950
7,952
0.04
0.44
GraceNotes
207
200
2,102
2,109
9.67
30.17
KeySignatures
4
3
6,054
6,055
0.06
0.52
MultiRests
5
5
1,947
1,947
0.26
0.74
Notes
239
243
120,676
120,672
0.20
35.73
TABLE XII: Fine-grained OMR-NED analysis across different symbol categories (Semantic encoding) on the PrIMuS dataset
Category
I
D
N1
N2
OMR-NED (%)
Errors (%)
Accidentals
140
61
15,415
15,494
0.65
16.24
Barlines
93
178
29,347
29,262
0.46
21.89
Clefs
1
0
7,953
7,954
0.01
0.08
Fermatas
5
0
406
411
0.61
0.40
GraceNotes
210
201
2,232
2,241
9.19
33.20
MeterSigns
0
0
4,233
4,233
0.00
0.00
TABLE XIII: Fine-grained OMR-NED analysis across different symbol categories (Agnostic encoding) on the PrIMuS dataset
This paper presents an optical music recognition (OMR) system tailored for PDF scores of string quartet pieces. The approach begins with a SegNet-based segmentation model to localize musical regions, followed by convolutional neural networks and image processing techniques to classify and extract musical symbols. To improve rhythmic precision, dynamic programming methods enforce time signature constraints and explicitly handle tuplets. Evaluation using a longest common subsequence metric across multiple feature matrices demonstrates robust accuracy in recognizing pitch, onset, duration, and accidentals on a benchmark dataset. Code and data are available at https://github.com/lattellie/25-omr.
Jin-Shiang Hu, Bo-Yuan Cheng, Li Su
Academia Sinica, Taiwan · University of British Columbia · National Taiwan University
Optical Music Recognition (OMR) has seen major progress in model design, with end-to-end methods now capable of recognising notation at all levels of complexity. However, the impact of this progress has been limited by the visual domains of available training datasets, which are largely born-digital. Existing large collections of sheet music in libraries and other heritage institutions contain predominantly manuscripts, whose visual domains are highly diverse and different, so existing OMR systems fail when applied in the real world. These institutions are often resource-constrained, so large in-domain datasets cannot be expected. We provide a first baseline on real-world manuscripts with complex piano notation in the resource-constrained scenario. Using fine-grained music notation graph (MuNG) annotations and the Smashcima synthesis tool, we then show that while some direct transcriptions of in-domain data remain essential, domain adaptation using synthetic musical manuscript images brings significant improvement. Furthermore, the symbols used do not need to be in-domain, so the expensive fine-grained annotation can be avoided. We thus bring OMR closer to one of its stated goals: preserving and promoting musical cultural heritage.
Jiří Mayer, Martina Dvořáková, Vojtěch Dvořák +6
Institute of Formal and Applied Linguistics, Charles University · Moravian Library
We propose Legato, a new end-to-end model for optical music recognition (OMR), a task of converting music score images to machine-readable documents. Legato is the first large-scale pretrained OMR model capable of recognizing full-page or multi-page typeset music scores and the first to generate documents in ABC notation, a concise, human-readable format for symbolic music. Bringing together a pretrained vision encoder with an ABC decoder trained on a dataset of more than 214K images, our model exhibits the strong ability to generalize across various typeset scores. We conduct comprehensive experiments on a range of datasets and metrics and demonstrate that Legato outperforms the previous state of the art. On our most realistic dataset, we see a 68.2% and 48.1% absolute error reduction on the standard metrics TEDn and OMR-NED, respectively.
Guang Yang, Victoria Ebert, Nazif Can Tamer +3
Paul G. Allen School of Computer Science & Engineering, University of Washington · Allen Institute for AI