LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR

Guang Yang; Victoria Ebert; Nazif Can Tamer; Brian Siyuan Zheng; Luiza Amador Pozzobon; Noah A. Smith

cs.CVJun 23, 2025

LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR

Authors: Guang Yang, Victoria Ebert, Nazif Can Tamer, Brian Siyuan Zheng, Luiza Amador Pozzobon, Noah A. Smith

Organizations: Paul G. Allen School of Computer Science & Engineering, University of Washington · Allen Institute for AI

Abstract

We propose Legato, a new end-to-end model for optical music recognition (OMR), a task of converting music score images to machine-readable documents. Legato is the first large-scale pretrained OMR model capable of recognizing full-page or multi-page typeset music scores and the first to generate documents in ABC notation, a concise, human-readable format for symbolic music. Bringing together a pretrained vision encoder with an ABC decoder trained on a dataset of more than 214K images, our model exhibits the strong ability to generalize across various typeset scores. We conduct comprehensive experiments on a range of datasets and metrics and demonstrate that Legato outperforms the previous state of the art. On our most realistic dataset, we see a 68.2% and 48.1% absolute error reduction on the standard metrics TEDn and OMR-NED, respectively.

Figures & tables

Figure 1: Model architecture. The input image is first cropped into overlapping segments with an aspect ratio of 1:4 or less, then resized and divided into four patches per segment (§ 4.2.1 ). The image patches are fed into a vision encoder (§ 4.2.2 ; parameters are frozen during training). The resulting latent embeddings serve as cross-attention keys and values in a transformer decoder, which autoregressively generates ABC tokens (§ 4.2.3 ). Special tokens <B> , <I> , and <E> denote <|begin_of_abc|> , <|image|> , and <|end_of_abc|> , respectively. For better visualization, here we use “_” to represent whitespace.

Figure 2: An example of our canonical ABC representation (below) with a MusicXML-rendered image (above).

Panel of Figure 3 — Figure 3: Example vocabulary items from tokenization.

1. Camera OpenScore String Quartets (252 pages)
	Metric	$n$	GPT-5	SMT++	Legato	Legato small
	TEDn	$252$	$90.2$	$98.6$	$\mathbf{62.2}$	$84.0$
	TEDn convert	$18$	$89.5$	$79.9$	$\mathbf{59.6}$	$82.7$
	OMR-NED	$252$	$97.3$	$94.8$	$\mathbf{58.3}$	$93.6$
2. Rendered OpenScore String Quartets (252 pages)
	TEDn	$252$	$92.5$	$97.9$	$\mathbf{61.2}$	$78.8$

Table 1: Experimental results on various datasets and metrics. Lower is better for all metrics.

n

represents the number of samples tested for the metric. TEDn is the primary metric, requiring outputs to be converted to MusicXML. TEDn convert : evaluated only on instances where SMT++ produces outputs that can be successfully converted to MusicXML; Legato outputs always converted to MusicXML successfully. OMR-NED is format agnostic, built on extraction of symbols from any format. For OMR-NED, **kern outputs are automatically corrected for syntax errors, while ABC is first converted to MusicXML (again, always successful in practice) before symbol extraction. OpenScore String Quartets is the most challenging dataset, since it has much denser score images. All metrics are explained in § 5 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix


Architecture	SMT++	SMT++	SMT++ large	SMT++ large	Legato
Training Dataset	GrandStaff †	GrandStaff †	GrandStaff †	PDMX-Synth	PDMX-Synth
Format & Tokenizer	**kern	ABC	ABC	ABC	ABC
TEDn	$97.7$	$74.6$	$73.0$	$80.5$	$29.5$
OMR-NED	$91.9$	$70.9$	$82.9$	$96.2$	$43.8$

Appendix

Table 2: Ablation study on IMSLP Piano Scores. Lower is better for all metrics. All metrics are explained in § 5 .

\dagger

: Following SMT++ style of data processing, i.e., syetem-level pre-training and curriculum fine-tuning.

Figure 4: Learning Curve of Legato. The validation loss and error rates are tested on the validation split of PDMX-Synth.

Dataset	Total Files	Failed Files	Failure Rate
Camera OpenScore String Quartets	$252$	$234$	$92.9\%$
Rendered OpenScore String Quartets	$252$	$224$	$88.9\%$
Camera OpenScore Lieder	$64$	$60$	$93.8\%$
Rendered OpenScore Lieder	$64$	$60$	$93.8\%$
IMSLP Piano Scores	$32$	$29$	$90.6\%$

Appendix

Table 3: Failure rates of **kern-to-MusicXML conversion on SMT++’s outputs.


Model	GPT-5			Gemini 2.5 Pro
Output Format	ABC	ABC	MusicXML	ABC	ABC	MusicXML
In-Context Learning		✓			✓
TEDn	$96.7$	$\mathbf{74.7}$	$86.6$	$87.8$	$78.6$	$80.3$
OMR-NED	$98.7$	$\mathbf{93.3}$	$94.4$	$95.4$	$93.7$	$94.3$

Appendix

Table 4: General-purpose VLMs’ ability on IMSLP Piano Scores.


Metric	SMT++	Legato	Legato small
${}^{\dagger}\mathrm{CER}_{\mathrm{kern}}$	$30.7$	$\mathbf{16.7}$	$50.6$
${}^{\dagger}\mathrm{SER}_{\mathrm{kern}}$	$42.6$	$\mathbf{19.1}$	$55.5$
${}^{\dagger}\mathrm{LER}_{\mathrm{kern}}$	$75.2$	$\mathbf{29.2}$	$73.5$
TEDn	$95.0$	$\mathbf{\hphantom{0}7.7}$	$35.0$
TEDn convert	$64.5$	$\mathbf{\hphantom{0}8.7}$	$35.9$

Appendix

Table 5: Results on OpenScore String Quartets violin parts. Lower is better for all metrics. TEDn is the primary metric, requiring outputs to be converted to MusicXML. TEDn convert : evaluated only on instances where SMT++ produces outputs that can be successfully converted to MusicXML; Legato outputs always converted to MusicXML successfully. OMR-NED is format agnostic, built on extraction of symbols from any format. For OMR-NED, **kern outputs are automatically corrected for syntax errors, while ABC is first converted to MusicXML (again, always successful in practice) before symbol extraction.

\dagger

: 10.2% of Legato’s outputs and 27.9% of Legato small ’s outputs are not convertible to **kern, so an empty prediction is used. OpenScore String Quartets is the most challenging dataset, since it has much denser score images. All metrics are explained in § 5 .

Figure 5: ABC error rates on PDMX-Synth test input with different aspect ratios. Error rates are reported by averaging over each bin. Legato is capable of recognizing multi-page scores.

Figure 6: Example (first system of Duetto No. 1 in E minor by Bach, BWV 802) from IMSLP Piano Scores (top), with output from Legato (middle) and SMT++ (bottom). Errors are marked in red boxes.

Panel of Figure 7 — Figure 7: Qualitative examples of piano scores. Since SMT++ is trained only on piano data, we illustrate results on well-known piano works. All score images are sourced from IMSLP (not rendered by MuseScore, abcm2ps, or Verovio) and cropped to a single system for clarity, though both SMT++ and Legato can process full pages. The examples progress from easy ( 7(a) ), to difficult ( 7(b) ), to ill-formed input ( 7(c) ). TEDn values are reported at system-level. Since the number of errors is too large for some of the examples, we did not mark them with red boxes to preserve readability.

Explore similar work

Jul 7, 2026cs.CV

LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music. Legato 2 features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis, following the horizontal lines of notation as they are read on the page, rather than treating the page as an undifferentiated image, enabling better scaling to arbitrarily long inputs. It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content, such as titles and annotations. The pipeline combines system-level segmentation with an autoregressive vision-LM to capture both local notation details and score structure. Across multiple datasets, Legato 2 consistently outperforms prior state of the art. We also show that symbolic transcriptions complement visual inputs for frontier language models, improving their interpretation of dense musical documents. Legato 2 establishes new state-of-the-art performance in both OMR and downstream sheet music understanding.

Guang Yang, Brian Siyuan Zheng, Victoria Ebert +1

Paul G. Allen School of Computer Science & Engineering, University of Washington · Allen Institute for AI

Multi-Instrument Music Transcription Symbolic Music Generation Transcript

Ask PDF

May 11, 2026cs.CV

Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training

Optical Music Recognition (OMR), the task of transcribing sheet music into a structured textual representation, is currently bottlenecked by a lack of large-scale, annotated datasets of real scans. This forces models to rely on either few-shot transfer or synthetic training pipelines that remain overly simplistic. A secondary challenge is encoding non-uniqueness: in the popular Humdrum **kern format for transcribing music, multiple different text encodings can render into the same visual sheet music. This one-to-many mapping creates a harder learning task and introduces high uncertainty during decoding. We propose Transcoda, an OMR system built on (i) an advanced synthetic data generation pipeline, (ii) a normalization of the **kern encoding to enforce a unique normal form and (iii) grammar-based decoding to ensure the syntactic correctness of the output. This approach allows us to train a compact 59M-parameter model in just 6 hours on a single GPU that outperforms billion-parameter baselines. Transcoda achieves the best score among state of the art baselines on a newly curated benchmark of synthetically rendered scores at 18.46% OMR-NED (compared to 43.91% for the next-best system, Legato) and reduces the error rate on historical Polish scans to 63.97% OMR-NED (down from 80.16% for SMT++).

Daniel Dratschuk, Paul Swoboda

Heinrich Heine University Düsseldorf, Germany

Multi-Instrument Music Transcription Lingdt-Vl-Ocr Synthetic Training Data

Ask PDF

Jun 12, 2025cs.CV

Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation

In this work, we introduce the Sheet Music Benchmark (SMB), a dataset of six hundred and eighty-five pages specifically designed to benchmark Optical Music Recognition (OMR) research. SMB encompasses a diverse array of musical textures, including monophony, pianoform, quartet, and others, all encoded in Common Western Modern Notation using the Humdrum **kern format. Alongside SMB, we introduce the OMR Normalized Edit Distance (OMR-NED), a new metric tailored explicitly for evaluating OMR performance. OMR-NED builds upon the widely-used Symbol Error Rate (SER), offering a fine-grained and detailed error analysis that covers individual musical elements such as note heads, beams, pitches, accidentals, and other critical notation features. The resulting numeric score provided by OMR-NED facilitates clear comparisons, enabling researchers and end-users alike to identify optimal OMR approaches. Our work thus addresses a long-standing gap in OMR evaluation, and we support our contributions with baseline experiments using standardized SMB dataset splits for training and assessing state-of-the-art methods.

Juan C. Martinez-Sevilla, Joan Cerveto-Serrano, Noelia Luna +4

Pattern Recognition and Artificial Intelligence Group, University of Alicante, Spain · Self-employed · Center for Computer Research in Music and Acoustics, Stanford University, USA +1

Topic pending

Ask PDF