cs.CVJul 7, 2026

LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

Authors: Guang YangBrian Siyuan ZhengVictoria EbertNoah A. Smith

Organizations: Paul G. Allen School of Computer Science & Engineering, University of Washington · Allen Institute for AI

Abstract

We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music. Legato 2 features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis, following the horizontal lines of notation as they are read on the page, rather than treating the page as an undifferentiated image, enabling better scaling to arbitrarily long inputs. It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content, such as titles and annotations. The pipeline combines system-level segmentation with an autoregressive vision-LM to capture both local notation details and score structure. Across multiple datasets, Legato 2 consistently outperforms prior state of the art. We also show that symbolic transcriptions complement visual inputs for frontier language models, improving their interpretation of dense musical documents. Legato 2 establishes new state-of-the-art performance in both OMR and downstream sheet music understanding.

Explore similar work

CardsList
  1. Optical Music Recognition for Real-World Manuscripts with Synthetic Data

    Jun 8, 2026Jiří Mayer, Martina Dvořáková, Vojtěch Dvořák +6Sheet MusicPiano

  2. Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation

    Jun 12, 2025Juan C. Martinez-Sevilla, Joan Cerveto-Serrano, Noelia Luna +4Sheet MusicOptical Character Recognition