Music structure analysis (MSA) aims to identify functional labels and their boundaries in music. Despite recent progress, three challenges remain. First, fine-grained annotations for MSA are scarce. Second, existing systems often share one encoder between boundary detection and functional labeling despite their different information requirements. Third, most end-to-end systems are audio-only and ignore structural cues from lyrics. To address these issues, we construct SheetSage-MSA, an MSA dataset derived from SheetSage, containing 13,880 cleaned excerpt annotations from 9,010 songs and approximately 82.5 hours of annotated audio. We further propose SCALE (Structural Context from Audio and Lyric Evidence), which uses separate task-specific audio pathways to model local transitions and long-range structural context, and incorporates timestamped lyrics and lyrical recurrence to refine chorus predictions. On the HarmonixSet benchmark, SCALE outperforms the existing state-of-the-art SongFormer by 3.5 and 4.2 percentage points in HR.5F and HR3F, respectively, while improving ACC by 0.9 points. An interactive demo page containing the source code is also available at https://multimodal-music-research-lab.github.io/SCALE/.
Figures & tables
Figure 1: SheetSage-MSA data creation pipeline based on the SheetSage-data repository [ 15 ] . Audio is downloaded from YouTube using links in Hooktheory.json.gz , which also provides alignments. Function labels come from Hooktheory_Raw.json.gz .
Figure 2: Overview of the proposed SCALE architecture. The model comprises three paths: a boundary path, a function path, and a lyrics path. MuQ [ 12 ] audio features are routed to the first two paths; the boundary path combines BiMamba [ 18 ] with global Transformer [ 19 ] context, while the function path uses a RoPE [ 20 ] Transformer for frame-level labels. The lyrics path extracts and line-splits lyrics with SoulX-Singer [ 21 ] and SaT [ 22 ] , encodes them with Longformer [ 23 ] , and uses similarity and recurrence features to refine the chorus logits.
Dataset
Train
Val
Test
SongFormDB-HX [ 13 ]
512
200
–
SheetSage-MSA (Proposed)
9,010
–
–
SongFormBench-HX [ 13 ]
–
–
200
Table 1: Dataset splits used in this work. SongFormDB-HX is used for training and validation, SheetSage-MSA for training only, and SongFormBench-HX for testing. All values denote numbers of songs.
Model
ACC
HR.5F
HR3F
Same HX training split
All-In-One (HX) [ 8 ]
0.704
0.530
0.704
LinkSeg (HX) [ 9 ]
0.769
0.589
0.775
SongFormer (HX) [ 13 ]
0.795
0.703
0.784
SCALE (HX)
0.798
0.730
0.821
Respective training settings
Table 2: Performance on SongFormBench-HX. Upper: models retrained on the same HX split. Lower: respective training settings, with baseline results reported by SongFormer [ 13 ] .
Figure 3: Analysis on Someday by Tegan and Sara. SCALE recovers more boundaries than SongFormer [ 13 ] but still misses some boundaries between repeated sections with the same function label.
Music Structure Analysis (MSA) aims to uncover the high-level organization of musical pieces. State-of-the-art methods are often based on supervised deep learning, but these methods are bottlenecked by the need for heavily annotated data and inherent structural ambiguities. In this paper, we propose an unsupervised evaluation of nine open-source, generic pre-trained deep audio models, on MSA. For each model, we extract barwise embeddings and segment them using three unsupervised segmentation algorithms (Foote's checkerboard kernels, spectral clustering, and Correlation Block-Matching (CBM)), focusing exclusively on boundary retrieval. Our results demonstrate that modern, generic deep embeddings generally outperform traditional spectrogram-based baselines, but not systematically. Furthermore, our unsupervised boundary estimation methodology generally yields stronger performance than recent linear probing baselines. Among the evaluated techniques, the CBM algorithm consistently emerges as the most effective downstream segmentation method. Finally, we highlight the artificial inflation of standard evaluation metrics and advocate for the systematic adoption of trimming'', or even double trimming'' annotations to establish more rigorous MSA evaluation standards.
Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarce, and accurate local predictions can still produce inconsistent musical sequences. We present SheetSage2, a unified music transcription framework that combines synthetic data, task-specific structured decoding, and autoregressive distillation. Automatically annotated MIDI, rendered into audio, provides scalable supervision across music understanding tasks. Task-specific structured decoders integrate complementary musical cues and their temporal dependencies to produce musically coherent scores. Autoregressive distillation further retains transcription accuracy without task-specific dynamic programming at inference. Across eight benchmark collections, a single SheetSage2-AR model exceeds the listed prior systems on 12 of 15 benchmark--metric pairs in our evaluation, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks. Model weights and inference code are publicly available.
Existing audio-to-score (A2S) systems primarily focus on classical music, and the application to popular music remains underexplored. This paper first presents the new SheetSage-A2S Dataset, which includes 61 hours of audio with **kern score encodings for 9,468 clips originating from 6,066 unique songs, the first of its kind to facilitate A2S research for popular music. Additionally, we improve on existing A2S approaches by using data augmentation and MuQ, a pretrained feature-extraction model for music audio, to enhance generalisation abilities and extract meaningful audio features. Results show that the proposed A2S model achieves 4.98% symbol error rate (SER) on the Quartets collection for classical music, which significantly outperforms the 15.3% SER from the existing state-of-the-art \cite{alfaro-contrerasTransformer2024}. Additionally, our model achieves 20.92% SER on the SheetSage-A2S dataset for popular music, serving as a strong benchmark for future research. The dataset, model, and code are made publicly available at: https://github.com/Multimodal-Music-Research-Lab/SheetSage2Kern_model.
Eoin Cummins, Zhongyi Huang, Alexandre D'Hooge +2
Great Bay University Dongguan, Guangdong, China · Guangxi Normal University Guilin, China · Shenzhen University Shenzhen, China