Structural Context from Audio and Lyric Evidence for Music Structure Analysis Using the New SheetSage-MSA Dataset

Hongbing Li; Yaolong Ju

cs.SDOct 19, 2026

Structural Context from Audio and Lyric Evidence for Music Structure Analysis Using the New SheetSage-MSA Dataset

Authors: Hongbing Li, Yaolong Ju

Organizations: Queen Mary University of London, London, UK · Great Bay University, Dongguan, China

Abstract

Music structure analysis (MSA) aims to identify functional labels and their boundaries in music. Despite recent progress, three challenges remain. First, fine-grained annotations for MSA are scarce. Second, existing systems often share one encoder between boundary detection and functional labeling despite their different information requirements. Third, most end-to-end systems are audio-only and ignore structural cues from lyrics. To address these issues, we construct SheetSage-MSA, an MSA dataset derived from SheetSage, containing 13,880 cleaned excerpt annotations from 9,010 songs and approximately 82.5 hours of annotated audio. We further propose SCALE (Structural Context from Audio and Lyric Evidence), which uses separate task-specific audio pathways to model local transitions and long-range structural context, and incorporates timestamped lyrics and lyrical recurrence to refine chorus predictions. On the HarmonixSet benchmark, SCALE outperforms the existing state-of-the-art SongFormer by 3.5 and 4.2 percentage points in HR.5F and HR3F, respectively, while improving ACC by 0.9 points. An interactive demo page containing the source code is also available at https://multimodal-music-research-lab.github.io/SCALE/.

Figures & tables

Figure 1: SheetSage-MSA data creation pipeline based on the SheetSage-data repository [ 15 ] . Audio is downloaded from YouTube using links in Hooktheory.json.gz , which also provides alignments. Function labels come from Hooktheory_Raw.json.gz .

Figure 2: Overview of the proposed SCALE architecture. The model comprises three paths: a boundary path, a function path, and a lyrics path. MuQ [ 12 ] audio features are routed to the first two paths; the boundary path combines BiMamba [ 18 ] with global Transformer [ 19 ] context, while the function path uses a RoPE [ 20 ] Transformer for frame-level labels. The lyrics path extracts and line-splits lyrics with SoulX-Singer [ 21 ] and SaT [ 22 ] , encodes them with Longformer [ 23 ] , and uses similarity and recurrence features to refine the chorus logits.

Dataset	Train	Val	Test
SongFormDB-HX [ 13 ]	512	200	–
SheetSage-MSA (Proposed)	9,010	–	–
SongFormBench-HX [ 13 ]	–	–	200

Table 1: Dataset splits used in this work. SongFormDB-HX is used for training and validation, SheetSage-MSA for training only, and SongFormBench-HX for testing. All values denote numbers of songs.

Same HX training split
Model	ACC	HR.5F	HR3F
All-In-One (HX) [ 8 ]	0.704	0.530	0.704
LinkSeg (HX) [ 9 ]	0.769	0.589	0.775
SongFormer (HX) [ 13 ]	0.795	0.703	0.784
SCALE (HX)	0.798	0.730	0.821
Respective training settings

Table 2: Performance on SongFormBench-HX. Upper: models retrained on the same HX split. Lower: respective training settings, with baseline results reported by SongFormer [ 13 ] .

Figure 3: Analysis on Someday by Tegan and Sara. SCALE recovers more boundaries than SongFormer [ 13 ] but still misses some boundaries between repeated sections with the same function label.

Music Information Retrieval

Music Information Retrieval develops computational methods for finding, classifying, describing, and analyzing music from audio, scores, and metadata.

See all 153 papers

Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis
Mar 28, 2026Axel MarmoretAudio Representation Learning Music Information Retrieval
BRAIN team, MEE, IMT Atlantique, Brest, France
SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision
Oct 4, 2026Junyan Jiang, Ruibin Yuan, Jiahao Pan +4Automatic Music Transcription Music Understanding
New York University · MBZUAI · ACE Studio +1
Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset
Aug 6, 2026Eoin Cummins, Zhongyi Huang, Alexandre D'Hooge +2Automatic Music Transcription
Great Bay University Dongguan, Guangdong, China · Guangxi Normal University Guilin, China · Shenzhen University Shenzhen, China

Mar 28, 2026J/K move · Enter open · S save

Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis

Axel Marmoret

BRAIN team, MEE, IMT Atlantique, Brest, France

Audio Representation Learning Music Information Retrieval

Open paper AskPDF arXiv

Music Structure Analysis (MSA) aims to uncover the high-level organization of musical pieces. State-of-the-art methods are often based on supervised deep learning, but these methods are bottlenecked by the need for heavily annotated data and inherent structural ambiguities. In this paper, we propose an unsupervised evaluation of nine open-source, generic pre-trained deep audio models, on MSA. For each model, we extract barwise embeddings and segment them using three unsupervised segmentation algorithms (Foote's checkerboard kernels, spectral clustering, and Correlation Block-Matching (CBM)), focusing exclusively on boundary retrieval. Our results demonstrate that modern, generic deep embeddings generally outperform traditional spectrogram-based baselines, but not systematically. Furthermore, our unsupervised boundary estimation methodology generally yields stronger performance than recent linear probing baselines. Among the evaluated techniques, the CBM algorithm consistently emerges as the most effective downstream segmentation method. Finally, we highlight the artificial inflation of standard evaluation metrics and advocate for the systematic adoption of trimming'', or even double trimming'' annotations to establish more rigorous MSA evaluation standards.

Abstract

Music Information Retrieval

Explore similar work

Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis

SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision

Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset