We present Gugak-VocalSet, the first publicly available audio dataset for Korean traditional singing with dense annotations of sigimsae, a structured pitch contour system of vocal expressions. Gugak-VocalSet contains 24.5 hours of unaccompanied solo vocal recordings across 305 tracks of representative repertoire curated by the National Gugak Center, performed by 20 professional singers in a controlled studio environment with fixed key and tempo. In addition to genre, lyrics, and mood/timbre tags, the dataset provides a unified sigimsae ontology and approximately 68,000 segments (71,700 instances including multi-label) of audio-grounded sigimsae annotations; the ontology adopts a functional-position-based hierarchy designed for consistent application across genres, and 14 expert annotators specializing in Korean traditional music performed segment-level labeling accordingly. Baseline experiments on sigimsae classification and event detection show that domain-informed F0 representations remain competitive with, and on classification surpass, frozen probing of a general-purpose music foundation model, while pitch and spectral cues play complementary roles in event-level detection. Companion page: https://www.danbinaerin.com/Gugak-VocalSet/
Figures & tables
Figure 1: Genre hierarchy of Gugak-VocalSet. Numbers show track counts per category.
Level
Type
Content
Song
Metadata
Title, Genre (top/mid/sub), Singer, Jangdan, Tempo, Time Sig., EN/KR Captions
Expert Tags
Key, Scale, Mood (80 tags), Timbre (38 tags)
Segment
Lyrics
Phrase-level (7–20 s)
Alignment
Lyrics–Audio Alignment
Event
Sigimsae
17 Ornament Types w/ Timestamps
Table 1: Summary of metadata and annotations in Gugak-VocalSet.
Name
Sym.
Contour
Description
Onset Sigimsae (앞시김새)
onset asc. slide (밀어내기)
Sliding up from a lower pitch to the main note
onset desc. flick (꺾어내기)
Briefly touching a higher pitch before flicking down to the main note
onset upper turn (감아내기 위)
Winding upward, passing through a higher pitch, then returning to the main note
onset lower turn (감아내기 아래)
Winding downward, passing through a lower pitch, then returning to the main note
Vibrato Sigimsae / (요성 시김새) / yoseong / 搖聲
Figure 2: The 17 sigimsae types in the unified ontology grouped by functional position, with notation symbol, pitch contour, and description.
Figure 3: Annotation examples: jeongak (top) and minsogak (bottom). Each panel shows lyric–audio alignment, F0 contour overlaid on a spectrogram, and sigimsae annotations color-coded by functional category.
Figure 4: Statistics of 71,742 sigimsae instances across 305 tracks. (a) Annotation count per type (log scale). (b) Duration distribution per type; labels show means μ .
Model
Input
Macro F1
Weighted F1
Task-specific CNNs (trained from scratch)
F0CNN
F0 contour
.441 ± .015
.530 ± .022
MelCNN
Mel spec
.400 ± .018
.510 ± .025
Frozen foundation models (best layer, MARBLE 2-layer MLP probe)
MERT-v1 95M (L7)
MERT emb.
.358 ± .011
.454 ± .017
CultureMERT-TA 95M (L7)
MERT emb.
.334 ± .022
.427 ± .031
Table 2: 17-category sigimsae classification (song-level 5-fold CV, mean±std). MERT-family models: best layer among L ∈{5,7,9,11,12} .
Figure 5: Row-normalized (recall) confusion matrix for 17-type classification (F0CNN, aggregated over 5 folds).
Within
Cross
Direction
M.F1
Acc
M.F1
Acc
J → M
.423 ± .007
.558 ± .005
.279 ± .002
.429 ± .010
M → J
.354 ± .004
.500 ± .006
.286 ± .013
.432 ± .035
Table 3: Cross-genre Macro F1 (F0CNN, 17-cat, 5-fold CV, 102 songs per genre, mean ± std over 3 seeds).
Model
Frame F1 M
Ev.F1 200ms
IoU F1 0.5
EDTCN (F0)
.324 ± .015
.675 ± .014
.661 ± .014
EDTCN (mel)
.269 ± .013
.763 ± .012
.712 ± .015
MERT-v1 95M (L5)
.311 ± .015
.756 ± .026
.706 ± .028
CultureMERT 95M (L5)
.294 ± .016
.747 ± .019
.700 ± .022
Table 4: Sigimsae event detection results on the 17-category ontology. MERT-family: best layer in L ∈{5,7,9,12} by frame Macro F1.
Computational analysis of music often relies on discrete representations, yet many musical traditions are organized around continuous pitch movement that resists segmentation into note-like units. For such traditions, the discrete units that analysis would build on are not given in advance. We address this gap by learning a vocabulary of local pitch-contour patterns directly from unlabeled audio, using a VQ-VAE that quantizes fixed-length contour segments into a finite codebook. To make the learned tokens stable across segmentation positions and small variations in timing and pitch range, we train the model with a reconstruction objective evaluated under the best alignment among a set of candidate temporal and pitch-domain transformations. Applied to Korean traditional music, the learned tokens recover information about expert-defined sigimsae categories without supervision, and in pansori individual tokens align with the two principal modes, Gyemyeonjo and Ujo, supporting their use as units for corpus-level analysis of contour-centric traditions.
Seonguk Ju, Seola Cho, Sooin Chung +2
Music & Arts Learning (MALer) Lab, Sogang University, South Korea · Graduate School of Culture Technology, KAIST, South Korea
The intonational structure of Seoul Korean has been defined with discrete tonal categories within the Autosegmental-Metrical model of intonational phonology. However, it is challenging to map continuous F0 contours to these invariant categories due to variable F0 realizations in real-world speech. Our paper proposes Dual-Glob, a deep supervised contrastive learning framework to robustly classify fine-grained pitch accent patterns in Seoul Korean. Unlike conventional local predictive models, our approach captures holistic F0 contour shapes by enforcing structural consistency between clean and augmented views in a shared latent space. To this aim, we introduce the first large-scale benchmark dataset, consisting of manually annotated 10,093 Accentual Phrases in Seoul Korean. Experimental results show that our Dual-Glob significantly outperforms strong baseline models with state-of-the-art accuracy (77.75%) and F1-score (51.54%). Therefore, our work supports AM-based intonational phonology using data-driven methodology, showing that deep contrastive learning effectively captures holistic structural features of continuous F0 contours.
Hyunjung Joo, GyeongTaek Lee
Department of Linguistics, Rutgers University · Hanyang Institute for Phonetics and Cognitive Sciences of Language (HIPCS) · Department of Smart Factory, Gachon University
Kunqu is one of the oldest extant forms of Chinese opera, yet its digitization remains scarce and difficult to annotate, with two challenges standing out: first, close analysis of Kunqu performance demands syllable-level lyrics alignment; second, analyzing Kunqu pronunciation requires its phonological attributes (e.g., rhyme) to be linked with each syllable. To address these challenges, we present KunSet, the first Kunqu performance corpus comprising 482,164 fine-grained annotations of syllable-level lyrics and phonological attributes in total media duration of 155.2 h with 106.3 h annotated. To ensure annotation quality, KunSet was preannotated by an iterative ML-assisted pipeline and corrected by over 200 trained annotators via a purpose-built online annotation system under a two-round review process. To assess KunSet as a music information retrieval (MIR) benchmark, we evaluate two tasks: lyrics alignment, by finetuning a forced-alignment model with an acoustic anchor decoder; and same-rhyme retrieval, using frozen Chinese HuBERT features. Models built with KunSet outperform their baselines. We release KunSet and model checkpoints to support MIR research on traditional-music corpora.
Lejun Min, Siqi Chen, Delong Wang +10
Stanford University · Peking University · University of California San Diego +3