ARIA: Audio-Driven Melody-Tone Relation Modeling for Cantonese Lyric Authoring

Shengyu Li; Jinting Wang; Li Liu

cs.CLOct 6, 2026

ARIA: Audio-Driven Melody-Tone Relation Modeling for Cantonese Lyric Authoring

Authors: Shengyu Li, Jinting Wang, Li Liu

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Cantonese lyric writing requires close alignment between lexical tones and melodic pitch. Existing melody-guided lyric generation methods typically rely on symbolic melody to generate lyrics. However, in real songwriting scenarios, melodies are often expressed as raw singing audio or hummed recordings, where pitch is implicit, noisy, and unstructured, making these methods difficult to apply directly. To address this limitation, we propose ARIA, a two-stage audio-driven melody-tone relation modeling framework for Cantonese lyric authoring that generates Cantonese lyrics from singing recordings with provided character-level timestamps. Specifically, we first design a Tri-Stream Relation-Aware Tone Estimator (TRATE) to predict 0243 sequences from timestamped singing audio by modeling multi-stream acoustic cues and relational tonal structure. We then propose a Decoupled Retrieval-Augmented Tone-Conditioned Lyric Generator (DRA-TCLG) to generate fluent lyrics conditioned on predicted tonal plans with retrieval-enhanced lexical guidance. Moreover, we construct a large-scale aligned audio-Jyutping-0243 dataset from real Cantonese singing recordings to support this new task. Experimental results demonstrate that ARIA achieves strong performance in both 0243 prediction and tone-consistent lyric generation, validating the effectiveness of the proposed framework.

Figures & tables

Figure 1: Motivation of ARIA . Existing melody-aware lyric systems typically rely on symbolic melody or manually specified tone constraints. In contrast, ARIA receives singing audio together with character-level lyric-slot boundaries, predicts an explicit 0243 tone–melody control sequence, and uses it as an interpretable interface for controllable Cantonese lyric generation.

Figure 2: Overall architecture of ARIA . The framework decouples audio-side tone-plan estimation and text-side lyric generation through a unified 0243 intermediate representation. In Stage 1, TRATE predicts 0243 control sequences from singing audio and character-level timestamps by modeling absolute, contour, and relative acoustic streams with gated fusion and relation-aware sequence modeling. In Stage 2, DRA-TCLG generates Cantonese lyrics conditioned on the predicted tonal plan, first learning a base tone-to-lyric mapping and then incorporating retrieval-enhanced lexical guidance from a frequency-aware 0243 dictionary.

Model	Acc $\uparrow$	F1 $\uparrow$	$\pm 1$ $\uparrow$	NED $\downarrow$	Params
Rule RQ	0.373	0.360	0.749	0.583	0
Rule AQ	0.370	0.357	0.747	0.585	0
Qwen Rel.	0.501	0.471	0.835	0.460	4.57B / 32.46M tr.
Qwen 3-Br.	0.565	0.537	0.844	0.398	4.57B / 32.46M tr.
TRATE	0.644	0.615	0.885	0.347	2.37M

Table 1: Audio-to- 0243 results. Acc, F1, and

\pm 1

denote TokAcc, Macro-F1, and Within

\pm 1

, respectively. Higher is better except for NED. The TRATE row reports the mean over three training seeds (13, 42, and 3407). Rule-based methods require no trainable parameters. For Qwen baselines, “tr.” denotes trainable parameters. The best results are highlighted in bold , and the second-best results are underlined .

Method	Alignment		Diversity
Method	Harmony $\uparrow$	Consistency $\uparrow$	Avg Sim $\downarrow$	Min Sim $\downarrow$	MaD1 $\uparrow$	MaD2 $\uparrow$	MiD1 $\uparrow$	MiD2 $\uparrow$
SongNet	0.4846	0.0375	0.4911	0.0479	0.9354	0.9862	0.0253	0.3341
SmBART	0.8186	0.6755	0.4277	0.0008	0.8830	0.9410	0.0297	0.4258
ToneCraft	0.9684	0.9347	0.5036	0.0541	0.9552	0.9855	0.0134	0.1403
DRA-TCLG (Ours)	0.9740	0.9421	0.5053	0.0193	0.9544	0.9944	0.0190	0.2967

Table 2: 0243 -to-lyric generation comparison across alignment and diversity metrics.

0243 -to-Lyrics (reference 0243 input)
Method	Alignment		Diversity
Method	Harmony $\uparrow$	Consistency $\uparrow$	Avg Sim $\downarrow$	Min Sim $\downarrow$	MaD1 $\uparrow$	MaD2 $\uparrow$	MiD1 $\uparrow$	MiD2 $\uparrow$
SongNet	0.5000	0.0353	0.4699	0.1034	0.9487	0.9894	0.1896	0.7352
SmBART	0.8277	0.6373	0.4138	0.0877	0.9037	0.9524	0.2273	0.7849
ToneCraft	0.9747	0.9466	0.4957	0.0997	0.9612	0.9882	0.0945	0.3763
DRA-TCLG (Ours)	0.9841	0.9687	0.4849	0.1100	0.9630	0.9951	0.1345	0.6246

Table 3: Comparison on the shared 500-sample test subset under two input regimes. The 0243 -to-lyrics systems receive reference 0243 sequences derived from the original lyrics, whereas TRATE+DRA-TCLG receives singing audio with character-level timestamps and first predicts its own 0243 plan. Harmony and Consistency are computed against the control sequence supplied to each generator. The table therefore provides a system-level cross-regime comparison rather than a same-input comparison.

Figure 3: TRATE ablation. The progression isolates richer token features, structured relation-aware modeling, and token-wise gated fusion.

Alignment
Metric	Method
Metric	Base	One-Stage	Two-Stage
Harmony $\uparrow$	0.9104	0.6449	0.9740
Consistency $\uparrow$	0.7619	0.1579	0.9421
Diversity
Avg Sim $\downarrow$	0.5269	0.4912	0.5053

Table 4: Ablation study of DRA-TCLG.

Metric	DRA-TCLG	ToneCraft	SmBART	SongNet
Tone–Mel.	3.185	2.967	2.654	2.029
Rhythm	3.256	2.998	2.671	2.152
Canto.	3.233	3.167	2.710	2.077
Lyric	3.279	3.054	2.756	2.202
Sing.	3.167	3.015	2.627	2.027
Pref.	160	142	89	22

Table 5: Human evaluation on 40 items with 12 Cantonese-speaking raters. Values are descriptive mean ratings except Pref., which reports 413 valid preference votes; 56 uncertain and 11 blank responses are excluded. Higher is better.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Tone	Category	Chao	Singing tier	0243
1	Yin level	55/53	high	3
2	Yin rising	35/25	high	3
3	Yin departing	33	mid-high	4
4	Yang level	11/21	low	0
5	Yang rising	13/23	mid-high	4
6	Yang departing	22	mid	2

Appendix

Table 6: Mapping Cantonese citation tones to relative singing-pitch tiers and the corresponding 0243 labels. The exact onset values of tones 2, 4, and 5 vary across descriptions, but the tier relations relevant for 0243 remain stable ( Khouw and Ciocca, 2007 ; Cheng et al., 2025 ) .

Tone	Strict	0243	Rationale
1	3	3	Same high tier as tone 2
2	9	3	Same high tier as tone 1
3	4	4	Same mid-high tier as tone 5
4	0	0	Unique low tier
5	5	4	Same mid-high tier as tone 3
6	2	2	Unique mid tier

Appendix

Table 7: Relation between the stricter nine-tone digit mnemonic and the relaxed 0243 collapse used in lyric writing. The relaxed form merges categories that occupy the same relative singing-pitch tier ( Lo and Huang, 1989 ; Huang, 2024a ; Huang, 2024b ; Cheng et al., 2025 ) .

Panel of Figure 4 — Figure 4: Practical dataset pipeline used in ARIA . The main quality-control effort is spent on character alignment, while the final 0243 sequence is derived deterministically from lyric-side Jyutping.

Stage	Main risk	What we do
Initial alignment	Pickup notes, melisma, and ornamentation blur where a lyric character should begin and end in the audio.	We inspect sampled examples and use them to monitor alignment quality instead of assuming the raw alignment output is always reliable.
Abnormal-data filtering	Some examples have durations that are implausibly long, implausibly short, or clearly mismatched with the lyric character count.	We remove these abnormal items before forming the final dataset so that TRATE is not trained on obvious alignment failures.
Jyutping conversion	The final label sequence depends on having a stable tone category for each aligned lyric character.	Once the lyrics are aligned, we obtain Cantonese Jyutping on the text side and use it as the basis for deterministic label derivation.
0243 derivation	Directly ranking pitch from audio would introduce extra ambiguity and subjective judgment.	We map Jyutping tone categories to 0243 deterministically, so no further manual verification of the final 0243 sequence is required.

Appendix

Table 8: How ARIA obtains alignment-aware 0243 labels in practice. The pipeline emphasizes alignment quality control and deterministic label derivation rather than manual audio-side ranking of the final labels.

Figure 5: Dataset characteristics of ARIA . The figure shows the distribution of lyric-line length, line duration, character duration, and their joint relation. The dataset is dominated by medium-length lines but exhibits a long tail in duration, motivating representations that model both token-local acoustics and sentence-internal relations.

Field	Description
sample_id	Line-level sample identifier
song_id	Source song identifier
line_id	Line index within the song
text_length	Number of lyric characters
char_bounds	Character start time and duration
label_0243	Token-level 0243 labels

Appendix

Table 9: Released non-audio annotation format for the Stage 1 audio-to- 0243 resource. Copyrighted audio files and local experimental paths are not redistributed.

Criterion	Why 0243 helps	Implication for ARIA
Important	It preserves the most rigid Cantonese lyric constraint: compatibility between lexical tone tier and melodic motion.	The downstream generator does not lose the core “fit to melody” condition at the representation boundary.
Effective	It matches long-standing lyric-writing practice and recent harmony-aware Cantonese generation models.	The control signal is interpretable, linguistically motivated, and aligned with real authoring behavior.
Efficient	Four discrete classes are lower-entropy and more transposition-robust than raw audio, continuous F0, or full note strings.	Annotation is easier to standardize, search space is smaller, and conditioning an LLM becomes simpler and more stable.

Appendix

Table 10: Why the 0243 sequence is a strong intermediate control representation for Cantonese lyric authoring.

Real-world lyric-writing step	Prior knowledge used by human lyricists	Corresponding ARIA design	Effect on the model
Listen to a demo and locate singable slots	Musical timing, phrase boundaries, and character-level rhythm.	TRATE takes singing audio with character boundaries $B=\{(s_{i},e_{i})\}_{i=1}^{N}$ .	The model predicts one controllable label per lyric position rather than a frame-level F0 trace.
Estimate tone-compatible melody tiers	Cantonese tones are organized by relative pitch height and contour, not only by absolute frequency.	TRATE uses absolute, contour, and line-internal relative feature streams.	Acoustic, melodic, and sentence-level evidence are separated before fusion.
Judge neighboring tone–melody fit	Lyricists compare adjacent syllables and avoid implausible large jumps when possible.	TRATE uses relation-aware attention and pairwise ordering heads to model short-range melodic relations.	The target is treated as relational, matching the ordinal nature of 0243 .
Use 0243 as a draftable authoring plan	The expert 0243 method collapses Cantonese tones into four practical singing tiers.	TRATE predicts level-4 labels and maps them deterministically to 0243 ; structured heads regularize register and contour factors.	Cantonese tonal knowledge enters supervision without giving the model lyric text at inference time.
Search and revise lexical candidates	Lyricists consult tone-compatible dictionaries and allow near alternatives for content.	DRA-TCLG conditions on $P$ , retrieves from a 0243 –lyrics dictionary, and refines candidates with an LLM.	The generator receives an explicit, human-readable lexical-planning signal instead of an opaque audio embedding.

Appendix

Table 11: How Cantonese, music, and lyric-writing priors are integrated into ARIA . TRATE corresponds to listening and tone-tier planning; DRA-TCLG corresponds to lexical search and controlled lyric drafting.

Noise	Acc $\uparrow$	F1 $\uparrow$	$\pm 1$ $\uparrow$	NED $\downarrow$
Clean	0.6459	0.6173	0.8840	0.3462
$\pm 25$ ms	0.6447	0.6167	0.8853	0.3465
$\pm 50$ ms	0.6391	0.6102	0.8837	0.3515
$\pm 100$ ms	0.6178	0.5871	0.8773	0.3704

Appendix

Table 12: Boundary-perturbation robustness of the fixed seed-3407 EMA checkpoint. Clean is evaluated once using the same feature-recomputation pipeline as the perturbed conditions; each nonzero noise condition reports the mean over three perturbation seeds.

Feature	Definition	Role
mean_log_f0	Mean log-F0 over frames assigned to the character.	Direct token-window pitch height before comparison with other tokens.
median_log_f0	Median log-F0 over the character window.	Robust absolute height summary for the token itself.
std_log_f0	Standard deviation of log-F0 in the character window.	Local acoustic dispersion, not normalized by line context.
min_log_f0	Minimum log-F0 in the character window.	Lower bound of the token’s observed pitch region.
max_log_f0	Maximum log-F0 in the character window.	Upper bound of the token’s observed pitch region.
q25_log_f0	25th percentile of character-window log-F0.	Absolute lower-quartile pitch statistic within the token.

Appendix

Table 13: Absolute acoustic features. These features summarize token-local F0 and timing evidence before line-level normalization or cross-token comparison.

Feature	Definition	Role
contour_log_f0 [0:15]	Sixteen evenly resampled log-F0 points across the character window.	Preserves pointwise within-character shape rather than collapsing the token into scalar statistics.
contour_voiced_mask [0:15]	Sixteen pointwise voiced/unvoiced indicators aligned to the resampled contour.	Attaches reliability evidence to each contour point.
contour_filled_mask [0:15]	Sixteen pointwise indicators showing where contour values were imputed.	Marks missing or filled trajectory regions in the contour sequence.
token_valid_voiced	Indicator that the character window contains native voiced frames before filling.	Provides a summary reliability flag for the extracted contour trajectory.

Appendix

Table 14: Contour features. These features encode the within-character F0 trajectory and its pointwise reliability.

Feature	Definition	Role
robust_z_median	Character median log-F0 minus line median, divided by the line interquartile range.	Defines height relative to the current lyric line.
zscore_median	Character median log-F0 minus line mean, divided by line standard deviation.	Provides line-normalized height; this is the scalar used by Rule RQ.
percent_rank_median	Stable percent rank of character median log-F0 among tokens in the line.	Captures ordinal line-internal height independent of absolute register.
prev_interval	Current median log-F0 minus previous-token median log-F0.	Encodes adjacent-token pitch relation to the left context.
next_interval	Next-token median log-F0 minus current median log-F0.	Encodes adjacent-token pitch relation to the right context.
prev_higher_flag	Indicator that the current token is higher than the previous token.	Represents directional relation between neighboring tokens.

Appendix

Table 15: Relative features. These features normalize pitch, duration, and position within each lyric line and encode local cross-token relations.

Model	Acc $\uparrow$	F1 $\uparrow$	$\pm 1$ $\uparrow$	NED $\downarrow$
Rule RQ	0.399	0.436	0.593	0.625
Rule AQ	0.394	0.430	0.608	0.626
Qwen Rel.	0.460	0.489	0.671	0.544
Qwen 3-Br.	0.394	0.409	0.658	0.614
TRATE	0.454	0.471	0.677	0.550

Appendix

Table 16: Mandarin-to-Cantonese transfer stress test. Models take Mandarin-version audio with character-level timestamps as input, while references are 0243 sequences derived from the corresponding Cantonese lyrics. Acc, F1, and

\pm 1

denote TokAcc, Macro-F1, and Within

\pm 1

. Higher is better except for NED.

Case	Input plan	GT hit	Output use	Plan match
Use	2433334	Yes	2	Exact
Non-use	22342433	Yes	0	Exact

Appendix

Table 17: Representative retrieval behavior. GT hit indicates that at least one retrieved candidate occurs in the reference lyric. Output use counts retrieved candidates appearing verbatim in the generated lyric. Both outputs follow the input plan exactly, showing that retrieved candidates are optional rather than copied answers.

Explore similar work

Mar 25, 2026eess.AS

YingMusic-Singer: Controllable Singing Voice Synthesis with Flexible Lyric Manipulation and Annotation-free Melody Guidance

Regenerating singing voices with altered lyrics while preserving melody consistency remains challenging, as existing methods either offer limited controllability or require laborious manual alignment. We propose YingMusic-Singer, a fully diffusion-based model enabling melody-controllable singing voice synthesis with flexible lyric manipulation. The model takes three inputs: an optional timbre reference, a melody-providing singing clip, and modified lyrics, without manual alignment. Trained with curriculum learning and Group Relative Policy Optimization, YingMusic-Singer achieves stronger melody preservation and lyric adherence than Vevo2, the most comparable baseline supporting melody control without manual alignment. We also introduce LyricEditBench, the first benchmark for melody-preserving lyric modification evaluation. The code, weights, benchmark, and demos are publicly available at https://github.com/ASLP-lab/YingMusic-Singer-Plus.

Chunbo Hao, Junjie Zheng, Guobin Ma +6

AI Lab, Giant Network, China

Singing Voice Conversion Melody Synthesis Diffusion Models Guidance

Ask PDF

Aug 4, 2026cs.SD

CLASVS: Continuous-Latent Autoregression for Melody-Preserving Lyric Editing in Singing Voice Synthesis

Reference-conditioned melody-preserving lyric editing replaces words while retaining a performance's timing, singer identity, and naturalness. Continuous-latent autoregression avoids finite codebooks and offers stepwise generation with learned stopping. Editing creates a conflict absent from ordinary reconstruction: training pairs reference cues with original lyrics, whereas inference asks revised lyrics to override source-lyric-correlated cues; one source-following patch can propagate through AR history. We introduce CLASVS. Its State-Control-Transition (SCT) routing keeps target-lyric and reference-melody controls persistent, returns semantic feedback on phonetic progress to the causal planner, and confines the previous latent patch to the local Transition. Progressive State-Control Grounding (PSCG) learns this contract through paired-edit-free, content-consistent Mandarin reconstruction. On two Mandarin benchmarks, CLASVS improves all four operations over discrete-AR Vevo2 and reduces macro-PER by 46.2%, while maintaining melody, singer similarity, and perceptual quality. Together, these results establish a strong continuous-AR operating point for score-annotation-free lyric edits and a basis for broader stepwise control. Audio demonstrations are available on our project page: https://piedpiperg.github.io/clasvs-demo/.

Yizhong Geng, Tian-Hao Zhang, Chunfeng Wang +7

Beijing University of Posts and Telecommunications, Beijing, China · Li Auto Inc., Beijing, China · Tsinghua University, Beijing, China

Singing Voice Conversion Mandarin

Ask PDF

Apr 20, 2026cs.SD

Aligning Language Models for Lyric-to-Melody Generation with Rule-Based Musical Constraints

Large Language Models (LLMs) show promise in lyric-to-melody generation, but models trained with Supervised Fine-Tuning (SFT) often produce musically implausible melodies with issues like poor rhythm and unsuitable vocal ranges, a phenomenon we term "constraint violation". To address this, we propose a novel alignment framework that instills musical knowledge without human annotation. We define rule-based musical constraints to automatically generate a preference dataset from an SFT model's outputs. The model is then aligned through a sequential process, first using Direct Preference Optimization (DPO) on paired preference data, followed by Kahneman-Tversky Optimization (KTO) on unpaired negative samples. Experimental results demonstrate that our aligned model substantially reduces rule violations and outperforms strong baselines in both objective and subjective evaluations, generating melodies with substantially improved musicality and coherence. An interactive demo with audio comparisons is available at https://arain233.github.io/AligningMelody-demo.

Hao Meng, Siyuan Zheng, Shuran Zhou +2

Zuoyebang Education Technology, Beijing, China

Large Language Model Alignment Preference Datasets

Ask PDF