Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody

Luigi Rivolta

cs.SDOct 18, 2026

Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody

Authors: Luigi Rivolta

Organizations: Independent Researcher

Abstract

People look up a song by humming it, but also by whistling it or singing it with the wrong words. Published query-by-humming systems and benchmarks focus on hums and sung queries, and no public dataset pairs whistles or wrong-lyrics singing with song recordings. We study both query types with a melody-only retrieval system: a key-normalized pitch track embedded by a 5.2M-parameter contour encoder, with a spectral-peak tracker for whistles. First, a physics-grounded generator renders synthetic whistles and nonsense-syllable singing from the vocal melodies of 11,083 songs. Training on them raises top-1 retrieval of held-out synthetic queries against an 11,050-song gallery from 0.180 to 0.251 for whistles and from 0.193 to 0.264 for wrong-lyrics singing, but the gain does not transfer: on the MLEnd test performers the best synthetic-only model gets 210 of 339 real whistles right at top-1, against 214 for the baseline. Second, a closed-set fine-tune on real MLEnd whistles with performer-disjoint splits raises whistle-to-hum top-1 from 0.631 (95% CI 0.581-0.681) to 0.673 (0.622-0.723), on known songs only. Third, two further generations of synthetic queries also fail to transfer: melody-conditioned whistles and la-la singing from a commercial neural music model leave real-whistle top-1 at 0.631, and an open-source pipeline (YingMusic-Singer-Plus singing with partly wrong, mumbled or nonsense lyrics, a DDSP whistle synthesizer, and a melody-error model fitted on real performers) gives at best 0.619 (paired difference -0.012, 95% CI -0.038 to +0.015). Across physics-based, commercial neural and open-source neural synthetic data, nothing transferred to real whistles, which points to collecting real queries as the missing step. Code is public; models and data will be released.

Figures & tables

Figure 1: Whistle F0 validation on MLEnd (S8): spectral-peak tracker vs. half-speed RMVPE. [real]

Figure 2: Top-1 by model and query set, generated from the committed result files. Left: synthetic hold-out queries [synthetic] . Right: MLEnd real whistles against other people’s hums [real] . D-041’s all-8 bar is not clean (it trained on real whistles of 6 of those songs); its held-out bar is.

Model	Whistle	Wrong lyrics	Both (8,526)
E2b (baseline)	0.180 / 0.291 / 0.220	0.193 / 0.314 / 0.235	0.186 / 0.303 / 0.227
D-040 (step 3,000)	0.226 / 0.342 / 0.267	0.238 / 0.365 / 0.283	0.232 / 0.354 / 0.275
D-041 best (9,500)	0.253 / 0.378 / 0.297	0.269 / 0.401 / 0.314	0.261 / 0.389 / 0.305
D-042 best (11,250)	0.251 / 0.374 / 0.295	0.264 / 0.399 / 0.312	0.258 / 0.387 / 0.303

Table 1: Synthetic hold-out [synthetic] : 4,263 queries per style, 11,050-song gallery. Top-1 / top-10 / MRR. One seed.

Set	E2b	D-040	D-041 best	D-042 best
HumTrans test (769)	0.996 / 1.000	0.991 / 1.000	0.993 / 1.000	0.992 / 1.000
HumTrans test, $+7$ st	0.996 / 1.000	0.995 / 1.000	0.995 / 1.000	0.993 / 1.000
MIR-QBSH 48 (4,431)	0.955 / 0.993	0.944 / 0.991	0.949 / 0.991	0.950 / 0.992

Table 2: Real hums against MIDI [real] . Top-1 / top-10.

† Trained on real whistles of the other 6 songs: only the Hakuna + Potter column is clean.
Model	Hakuna + Potter (90)	Other 6 songs (249)	All 8 (339)
E2b	0.589 / 0.711 / 0.698 (53)	0.647 / 0.791 / 0.742 (161)	0.631 / 0.770 / 0.731 (214)
D-040	0.656 / 0.789 / 0.741 (59)	0.594 / 0.779 / 0.720 (148)	0.611 / 0.782 / 0.725 (207)
D-041 best †	0.456 / 0.700 / 0.620 (41)	0.707 / 0.863 / 0.803 (176)	0.640 / 0.820 / 0.754 (217)
D-042 best	0.622 / 0.722 / 0.709 (56)	0.618 / 0.763 / 0.724 (154)	0.619 / 0.752 / 0.720 (210)

Table 3: MLEnd test whistles

\to

other people’s hums, 8-way closed set [real] . Top-1 / top-3 / MRR, with correct top-1 counts in parentheses.

Model	Train performers	Top-1 [clip 95% CI]	Top-3	MRR
E2b zero-shot	0	0.631 [0.581, 0.681]	0.770	0.731
FT 10%	14	0.634 [0.584, 0.684]	0.776	0.735
FT 25%	34	0.625 [0.575, 0.678]	0.794	0.738
FT 50%	68	0.655 [0.605, 0.705]	0.811	0.757
FT 100%	136	0.673 [0.622, 0.723]	0.820	0.768

Table 4: Closed-set real-whistle training on MLEnd’s 8 songs [real] (D-044). Whistle

\to

other people’s hums, 8-way; test performers disjoint from training. Top-1 shows clip 95% CI; MRR is mean

1/(\mathrm{rank}+1)

from the same ranks. Source: docs/paper/results/d044/results.json .

MLEnd test whistles $\to$ hums [real] : E2b 0.631 [0.581, 0.681] (339), 0.589 [0.489, 0.689] (held-out 90);
	Phase A (32 classics)	Phase B (79 chart songs)
Sections accepted / blocked	37 / 55	54 / 183
Clips (whistle + la-la)	74	106
La-la top-1 / top-10	6/37 / 11/37	17/53 / 22/53
Whistle top-1 / top-10	3/37 / 11/37	7/53 / 11/53
D-047 fine-tune: identical (best checkpoint = step 0).

Table 5: ElevenLabs conditioned queries [synthetic] (D-045, D-046) and fine-tuning on them, scored on real whistles [real] (D-047). Judge: E2b, top-1 against 1,051 songs. Uploads rejected by the provider’s copyright screen are still billed.

	All 8 songs (339)		Hakuna + Potter, held out (90)			P90 (304)
Model	Top-1 [CI]	$\Delta$ [CI]	Top-1 [CI]	$\Delta$ [CI]	Top-3	Top-1 [CI]
E2b	0.631 [0.581, 0.681]	–	0.589 [0.489, 0.689]	–	0.711	0.648 [0.592, 0.701]
D-048 best (= step 0)	identical to E2b
D-048 last	0.546 [0.496, 0.599]	n/a	0.533 [0.433, 0.633]	n/a	0.756	0.549 [0.493, 0.605]
D-049 A best (= step 0)	identical to E2b
D-049 A last	0.546 [0.493, 0.599]	$-0.086$ [ $-0.130$ , $-0.044$ ]	0.511 [0.411, 0.611]	$-0.078$ [ $-0.156$ , $0.000$ ]	0.733	0.553 [0.497, 0.609]

Table 6: Open-source synthetic queries, scored on MLEnd test whistles

\to

other people’s hums [real] (8-way; D-044 protocol). Top-1 with clip 95% bootstrap CI;

\Delta

is the paired bootstrap top-1 difference against E2b on the same clips. “Best” = checkpoint selected on MLEnd validation performers; “last” = step 1,500. P90 is the melody-filtered subset (Section 9 , D-050). Sources: docs/paper/results/d048 , d049 , d050 .

Music Information Retrieval

Music Information Retrieval develops computational methods for finding, classifying, describing, and analyzing music from audio, scores, and metadata.

See all 153 papers

Jul 30, 2026

VocalRender: Score-Native Singing Voice Synthesis for Real-World Composition

Existing singing voice synthesis systems often require predefined durations, explicit duration prediction, or time-aligned acoustic guidance, which limits their compatibility with practical composition workflows. We propose VocalRender, a score-native system that directly synthesizes singing from lyrics, pitches, symbolic note values, and tempo. It uses an interleaved lyric--note representation and an autoregressive diffusion model to generate continuous acoustic latents while predicting the output length, eliminating the need for explicit duration prediction. Trained on a 2,300-hour singing dataset, VocalRender achieves strong intelligibility, strong melody control, and high speaker similarity across both in-domain and out-of-domain benchmarks. Notably, it outperforms the strongest baseline by

0.42

points in naturalness CMOS, demonstrating the effectiveness of our proposed score-native architecture.

Yukun Chen, Tianrui Wang, Zhaoxi Mu +2

Xi’an Jiaotong University · Nanyang Technological University · Ant Group +1

Singing Voice Synthesis Audio Diffusion Models Audio Generation

Ask PDF

Mar 25, 2026

YingMusic-Singer: Controllable Singing Voice Synthesis with Flexible Lyric Manipulation and Annotation-free Melody Guidance

Regenerating singing voices with altered lyrics while preserving melody consistency remains challenging, as existing methods either offer limited controllability or require laborious manual alignment. We propose YingMusic-Singer, a fully diffusion-based model enabling melody-controllable singing voice synthesis with flexible lyric manipulation. The model takes three inputs: an optional timbre reference, a melody-providing singing clip, and modified lyrics, without manual alignment. Trained with curriculum learning and Group Relative Policy Optimization, YingMusic-Singer achieves stronger melody preservation and lyric adherence than Vevo2, the most comparable baseline supporting melody control without manual alignment. We also introduce LyricEditBench, the first benchmark for melody-preserving lyric modification evaluation. The code, weights, benchmark, and demos are publicly available at https://github.com/ASLP-lab/YingMusic-Singer-Plus.

Chunbo Hao, Junjie Zheng, Guobin Ma +6

AI Lab, Giant Network, China

Singing Voice Synthesis Controllable Music Generation Audio Diffusion Models Audio Generation

Ask PDF

Oct 23, 2026

MusicTwins: Training Audio-Language Models to Listen like Their Persona

People differ in how they perceive music, what associations they make with it, what they like about it, and how they respond to it. Computational models intended to simulate people in environments that include music should account for these differences. Although LLMs have been shown to reproduce aspects of human behavior, whether audio-language models (ALMs) can reproduce a specific person's responses to textual and musical stimuli remains underexplored. We introduce MusicTwins, a framework that combines persona summaries, personal memory, and post-training to generate person-specific responses. We construct persona representations from human writings, capturing background, views, personality, writing style, and a musical profile, and post-train Audio Flamingo Next through supervised fine-tuning and reinforcement learning. Evaluation on ten held-out personas shows notable improvements for Audio Flamingo Next in persona fidelity, including personality, writing style, and responses to music, supporting MusicTwins as a foundation for training audio-language models to listen like their persona.

Milan Liessens Dujardin, Kevin Miao

Bryel Labs, UC Berkeley · Bryel Labs

Audio-Language Models Personalized Language Model Alignment Personality Modeling in Language Models

Ask PDF

Abstract

Music Information Retrieval

Explore similar work

VocalRender: Score-Native Singing Voice Synthesis for Real-World Composition

YingMusic-Singer: Controllable Singing Voice Synthesis with Flexible Lyric Manipulation and Annotation-free Melody Guidance

MusicTwins: Training Audio-Language Models to Listen like Their Persona