Dissonance Spectrum explicitly models perceptual frequency interactions for better music understanding

Tianle Wang; Xinyi Tong; Liangke Zhao; Jishang Chen; Sirui Zhang; Haoxin Zhang; Xin Jin; Duo Xu

cs.SDAug 26, 2026

Dissonance Spectrum explicitly models perceptual frequency interactions for better music understanding

Authors: Tianle Wang, Xinyi Tong, Liangke Zhao, Jishang Chen, Sirui Zhang, Haoxin Zhang, Xin Jin, Duo Xu,

Organizations: Beijing Institute for General Artificial Intelligence, Beijing, China · Central Conservatory of Music, Beijing, China · Peking University, Beijing, China

Abstract

Conventional music representations describe acoustic energy over time and frequency but do not explicitly expose relations among simultaneous frequency components. We introduce the Dissonance Spectrum (DS), a nonnegative time--frequency representation that applies a tolerance-based rational pitch-relation kernel with logarithmic harmonic distance to a constant-Q spectrum and attributes aggregate pairwise interactions back to individual frequency bins. Controlled music-theory tests show strong ordinal agreement for intervals, harmonic-function connections, and church modes, and weaker but significant agreement across diverse chord voicings. DS is then encoded by a lightweight parallel branch whose zero-initialized residual projection preserves the baseline function at initialization. Across six paired training seeds in open-ended music question answering and categorical and dimensional music emotion recognition, DS obtains the highest mean on every reported endpoint relative to the unchanged baseline, a parameter-matched Gaussian-input branch, and an architecture-matched magnitude-CQT branch. These results support DS as an interpretable, complementary representation, while listener-specific perception and broader task coverage remain open problems.

Figures & tables

Panel of Figure 2 — Figure 2: Conceptual DS attribution. Left: intrinsic DS aggregates pairwise relations within one spectrum. Right: cross-reference DS evaluates the target spectrum against a separate reference while retaining target-bin attribution.

Figure 3: Frequency-axis matrix implementation of cross-reference DS. Intrinsic DS follows by setting the reference and target CQT matrices equal.

Figure 4: Plug-and-play DS integration in Music2Emo (top) and MU-LLaMA (bottom). The DS branch provides key/value tokens to a gated residual adapter while the original projection provides the query; snowflakes and flames mark frozen and fine-tuned modules.

Test	$n$	Spearman $\rho$	Kendall $\tau_{b}$
Intervals	13	.951 ( $6.36\!\times\!10^{-7}$ )	.846 ( $5.20\!\times\!10^{-6}$ )
Chord quality (4 exemplars)	4	1.000 (–)	1.000 (–)
Chord quality (13 voicings)	13	.626 (.022)	.462 (.030)
Functional connections	7	.794 (.033)	.655 (.054)
Church modes	7	.893 (.0068)	.810 (.0107)

Table 1: Controlled validation. Parentheses give two-sided

p

-values; the four-class row is an ordering check.

Figure 5: Interval validation: audio DS maxima align with the sampled dissonance curve and the predefined rank reference.

Metric	Baseline	Gaussian	CQT	DS	$\Delta$
BLEU $\uparrow$	.2987 $\pm$ .0019	.2985 $\pm$ .0019	.3056 $\pm$ .0015	.3074 $\pm$ .0014	$+.0087$
METEOR $\uparrow$	.3761 $\pm$ .0017	.3759 $\pm$ .0018	.3838 $\pm$ .0014	.3857 $\pm$ .0013	$+.0096$
ROUGE-L $\uparrow$	.4556 $\pm$ .0018	.4554 $\pm$ .0019	.4643 $\pm$ .0016	.4671 $\pm$ .0015	$+.0115$
BERTScore-R $\uparrow$	.8952 $\pm$ .0007	.8950 $\pm$ .0006	.8996 $\pm$ .0006	.9024 $\pm$ .0015	$+.0072$
Test loss $\downarrow$	.625 $\pm$ .004	.626 $\pm$ .004	.607 $\pm$ .003	.600 $\pm$ .002	$-.025$
Perplexity $\downarrow$	1.868 $\pm$ .008	1.870 $\pm$ .008	1.836 $\pm$ .005	1.822 $\pm$ .004	$-.046$

Table 2: MusicQA results over six paired training seeds (mean

\pm

SD).

\Delta

is DS minus Baseline.

Metric	Baseline	Gaussian	CQT	DS
J-PR	$.1539$	$.1537$	$.1564$	.1580
J-ROC	$.7806$	$.7801$	$.7828$	.7841
DEAM-V	$.5169$	$.5164$	$.5272$	.5355
DEAM-A	$.6209$	$.6202$	$.6260$	.6291
Emo-V	$.6487$	$.6479$	$.6575$	.6642
Emo-A	$.7598$	$.7590$	$.7642$	.7668

Table 3: Music2Emo results over six paired seeds. J-PR/J-ROC are macro tag averages; V/A denote valence/arousal

R^{2}

. Only

\overline{R^{2}}_{\mathrm{VA}}

reports mean

\pm

SD.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Index alignment for the frequency-axis correlation form of intrinsic DS.

Figure 2: Vectorized intrinsic-DS computation after replacing the explicit pairwise sum by valid correlation.

Figure 3: Index alignment for cross-DS correlation using a fixed or time-varying reference spectrum.

Figure 4: Vectorized cross-DS computation using a reusable reference-relation vector.

Item	Setting
Audio / frame rate	22.05 kHz mono / 4 frames s -1
CQT grid	8 octaves, 72 bins octave -1 , $K=576$ , $f_{\min}=\mathrm{C1}$
Kernel	$Q=60$ , relative tolerance $\alpha=.01$ , octave folding
Preprocessing	linear magnitude, relative-frame soft gate at 0.1, context window 4 s
Peak selection	prominence .015; other peak filters disabled
Scaling	target and reference normalized separately; final factor $1/K$

Appendix

Table 1: Controlled-validation configuration.

Figure 5: Four-octave view of the normalized relation kernel, illustrating its octave-folded repetition. Integer-semitone samples are marked for reference.

Note	Rank	$D_{\max}$	Note	Rank	$D_{\max}$
C4	1	.000253	G4	3	.001145
C $\sharp$ 4	13	.004379	G $\sharp$ 4	7	.001913
D4	11	.003476	A4	6	.001703
D $\sharp$ 4	8	.002463	A $\sharp$ 4	10	.002583
E4	5	.002298	B4	12	.002802
F4	4	.001614	C5	2	.000030

Appendix

Table 2: Interval results under the C4 tonic reference.

Figure 6: Interval validation details: audio DS maxima, sampled kernel values, and the predefined rank reference.

Figure 7: Representative interval DS maps for unison, minor second, perfect fifth, and octave.

Chord	Semitones	Rank	$D_{\max}$	Chord	Semitones	Rank	$D_{\max}$
Major-1	[0,4,7]	1	.005691	Sus-1	[0,7,17]	7	.005641
Major-2	[0,3,8]	5	.006941	Sus-2	[0,2,7]	6	.006758
Major-3	[0,9,17]	3	.006299	Sus-3	[0,10,17]	4	.007769
Minor-1	[0,3,7]	2	.006058	Dim-1	[0,3,6]	12	.015950
Minor-2	[0,4,9]	10	.006738	Dim-2	[0,9,15]	9	.008581
Minor-3	[0,8,17]	8	.006363	Dim-3	[0,9,18]	11	.008172

Appendix

Table 3: Extended chord-quality validation. Semitone arrays are relative to C4.

Figure 8: Representative chord-quality ordering.

Figure 9: Extended chord-quality and voicing results.

Figure 10: Representative tonic-referenced chord-quality spectra.

File	First chord	Group	Rank	$D_{\max}$
CC	C–E–G	tonic	1	.003848
DC	D–F–A	predominant	2	.009701
EC	E–G–B	tonic	1	.005311
FC	F–A–C	predominant	2	.008495
G7C	G–B–D–F	dominant	3	.010312
AC	A–C–E	tonic	1	.005695

Appendix

Table 4: Chord connections to C major.

Figure 11: Functional chord-connection results.

Figure 12: Reference-conditioned spectra for the seven chord connections.

Mode	Rank	$D_{\mathrm{sum}}$
Ionian	1	.030220
Mixolydian	2	.030273
Lydian	3	.033697
Dorian	4	.030704
Aeolian	5	.031444
Phrygian	6	.033937

Appendix

Table 5: Church-mode results.

Figure 13: Church-mode total DS and the predefined ordinal reference used in the main-paper correlation.

Figure 14: Circular peak patterns for all available scale recordings. Only the seven church modes are used for the ordinal correlation in the main paper.

Figure 15: DS curves for the available scale recordings, illustrating scale-dependent consonance profiles.

Audio	Duration (s)	$D_{\max}$	Audio	Duration (s)	$D_{\max}$
Saxophone	14.005	.003134	Violins	15.531	.005633
Guitar	14.005	.003170	Alto	15.214	.007333
Piano	14.099	.004385	Trumpet	14.005	.011913
Flute	15.724	.029903	24-TET sequence	26.005	.003751

Appendix

Table 6: Exploratory timbre and microtonal examples.

Figure 16: Exploratory 24-TET spectra at 50-cent resolution. The timbre values in Table 6 are descriptive and are not assigned a universal ranking.

Figure 17: Primary seven-level result: the absolute-amplitude level proxy changes substantially while the maximum DS peak remains tightly concentrated.

Panel of Figure 18 — Figure 18: Level-controlled stimuli. Left: CQT magnitude. Right: intrinsic DS.

Panel of Figure 19 — Figure 19: Within-file crescendo visualization. Left: CQT magnitude. Right: intrinsic DS.

Metric	vs. Baseline	vs. Gaussian	vs. CQT
BLEU $\uparrow$	+.0087	+.0089	+.0018
METEOR $\uparrow$	+.0096	+.0098	+.0019
ROUGE-L $\uparrow$	+.0115	+.0117	+.0028
BERTScore-R $\uparrow$	+.0072	+.0074	+.0028
Test loss $\downarrow$	-.025	-.026	-.007
Perplexity $\downarrow$	-.046	-.048	-.014

Appendix

Table 7: Additional MusicQA comparisons. Entries are DS minus the indicated control; negative values are improvements for loss and perplexity.

ID	Audio ID	Dataset track index	Original Token F1	ID	Audio ID	Dataset track index	Original Token F1
01	012895.wav	661609	.000	06	013055.wav	873804	.071
02	012815.wav	477309	.038	07	012553.wav	8308	.111
03	012725.wav	267904	.063	08	012986.wav	796003	.073
04	012786.wav	387501	.087	09	012972.wav	768506	.094
05	012750.wav	345903	.000	10	013063.wav	875005	.103

Appendix

Table 8: Provenance and score ownership for the MusicQA audit examples. Audio ID and dataset track index reproduce the audio and trackId fields in the accompanying demo. Original Token F1 applies only to the original MU-LLaMA output.

ID	Question	Reference answer	Original MU-LLaMA	DS-augmented MU-LLaMA
01	What is the term used to describe a style of music that features a strong emphasis on melody and harmony?	Acoustic rock is a term used to describe a style of music that features a strong emphasis on melody and harmony. Acoustic rock typically combines elements of rock and folk music, with acoustic guitar, harmonica, and vocals as the main instruments.	Classical.	Acoustic rock.
02	What type of orchestral music is characterized by complex harmonies and intricate compositions?	Progressive orchestral. Progressive orchestral music incorporates elements of progressive rock, avant-garde, and experimental music, and is characterized by complex harmonies and intricate compositions.	Symphonic music.	Progressive orchestral music.
03	What can be inferred from the audio?	The audio blends folk, punk rock, rock, and ska. It uses acoustic guitars, bass, drums, and occasional brass and strings, with gruff vocals, harmonies, and call-and-response sections.	The song is about a man’s love for a woman.	The music blends folk, punk rock, ska, and acoustic instrumentation.
04	Describe the audio in detail.	The audio is a soothing and upbeat tune with a catchy melody and harmonious vocals. Harp, synthesizer, and piano create an elegant, layered, and enjoyable arrangement.	The music is described as slow and weird.	The music is described as soothing, upbeat, and harmonious.
05	What is the genre of music that is characterized by a slow and smooth rhythm, often played on a saxophone?	Bossanova. It originated in Brazil and is known for its relaxed and romantic sound, often featuring a slow and smooth rhythm.	Jazz.	Bossanova.

Appendix

Table 9: MusicQA answer-level audit examples, cases 01–05. Reference answers and dataset identifiers are reproduced from the accompanying demo.

ID	Question	Reference answer	Original MU-LLaMA	DS-augmented MU-LLaMA
06	What can be inferred from the audio?	The audio appears to be a movie or TV soundtrack with a moody, atmospheric character. Its slow, mellow melody and layered guitars and piano evoke longing, nostalgia, and introspection.	The song is about love and heartbreak.	The music feels moody, mellow, nostalgic, and introspective.
07	Describe the audio in detail.	A relaxing instrumental composition combining electronic, lounge, and oriental music. Its slow, smooth tempo and minimal instrumentation create a calming atmosphere.	The music is described as having a weird and otherworldly quality.	The music is described as relaxing, smooth, and gently oriental.
08	What can be inferred from the audio?	A relaxing and upbeat trip-hop track for chilling out. Synthesizer, strings, bass, drums, Rhodes, and trumpet create a dreamy, smooth, and sophisticated atmosphere.	The song is about love and heartbreak.	The track is relaxing, upbeat, dreamy, and sophisticated.
09	Describe the audio in detail.	A calm orchestral composition. Acoustic bass, cello, piano, violin, bell, and oboe create a warm, elegant, magical, and serene piece for relaxing.	The music is described as having a weird and otherworldly quality.	The music is described as calm, warm, elegant, and serene.
10	Describe the audio in detail.	A fusion of classical and ambient music with flute and synthesizer, a slow and mellow pace, bass and electric guitar foundation, and soulful background vocals.	The music is described as having a weird and scary atmosphere.	The music is described as slow, mellow, soulful, and ambient.

Appendix

Table 10: MusicQA answer-level audit examples, cases 06–10 (continued).

Variant	BERTScore-R $\uparrow$	$\Delta$ vs. proposed
Global DS	.8985 $\pm$ .0007	$-.0030$
Randomized kernel	.8992 $\pm$ .0009	$-.0023$
Temporal DS, shuffled	.8997 $\pm$ .0007	$-.0018$
Temporal DS, pre-projection	.9004 $\pm$ .0006	$-.0011$
Temporal DS, proposed	.9015 $\pm$ .0013	–

Appendix

Table 11: Exploratory MusicQA mechanism analysis.

Variant	Emotion $\overline{R^{2}}_{\mathrm{VA}}$ $\uparrow$
Global DS	.6524 $\pm$ .0015
Temporal DS, shuffled	.6550 $\pm$ .0017
Global DS, early concatenation	.6558 $\pm$ .0016
Temporal DS, proposed	.6589 $\pm$ .0018

Appendix

Table 12: Exploratory Music2Emo mechanism analysis.

Source	BLEU	METEOR	ROUGE-L	BERTScore-R
Original report (released script)	.3060	.3850	.4660	.9010
Our baseline (specified metrics)	.2987	.3761	.4556	.8952
Absolute difference	.0073	.0089	.0104	.0058

Appendix

Table 13: Contextual MU-LLaMA comparison. The original values use the released scoring script on 4,500 QA pairs from 500 tracks ( Liu et al. 2024 ) ; our values use the explicitly specified evaluator and are six-seed means on the custom 5,040-pair, 560-track set.

Source	J-PR	J-ROC	DEAM-V	DEAM-A	Emo-V	Emo-A	PM-V	PM-A
Original report	.1543	.7810	.5184	.6228	.6512	.7616	.5473	.7940
Our baseline	.1539	.7806	.5169	.6209	.6487	.7598	.5451	.7926
Absolute difference	.0004	.0004	.0015	.0019	.0025	.0018	.0022	.0014

Appendix

Table 14: Contextual Music2Emo comparison. The original report uses its 30-second segment augmentation and original training protocol ( Kang and Herremans 2025 ) ; our baseline uses the modified fixed-excerpt protocol described here.

Seed	Baseline	Gaussian	CQT	DS
17	.8943	.8942	.8990	.9001
42	.8950	.8948	.8996	.9017
101	.8954	.8951	.8998	.9027
2025	.8948	.8947	.8989	.9028
2026	.8962	.8960	.8997	.9024
3407	.8955	.8952	.9006	.9047

Appendix

Table 15: MusicQA BERTScore-R for the six paired seeds.

Seed	Baseline	Gaussian	CQT	DS
17	.6453	.6449	.6519	.6560
42	.6472	.6466	.6542	.6587
101	.6481	.6476	.6552	.6596
2025	.6464	.6457	.6525	.6580
2026	.6494	.6488	.6557	.6602
3407	.6476	.6468	.6555	.6609

Appendix

Table 16: Music2Emo

\overline{R^{2}}_{\mathrm{VA}}

for the six paired seeds.

Task	Comparison	Mean $\Delta$	$p_{\mathrm{H}}^{(t)}$	$p_{\mathrm{H}}^{(\mathrm{sign})}$
MusicQA	DS vs. Baseline	+.0072	$<.001$	.0938
MusicQA	DS vs. Gaussian	+.0074	$<.001$	.0938
MusicQA	DS vs. CQT	+.0028	.0017	.0938
Music2Emo	DS vs. Baseline	+.0116	$<.001$	.0938
Music2Emo	DS vs. Gaussian	+.0122	$<.001$	.0938
Music2Emo	DS vs. CQT	+.0047	$<.001$	.0938

Appendix

Table 17: Paired comparisons on the two designated primary endpoints using unrounded seed-level values.

p_{\mathrm{H}}^{(t)}

is Holm-adjusted paired-

t

;

p_{\mathrm{H}}^{(\mathrm{sign})}

is the Holm-adjusted exact two-sided sign test based only on paired directions. Each task has one three-comparison Holm family.

Music Understanding

See all 21 papers

Neural Music Enhancement with Dual Time-Frequency Spectral Representations for Prediction and Discrimination
Sep 3, 2026Fei Liu, Yang Ai, Zhen-Hua LingFrequency-Domain Feature Learning
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China
MERIT: Learning Disentangled Music Representations for Audio Similarity
May 26, 2026Abhinaba Roy, Junyi Liang, Dorien HerremansDisentangled Representation Learning Audio Representation Learning
Singapore University of Technology and Design
Perceptually Motivated Alignment and Interpolation of Pitch-Aligned Time-Frequency Representations
Sep 25, 2026Shahan Nercessian, Jeff Sontag, Alejandro KoretzkyMusic Information Retrieval
Splice New York, USA

Sep 3, 2026J/K move · Enter open · S save

Neural Music Enhancement with Dual Time-Frequency Spectral Representations for Prediction and Discrimination

Fei Liu, Yang Ai, Zhen-Hua Ling

National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China

Frequency-Domain Feature Learning

Open paper AskPDF arXiv

Non-professional music recordings shared online often suffer from background noise and reverberation, degrading perceived quality and limiting reuse. This paper proposes DSME, a music enhancement model based on dual time-frequency spectral representations. Within a generative adversarial framework, DSME uses short-time Fourier transform (STFT) spectra for generation and constant-Q transform (CQT) spectra for discrimination. Leveraging STFT's fixed window, invertibility, and predictability, the generator estimates clean amplitude-phase spectra from degraded inputs and reconstructs waveforms via inverse STFT. Exploiting CQT's log-frequency, variable-window structure aligned with musical octaves, we design an octave-segmented CQT discriminator. We also introduce a chroma-spectrum loss to emphasize pitch and harmonic consistency. Experiments show DSME outperforms baselines in objective and subjective tests, validating the effectiveness of the dual-spectrum approach.

Abstract

Music Understanding

Explore similar work

Neural Music Enhancement with Dual Time-Frequency Spectral Representations for Prediction and Discrimination

MERIT: Learning Disentangled Music Representations for Audio Similarity

Perceptually Motivated Alignment and Interpolation of Pitch-Aligned Time-Frequency Representations