Fourier is Frontier: Frequency-Aware Autoencoding for High-Fidelity Music Reconstruction

Kangdi Wang; Yusheng Dai; Jin Xu

cs.SDAug 20, 2026

Fourier is Frontier: Frequency-Aware Autoencoding for High-Fidelity Music Reconstruction

Authors: Kangdi Wang, Yusheng Dai, Jin Xu

Organizations: Qwen Team, Alibaba · Monash University

Abstract

Continuous-latent audio autoencoders form the backbone of latent music generators, yet decoders at high compression rates commonly exhibit three failure modes: high-frequency loss, phase incoherence, and stereo-image collapse. These share a structural root: waveform autoencoders lack an explicit frequency axis, leaving no handle for targeted per-band correction. Among five matched-budget representations, the complex STFT achieves the lowest full-band and high-frequency spectral distances, providing direct access to magnitude and phase at every bin. Building on this, we present ear-VAE2, a complex-spectral autoencoder with cross-channel interaction. Spec-SnakeBeta learns a periodic activation per frequency bin with frequency-dependent initialization, outperforming other activation variants while using fewer parameters than the fully independent variant. Duplex-Aware Refiner applies band-specific corrections to magnitude and phase following duplex theory of sound localization. On the 546-track Song Describer Dataset, ear-VAE2 achieves the best point estimates on five of seven reconstruction metrics. The Duplex-Aware Refiner reduces Mel Distance by 19.4% and uses ~45% fewer residual-output dimensions than the Unconstrained Refiner, while also lowering spectral distances, spatial-cue errors, and receiving higher ratings from professional engineers. The downstream generator using ear-VAE2 latents achieves better point estimates on all 12 automatic metrics.Demo page is available at https://eps-acoustic-revolution-lab.github.io/EAR_VAE2/.

Figures & tables

Figure 1: Architecture of $\varepsilon$ ar-VAE2 . Stereo audio is transformed via STFT into a complex spectrogram; a Spec Encoder compresses it to a 25 Hz continuous latent and a Spec Decoder reconstructs a coarse complex spectrogram; the highest-frequency (Nyquist) STFT bin is removed to facilitate frequency downsampling. The Duplex-Aware Refiner subsequently applies band-specific complex corrections and restores the Nyquist bin to produce the full-bin spectrogram, which the iSTFT resynthesizes into a waveform.

Figure 2: Spec-SnakeBeta analysis. (a) Activation curves at representative frequencies (0.5, 3, 10, 20 kHz) vs. channel-wise SnakeBeta (black dashed). (b) Learned deviation

\Delta\alpha_{f}=\alpha^{\mathrm{trained}}_{f}-\alpha^{(0)}_{f}

from frequency-proportional initialization across encoder layers.

Figure 3: Frequency-resolved diagnosis of the trained encoder–decoder before refinement. (a) Learned frequency-indexed $\tilde{\beta}_{f}$ per encoder layer; since $\tilde{\beta}_{f}$ is initialized to zero, the values also represent displacements from initialization. (b) Per-frequency magnitude error. (c) Weighted angular error; the horizontal line marks the $\pi/2$ random-phase baseline. Vertical lines mark the refiner band boundaries, and the shaded region spans 0.5–5 kHz.

Representation	STFT $\downarrow$	STFT HF $\downarrow$	STFT LF $\downarrow$
Waveform Patch (Conv2D)	1.294	1.513	0.986
Complex STFT (real/imag)	1.122	1.178	1.137
Magnitude-only	1.228	1.376	1.213
Waveform (Conv1D)	1.589	1.740	1.358
Patch Transformer	2.222	2.350	1.749

Table 1: Targeted spectral results from the matched-budget representation study on the Song Describer Dataset. STFT HF / STFT LF are STFT Distance above

8

kHz and below

2

kHz (metric bands, not the refiner Low/Mid/High bands). Complex STFT gives the lowest full-band and high-frequency distances, while waveform patch gives the lowest low-frequency distance. Bold : best; underline : second best.

	Config		Temporal	Spectral		Mel		Stereo
System	SR	Lat. rate	SI-SDR $\uparrow$	STFT $\downarrow$	STFT log $\downarrow$	Mel $\downarrow$	Mel log $\downarrow$	CCPC $\uparrow$	SPE $\downarrow$
$\varepsilon$ ar-VAE	44.1	43.1	12.4	0.880	0.079	0.509	0.095	0.973	0.267
SA-Open	44.1	21.5	6.7	1.016	0.089	0.612	0.106	0.933	0.276
LeVo 2	48	25.0	8.1	0.971	0.086	0.599	0.103	0.947	0.273
SAME-L	44.1	10.8	12.5	0.986	0.079	0.539	0.096	0.970	0.276
$\varepsilon$ ar-VAE2 (without refiner)	48	25.0	10.9	0.916	0.078	0.572	0.093	0.966	0.268

Table 2: Objective reconstruction on the Song Describer Dataset (546 tracks). Bold : best point estimate; underline : second-best. STFT and Mel follow the auraloss STFT magnitude L1 distance; STFT log and Mel log apply

\operatorname{log1p}

to the spectrogram before the same L1 distance. The two

\varepsilon

ar-VAE2 rows report the pipeline without and with the Duplex-Aware Refiner. Systems operate at their native configurations, so this is a competitive comparison rather than a matched-budget comparison.

	Efficiency	Spectral		Duplex-Cue				Subjective
	Out. dims/ch	Mel $\downarrow$	STFT $\downarrow$	Low IPD $\downarrow$	High IPD $\downarrow$	Low ILD $\downarrow$	High ILD $\downarrow$	Rating $\uparrow$
Banded (ours)	532	0.658	1.019	0.7963	1.2907	0.1690	0.9804	0.75
Unconstrained Refiner	962	0.705	1.120	0.8155	1.2825	0.1840	1.1495	0.66

Table 3: Banded versus Unconstrained Refiner. Both variants use the same frozen decoder and training protocol on the Song Describer Dataset. Mel and STFT denote Mel Distance and STFT Distance. Low/High IPD (interaural phase difference) and ILD (interaural level difference) are mean absolute deviations from the reference, computed over the Low (

<

1.5 kHz) and High (

>

4 kHz) refiner bands rather than the HF/LF metric bands of § 4.2 . IPD errors are in radians and ILD errors in decibels, so magnitudes are comparable only within a column; Appendix J.1 gives both definitions. Rating is the mean paired rating on a 1–10 scale mapped to

[0,1]

. Bold : best.

	Temporal	Spectral		Mel		Stereo
Variant	SI-SDR $\uparrow$	STFT $\downarrow$	STFT log $\downarrow$	Mel $\downarrow$	Mel log $\downarrow$	CCPC $\uparrow$
ELU	2.38	1.313	0.106	1.114	0.124	0.886
SiLU	1.78	1.334	0.104	1.126	0.123	0.883
GELU	1.78	1.382	0.107	1.193	0.125	0.881
SnakeBeta-C	3.40	1.379	0.106	1.194	0.124	0.897
Spec-SnakeBeta-F-Uniform	3.46	1.365	0.106	1.206	0.120	0.896

Table 4: Activation ablation.

C

is channel-indexed (feature channel, not left/right audio),

F

is frequency-indexed and channel-shared, and

CF

is channel–frequency independent. STFT and Mel follow the auraloss STFT magnitude L1 distance; STFT log and Mel log apply

\operatorname{log1p}

before the same L1 distance. Bold : best; underline : second best.

System	SongBench ( $\uparrow$ )							SongEval ( $\uparrow$ )
System	Mel- ody	Arran- gement	Music- ality	Vo- cal	Instru- ment	Mix- ing	Struc- ture	Coher- ence	Music- ality	Memor- ability	Clar- ity	Natur- alness
LeVo 2 VAE	6.56	6.88	5.71	6.88	6.95	6.59	6.38	4.31	4.17	4.24	4.20	4.10
$\varepsilon$ ar-VAE2 (-)	6.88	7.17	6.04	7.30	7.19	7.02	6.75	4.39	4.25	4.33	4.29	4.18
$\varepsilon$ ar-VAE2	7.02	7.24	6.15	7.39	7.23	7.11	6.81	4.42	4.27	4.38	4.44	4.25

Table 5: Automatic downstream generation evaluation under matched conditioning and classifier-free guidance (CFG; 100 English and 100 Chinese songs per system). The same renderer architecture is retrained for each latent space; the two

\varepsilon

ar-VAE2 rows share the same latent and coarse decoder, and the

\varepsilon

ar-VAE2 (-) variant omits the optional duplex-aware refiner. Bold : best; underline : second best.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Aggregate				Freq-Band
Representation	SI-SDR $\uparrow$	CCPC $\uparrow$	Mel $\downarrow$	STFT $\downarrow$	STFT HF $\downarrow$	STFT LF $\downarrow$
Waveform Patch (Conv2D)	$+$ 3.98	0.865	1.184	1.294	1.513	0.986
Complex STFT (real/imag)	$-$ 10.49	0.800	1.523	1.122	1.178	1.137
Magnitude-only	$-$ 42.16	0.756	1.568	1.228	1.376	1.213
Waveform (Conv1D)	$-$ 33.56	0.732	2.009	1.589	1.740	1.358
Patch Transformer	$-$ 52.04	0.697	2.828	2.222	2.350	1.749

Appendix

Table 6: Input-representation study on the Song Describer Dataset (matched budget). Bold : best; underline : second best. STFT HF / STFT LF are STFT Distance above

8

kHz and below

2

kHz (metric bands, not the refiner Low/Mid/High bands).

Figure 4: Reconstruction spectrograms from the matched-budget representation study (48 kHz, $n_{\text{fft}}{=}2048$ , hop ${=}128$ ). (A) complex STFT; (B) waveform patch; (C) magnitude-only; (D) waveform Conv1D; (E) patch Transformer; and (GT) ground truth. Complex STFT and waveform patch share the same Conv2D backbone.

Figure 5: Encoder gradient $\ell_{2}$ -norm (left) and magnitude reconstruction loss (right) over training, recorded during the same 100k-step runs under the shared protocol.

Check	Metric ID	Evidence	Condition
Meta & encoder	encoder_tag	Encoder string in container/stream tags	flag if lavf/lavc/ffmpeg/lame
	mutagen_tags	Metadata sweep for conversion traces	flag if transcode/convert
Bitrate	has_non_monotonic_pts	PTS monotonicity	pass if false
	bitrate_utilization	Realized / nominal bitrate	pass if $\geq 0.6$
	top10_bitrate_mean	Complex-segment mean bitrate	pass if $\geq 1.2\times$ mean
	vbr_variation	Bitrate coefficient of variation	pass if $\sigma/\mu\geq 0.15$

Appendix

Table 7: Metric-level checklist used by the acoustic pipeline. Each row pairs a rule-engine identifier with a brief description and its pass/flag condition.

Figure 6: High-frequency cutoff detector examples. Blue: Welch spectrum of a verification segment; red dashed: detected cutoff

f_{c}

; gray dotted: independently estimated edge noise floor

\hat{\eta}

Figure 7: Sample-level comparison over 80 ms at 48 kHz; the dashed line marks the same reference instant in every panel. (a) Original waveform. (b) A naive mixed-center round-trip (analysis center=False , synthesis center=True ) shifts the waveform left by one hop (

{\approx}10

ms). A fully uncentered Hann round-trip is instead rejected by the boundary NOLA check and therefore has no reconstructed trace. (c) Our explicitly padded wrapper produces

T

frames and recovers sample-level alignment.

Figure 8: Generator gradient-norm dynamics for Muon and AdamW, measured over a common training window (steps 40k–68k) in which all three runs are stable. (A) Per-step generator gradient norm

\|g\|

for Muon and two AdamW runs (AdamW 1, AdamW 2), shown on a shared linear scale; Muon and AdamW 2 use the same global batch size and AdamW 1 uses a larger batch. (B) Local spike rate: the percentage of steps at which

\|g\|

exceeds

3\times

(solid) or

5\times

(hatched) its trailing 500-step median.

Category	Param	Stage 1	Stage 2	Stage 3
Optimizer	Generator optim.	Muon	Muon	AdamW
	Generator LR	$10^{-4}$	$1.5{\times}10^{-4}$	$1.5{\times}10^{-4}$
	Disc. optim.	—	Muon	AdamW
	Disc. LR	—	$3{\times}10^{-5}$	$10^{-6}$ ( $D_{w}$ ) / $10^{-5}$ ( $D_{s}$ )
Schedule	Training duration	1 epoch	5 epochs	3 epochs
	Batch size / GPU	4	1	1

Appendix

Table 8:

\varepsilon

ar-VAE2 training configuration across three stages. Stage 1 is reconstruction-only; Stage 2 adds waveform-domain adversarial training; Stage 3 freezes the encoder–decoder and trains the refiner with

D_{w}

and

D_{s}

. Adv/FM denotes adversarial (Adv) / feature-matching (FM) loss weights.

Critic	logit(real)	logit(fake)	gap	Note
$D_{w}^{(2)}$	0.619	0.409	0.210	Stage 2 checkpoint
$D_{w}^{(3)}$	0.551	0.366	0.186	after Stage 3 continuation
$D_{s}$	0.485	0.034	0.451	trained spectral
$D_{s}{\circ}P$	0.485	0.449	0.036	gauge removed
$D_{s}^{\text{rand}}$	$-$ 0.003	$-$ 0.002	0.001	random init

Appendix

Table 9: Discriminability probe at the Stage 3 boundary (

n{=}48

segments): mean real/fake logits and real–fake gap.

D_{w}^{(2)}

D_{w}^{(3)}

are

D_{w}

at the end of Stage 2 and after Stage 3 continuation;

D_{s}

is the trained spectral-domain discriminator;

D_{s}{\circ}P

scores inputs pre-projected onto the consistent subspace (gauge component removed);

D_{s}^{\text{rand}}

is a randomly initialized spectral-domain discriminator.

Gradient	null-space energy fraction
$g_{w}^{(2)}$ (Stage 2 $D_{w}$ )	0.0002
$g_{w}^{(3)}$ (Stage 3 $D_{w}$ )	0.0002
$g_{s}$ ( $D_{s}$ , direct)	0.9529

Appendix

Table 10: Gradient anatomy at the Stage 3 boundary (

n{=}24

): fraction of each generator-side gradient’s energy inside the synthesis-null subspace.

g_{w}^{(2)},g_{w}^{(3)}

are waveform-path gradients (through the iSTFT) from Stage 2 and Stage 3

D_{w}

;

g_{s}

D_{s}

’s direct gradient.

Figure 9: Stage 3 consistency trajectory (

n{=}18

segments), scored by the fixed final spectral discriminator

D_{s}

. Left axis : inconsistency

\|S{-}PS\|/\|S\|

of the refiner output (

t{=}0

is the Stage 2 final generator). Right axis :

D_{s}

’s fake logit; the dotted line marks its real baseline (

0.485

). The steepest descent follows the end of

D_{s}

’s warmup. The coincident spectral-metric improvement is correlational, not an attribution (§ G.4 ).

Score	Anchor Description
8–10	Frequency balance suits the genre; mid/side energy ratio appropriate; no masking between instruments; dynamics well controlled with natural transients; spatial staging coherent with adequate depth; harmonic coloration (if any) enhances rather than muddies.
5–7	Minor spectral gaps or slight imbalance in one band; compression artifacts occasionally audible (pumping, over-limited transients); spatial image somewhat flat but without obvious panning errors; timbral quality acceptable with minor coloration.
2–4	Noticeable frequency-band dropout or harshness (e.g., sibilance, muddy low-mids); dynamics feel unnatural (over-compressed or under-controlled); spatial image collapsed or inconsistent; timbral distortion clearly audible.
1	Severe artifacts across multiple dimensions; reconstruction quality unacceptable for professional evaluation.

Appendix

Table 11: Subjective quality rubric. Engineers assign a single holistic score on a 1–10 scale; anchor descriptions guide consistent interpretation.

Model	centroid ${}_{\text{low}}$ (Hz)	centroid ${}_{\text{high}}$ (Hz)	$\rho$	inverted
$\varepsilon$ ar-VAE2 (complex STFT)	323	137	2.36	✓
$\varepsilon$ ar-VAE (waveform)	477	249	1.82	✓
LeVo 2	392	736	0.53	×
SAME-L	391	487	0.79	×
SA-Open	363	393	0.91	×

Appendix

Table 12: Latent temporal-frequency probe (spectral centroid of the separately decoded low-/high-temporal-frequency latent halves; averaged over our internal evaluation set).

\rho>1

(“inverted”) means the rapidly varying latent decodes to audio with a lower spectral centroid.

\varepsilon

ar-VAE2 and

\varepsilon

ar-VAE are inverted; the other three models are not.

Figure 10: Latent temporal-frequency probe for one track (Track 1004034; temporal split at 0.5). Rows (top to bottom): ground-truth audio; full reconstruction; low-temporal-frequency latent half decoded; high-temporal-frequency latent half decoded. Columns (left to right): $\varepsilon$ ar-VAE2 , LeVo 2, SAME-L, $\varepsilon$ ar-VAE , and Stable Audio Open. For $\varepsilon$ ar-VAE2 , the rapidly varying (high) latent half decodes to audio with a lower spectral centroid, matching Table 12 .

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

SAME: A Semantically-Aligned Music Autoencoder
May 18, 2026Julian D. Parker, Zach Evans, CJ Carr +4Audio Representation Learning Neural Audio Codecs
Stability AI
A Generative-First Neural Audio Autoencoder
Feb 17, 2026Jonah Casebeer, Ge Zhu, Zhepei Wang +1Autoencoders Audio Representation Learning
Adobe Research
Latent Fourier Transform
Apr 20, 2026Mason Wang, Cheng-Zhi Anna HuangControllable Music Generation Frequency-Domain Feature Learning
CSAIL

May 18, 2026J/K move · Enter open · S save

SAME: A Semantically-Aligned Music Autoencoder

Julian D. Parker, Zach Evans, CJ Carr, Zachary Zukowski+3

Stability AI

Audio Representation Learning Neural Audio Codecs Autoencoders

Open paper AskPDF arXiv

Latent representations are at the heart of the majority of modern generative models. In the audio domain they are typically produced by a neural-audio-codec autoencoder. In this work we introduce SAME (Semantically-Aligned Music autoEncoder), an autoencoder for stereo music and general audio that reaches a 4096

\times

temporal compression ratio while maintaining reconstruction quality and downstream generative performance. We achieve this by combining a tranformer-based backbone with set of semantic regularisation approaches, phase-aware reconstruction losses and improved discriminator designs. The architecture delivers substantial computational cost benefits, through both its high compression ratio and its reliance on well-optimised transformer primitives. Two variants (a large SAME-L and a CPU-deployable SAME-S) are released in open-weights form.

Abstract

Audio Representation Learning

Explore similar work

SAME: A Semantically-Aligned Music Autoencoder

A Generative-First Neural Audio Autoencoder

Latent Fourier Transform