Continuous-latent audio autoencoders form the backbone of latent music generators, yet decoders at high compression rates commonly exhibit three failure modes: high-frequency loss, phase incoherence, and stereo-image collapse. These share a structural root: waveform autoencoders lack an explicit frequency axis, leaving no handle for targeted per-band correction. Among five matched-budget representations, the complex STFT achieves the lowest full-band and high-frequency spectral distances, providing direct access to magnitude and phase at every bin. Building on this, we present ear-VAE2, a complex-spectral autoencoder with cross-channel interaction. Spec-SnakeBeta learns a periodic activation per frequency bin with frequency-dependent initialization, outperforming other activation variants while using fewer parameters than the fully independent variant. Duplex-Aware Refiner applies band-specific corrections to magnitude and phase following duplex theory of sound localization. On the 546-track Song Describer Dataset, ear-VAE2 achieves the best point estimates on five of seven reconstruction metrics. The Duplex-Aware Refiner reduces Mel Distance by 19.4% and uses ~45% fewer residual-output dimensions than the Unconstrained Refiner, while also lowering spectral distances, spatial-cue errors, and receiving higher ratings from professional engineers. The downstream generator using ear-VAE2 latents achieves better point estimates on all 12 automatic metrics.Demo page is available at https://eps-acoustic-revolution-lab.github.io/EAR_VAE2/.
Figures & tables
Figure 1: Architecture of ε ar-VAE2 . Stereo audio is transformed via STFT into a complex spectrogram; a Spec Encoder compresses it to a 25 Hz continuous latent and a Spec Decoder reconstructs a coarse complex spectrogram; the highest-frequency (Nyquist) STFT bin is removed to facilitate frequency downsampling. The Duplex-Aware Refiner subsequently applies band-specific complex corrections and restores the Nyquist bin to produce the full-bin spectrogram, which the iSTFT resynthesizes into a waveform.
Figure 2: Spec-SnakeBeta analysis. (a) Activation curves at representative frequencies (0.5, 3, 10, 20 kHz) vs. channel-wise SnakeBeta (black dashed). (b) Learned deviation Δαf=αftrained−αf(0) from frequency-proportional initialization across encoder layers.
Figure 3: Frequency-resolved diagnosis of the trained encoder–decoder before refinement. (a) Learned frequency-indexed β~f per encoder layer; since β~f is initialized to zero, the values also represent displacements from initialization. (b) Per-frequency magnitude error. (c) Weighted angular error; the horizontal line marks the π/2 random-phase baseline. Vertical lines mark the refiner band boundaries, and the shaded region spans 0.5–5 kHz.
Representation
STFT ↓
STFT HF ↓
STFT LF ↓
Waveform Patch (Conv2D)
1.294
1.513
0.986
Complex STFT (real/imag)
1.122
1.178
1.137
Magnitude-only
1.228
1.376
1.213
Waveform (Conv1D)
1.589
1.740
1.358
Patch Transformer
2.222
2.350
1.749
Table 1: Targeted spectral results from the matched-budget representation study on the Song Describer Dataset. STFT HF / STFT LF are STFT Distance above 8 kHz and below 2 kHz (metric bands, not the refiner Low/Mid/High bands). Complex STFT gives the lowest full-band and high-frequency distances, while waveform patch gives the lowest low-frequency distance. Bold : best; underline : second best.
Config
Temporal
Spectral
Mel
Stereo
System
SR
Lat. rate
SI-SDR ↑
STFT ↓
STFT log ↓
Mel ↓
Mel log ↓
CCPC ↑
SPE ↓
ε ar-VAE
44.1
43.1
12.4
0.880
0.079
0.509
0.095
0.973
0.267
SA-Open
44.1
21.5
6.7
1.016
0.089
0.612
0.106
0.933
0.276
LeVo 2
48
25.0
8.1
0.971
0.086
0.599
0.103
0.947
0.273
SAME-L
44.1
10.8
12.5
0.986
0.079
0.539
0.096
0.970
0.276
ε ar-VAE2 (without refiner)
48
25.0
10.9
0.916
0.078
0.572
0.093
0.966
0.268
Table 2: Objective reconstruction on the Song Describer Dataset (546 tracks). Bold : best point estimate; underline : second-best. STFT and Mel follow the auraloss STFT magnitude L1 distance; STFT log and Mel log apply log1p to the spectrogram before the same L1 distance. The two ε ar-VAE2 rows report the pipeline without and with the Duplex-Aware Refiner. Systems operate at their native configurations, so this is a competitive comparison rather than a matched-budget comparison.
Efficiency
Spectral
Duplex-Cue
Subjective
Out. dims/ch
Mel ↓
STFT ↓
Low IPD ↓
High IPD ↓
Low ILD ↓
High ILD ↓
Rating ↑
Banded (ours)
532
0.658
1.019
0.7963
1.2907
0.1690
0.9804
0.75
Unconstrained Refiner
962
0.705
1.120
0.8155
1.2825
0.1840
1.1495
0.66
Table 3: Banded versus Unconstrained Refiner. Both variants use the same frozen decoder and training protocol on the Song Describer Dataset. Mel and STFT denote Mel Distance and STFT Distance. Low/High IPD (interaural phase difference) and ILD (interaural level difference) are mean absolute deviations from the reference, computed over the Low ( < 1.5 kHz) and High ( > 4 kHz) refiner bands rather than the HF/LF metric bands of § 4.2 . IPD errors are in radians and ILD errors in decibels, so magnitudes are comparable only within a column; Appendix J.1 gives both definitions. Rating is the mean paired rating on a 1–10 scale mapped to [0,1] . Bold : best.
Temporal
Spectral
Mel
Stereo
Variant
SI-SDR ↑
STFT ↓
STFT log ↓
Mel ↓
Mel log ↓
CCPC ↑
ELU
2.38
1.313
0.106
1.114
0.124
0.886
SiLU
1.78
1.334
0.104
1.126
0.123
0.883
GELU
1.78
1.382
0.107
1.193
0.125
0.881
SnakeBeta-C
3.40
1.379
0.106
1.194
0.124
0.897
Spec-SnakeBeta-F-Uniform
3.46
1.365
0.106
1.206
0.120
0.896
Table 4: Activation ablation. C is channel-indexed (feature channel, not left/right audio), F is frequency-indexed and channel-shared, and CF is channel–frequency independent. STFT and Mel follow the auraloss STFT magnitude L1 distance; STFT log and Mel log apply log1p before the same L1 distance. Bold : best; underline : second best.
System
SongBench ( ↑ )
SongEval ( ↑ )
Mel- ody
Arran- gement
Music- ality
Vo- cal
Instru- ment
Mix- ing
Struc- ture
Coher- ence
Music- ality
Memor- ability
Clar- ity
Natur- alness
LeVo 2 VAE
6.56
6.88
5.71
6.88
6.95
6.59
6.38
4.31
4.17
4.24
4.20
4.10
ε ar-VAE2 (-)
6.88
7.17
6.04
7.30
7.19
7.02
6.75
4.39
4.25
4.33
4.29
4.18
ε ar-VAE2
7.02
7.24
6.15
7.39
7.23
7.11
6.81
4.42
4.27
4.38
4.44
4.25
Table 5: Automatic downstream generation evaluation under matched conditioning and classifier-free guidance (CFG; 100 English and 100 Chinese songs per system). The same renderer architecture is retrained for each latent space; the two ε ar-VAE2 rows share the same latent and coarse decoder, and the ε ar-VAE2 (-) variant omits the optional duplex-aware refiner. Bold : best; underline : second best.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Aggregate
Freq-Band
Representation
SI-SDR ↑
CCPC ↑
Mel ↓
STFT ↓
STFT HF ↓
STFT LF ↓
Waveform Patch (Conv2D)
+ 3.98
0.865
1.184
1.294
1.513
0.986
Complex STFT (real/imag)
− 10.49
0.800
1.523
1.122
1.178
1.137
Magnitude-only
− 42.16
0.756
1.568
1.228
1.376
1.213
Waveform (Conv1D)
− 33.56
0.732
2.009
1.589
1.740
1.358
Patch Transformer
− 52.04
0.697
2.828
2.222
2.350
1.749
Appendix
Table 6: Input-representation study on the Song Describer Dataset (matched budget). Bold : best; underline : second best. STFT HF / STFT LF are STFT Distance above 8 kHz and below 2 kHz (metric bands, not the refiner Low/Mid/High bands).
Figure 4: Reconstruction spectrograms from the matched-budget representation study (48 kHz, nfft=2048 , hop =128 ). (A) complex STFT; (B) waveform patch; (C) magnitude-only; (D) waveform Conv1D; (E) patch Transformer; and (GT) ground truth. Complex STFT and waveform patch share the same Conv2D backbone.
Figure 5: Encoder gradient ℓ2 -norm (left) and magnitude reconstruction loss (right) over training, recorded during the same 100k-step runs under the shared protocol.
Check
Metric ID
Evidence
Condition
Meta & encoder
encoder_tag
Encoder string in container/stream tags
flag if lavf/lavc/ffmpeg/lame
mutagen_tags
Metadata sweep for conversion traces
flag if transcode/convert
Bitrate
has_non_monotonic_pts
PTS monotonicity
pass if false
bitrate_utilization
Realized / nominal bitrate
pass if ≥0.6
top10_bitrate_mean
Complex-segment mean bitrate
pass if ≥1.2× mean
vbr_variation
Bitrate coefficient of variation
pass if σ/μ≥0.15
Appendix
Table 7: Metric-level checklist used by the acoustic pipeline. Each row pairs a rule-engine identifier with a brief description and its pass/flag condition.
Figure 6: High-frequency cutoff detector examples. Blue: Welch spectrum of a verification segment; red dashed: detected cutoff fc ; gray dotted: independently estimated edge noise floor η^ .
Figure 7: Sample-level comparison over 80 ms at 48 kHz; the dashed line marks the same reference instant in every panel. (a) Original waveform. (b) A naive mixed-center round-trip (analysis center=False , synthesis center=True ) shifts the waveform left by one hop ( ≈10 ms). A fully uncentered Hann round-trip is instead rejected by the boundary NOLA check and therefore has no reconstructed trace. (c) Our explicitly padded wrapper produces T frames and recovers sample-level alignment.
Figure 8: Generator gradient-norm dynamics for Muon and AdamW, measured over a common training window (steps 40k–68k) in which all three runs are stable. (A) Per-step generator gradient norm ∥g∥ for Muon and two AdamW runs (AdamW 1, AdamW 2), shown on a shared linear scale; Muon and AdamW 2 use the same global batch size and AdamW 1 uses a larger batch. (B) Local spike rate: the percentage of steps at which ∥g∥ exceeds 3× (solid) or 5× (hatched) its trailing 500-step median.
Category
Param
Stage 1
Stage 2
Stage 3
Optimizer
Generator optim.
Muon
Muon
AdamW
Generator LR
10−4
1.5×10−4
1.5×10−4
Disc. optim.
—
Muon
AdamW
Disc. LR
—
3×10−5
10−6 ( Dw ) / 10−5 ( Ds )
Schedule
Training duration
1 epoch
5 epochs
3 epochs
Batch size / GPU
4
1
1
Appendix
Table 8: ε ar-VAE2 training configuration across three stages. Stage 1 is reconstruction-only; Stage 2 adds waveform-domain adversarial training; Stage 3 freezes the encoder–decoder and trains the refiner with Dw and Ds . Adv/FM denotes adversarial (Adv) / feature-matching (FM) loss weights.
Critic
logit(real)
logit(fake)
gap
Note
Dw(2)
0.619
0.409
0.210
Stage 2 checkpoint
Dw(3)
0.551
0.366
0.186
after Stage 3 continuation
Ds
0.485
0.034
0.451
trained spectral
Ds∘P
0.485
0.449
0.036
gauge removed
Dsrand
− 0.003
− 0.002
0.001
random init
Appendix
Table 9: Discriminability probe at the Stage 3 boundary ( n=48 segments): mean real/fake logits and real–fake gap. Dw(2) / Dw(3) are Dw at the end of Stage 2 and after Stage 3 continuation; Ds is the trained spectral-domain discriminator; Ds∘P scores inputs pre-projected onto the consistent subspace (gauge component removed); Dsrand is a randomly initialized spectral-domain discriminator.
Gradient
null-space energy fraction
gw(2) (Stage 2 Dw )
0.0002
gw(3) (Stage 3 Dw )
0.0002
gs ( Ds , direct)
0.9529
Appendix
Table 10: Gradient anatomy at the Stage 3 boundary ( n=24 ): fraction of each generator-side gradient’s energy inside the synthesis-null subspace. gw(2),gw(3) are waveform-path gradients (through the iSTFT) from Stage 2 and Stage 3 Dw ; gs is Ds ’s direct gradient.
Figure 9: Stage 3 consistency trajectory ( n=18 segments), scored by the fixed final spectral discriminator Ds . Left axis : inconsistency ∥S−PS∥/∥S∥ of the refiner output ( t=0 is the Stage 2 final generator). Right axis : Ds ’s fake logit; the dotted line marks its real baseline ( 0.485 ). The steepest descent follows the end of Ds ’s warmup. The coincident spectral-metric improvement is correlational, not an attribution (§ G.4 ).
Score
Anchor Description
8–10
Frequency balance suits the genre; mid/side energy ratio appropriate; no masking between instruments; dynamics well controlled with natural transients; spatial staging coherent with adequate depth; harmonic coloration (if any) enhances rather than muddies.
5–7
Minor spectral gaps or slight imbalance in one band; compression artifacts occasionally audible (pumping, over-limited transients); spatial image somewhat flat but without obvious panning errors; timbral quality acceptable with minor coloration.
2–4
Noticeable frequency-band dropout or harshness (e.g., sibilance, muddy low-mids); dynamics feel unnatural (over-compressed or under-controlled); spatial image collapsed or inconsistent; timbral distortion clearly audible.
1
Severe artifacts across multiple dimensions; reconstruction quality unacceptable for professional evaluation.
Appendix
Table 11: Subjective quality rubric. Engineers assign a single holistic score on a 1–10 scale; anchor descriptions guide consistent interpretation.
Model
centroid low (Hz)
centroid high (Hz)
ρ
inverted
ε ar-VAE2 (complex STFT)
323
137
2.36
✓
ε ar-VAE (waveform)
477
249
1.82
✓
LeVo 2
392
736
0.53
×
SAME-L
391
487
0.79
×
SA-Open
363
393
0.91
×
Appendix
Table 12: Latent temporal-frequency probe (spectral centroid of the separately decoded low-/high-temporal-frequency latent halves; averaged over our internal evaluation set). ρ>1 (“inverted”) means the rapidly varying latent decodes to audio with a lower spectral centroid. ε ar-VAE2 and ε ar-VAE are inverted; the other three models are not.
Figure 10: Latent temporal-frequency probe for one track (Track 1004034; temporal split at 0.5). Rows (top to bottom): ground-truth audio; full reconstruction; low-temporal-frequency latent half decoded; high-temporal-frequency latent half decoded. Columns (left to right): ε ar-VAE2 , LeVo 2, SAME-L, ε ar-VAE , and Stable Audio Open. For ε ar-VAE2 , the rapidly varying (high) latent half decodes to audio with a lower spectral centroid, matching Table 12 .
Latent representations are at the heart of the majority of modern generative models. In the audio domain they are typically produced by a neural-audio-codec autoencoder. In this work we introduce SAME (Semantically-Aligned Music autoEncoder), an autoencoder for stereo music and general audio that reaches a 4096× temporal compression ratio while maintaining reconstruction quality and downstream generative performance. We achieve this by combining a tranformer-based backbone with set of semantic regularisation approaches, phase-aware reconstruction losses and improved discriminator designs. The architecture delivers substantial computational cost benefits, through both its high compression ratio and its reliance on well-optimised transformer primitives. Two variants (a large SAME-L and a CPU-deployable SAME-S) are released in open-weights form.
Neural autoencoders underpin generative models. Practical, large-scale use of neural autoencoders for generative modeling necessitates fast encoding, low latent rates, and a single model across representations. Existing approaches are reconstruction-first: they incur high latent rates, slow encoding, and separate architectures for discrete vs. continuous latents and for different audio channel formats, hindering workflows from preprocessing to inference conditioning. We introduce a generative-first architecture for audio autoencoding that increases temporal downsampling from 2048x to 3360x and supports continuous and discrete representations and common audio channel formats in one model. By balancing compression, quality, and speed, it delivers 10x faster encoding, 1.6x lower rates, and eliminates channel-format-specific variants while maintaining competitive reconstruction quality. This enables applications previously constrained by processing costs: a 60-second mono signal compresses to 788 tokens, making generative modeling more tractable.
We introduce the Latent Fourier Transform (LatentFT), a framework that provides novel frequency-domain controls for generative music models. LatentFT combines a diffusion autoencoder with a latent-space Fourier transform to separate musical patterns by timescale. By masking latents in the frequency domain during training, our method yields representations that can be manipulated coherently at inference. This allows us to generate musical variations and blends from reference examples while preserving characteristics at desired timescales, which are specified as frequencies in the latent space. LatentFT parallels the role of the equalizer in music production: while traditional equalizers operates on audible frequencies to shape timbre, LatentFT operates on latent-space frequencies to shape musical structure. Experiments and listening tests show that LatentFT improves condition adherence and quality compared to baselines. We also present a technique for hearing frequencies in the latent space in isolation, and show different musical attributes reside in different regions of the latent spectrum. Our results show how frequency-domain control in latent space provides an intuitive, continuous frequency axis for conditioning and blending, advancing us toward more interpretable and interactive generative music models.