Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.
Figures & tables
Figure 1: Comparison of ”binaural images” split by discrete agent actions. On the left, a segment of Mel spectrogram. On the right, a segment of raw STFT output.
Figure 2: Architecture overview of the Mel Spectrogram VAE. The network accepts binaural Mel spectrograms as input, which are processed by an encoder consisting of three 2D convolutional layers paired with ReLU activations. The flattened feature map is passed through two parallel linear layers to predict the mean and log-variance of a 32-dimensional multivariate Gaussian posterior. Following the reparameterization trick, a sampled latent vector is passed to the decoder. The decoder reconstructs the signal using an initial linear layer followed by three 2D transposed convolutional layers with ReLU activations. The final transposed convolution, however, utilizes a Sigmoid activation instead of ReLU to bound the reconstructed output elements strictly within a [0,1] range.
Figure 3: Architecture overview of the STFT 4-Channel Stacked VAE. The architecture accepts binaural STFT magnitude and phase as a 4-channel input. Aside from the differing input dimensions, the network topology is functionally identical to the Mel Spectrogram VAE.
Figure 4: Architecture overview of the CVAE network, taking binaural complex STFT as input. The encoder comprises five complex 2D convolutional layers, each paired with batch normalization (omitted for visual clarity) and complex ReLU activations. The features then feed into three parallel complex linear branches with 256 dimensions, predicting the means ( μ ), log-variances ( logσ ), and pseudo-variances ( δ ). Following the complex reparameterization trick using auxiliary real and imaginary standard normal noise distributions ϵr and ϵi , the sampled latent vector is passed to the decoder. The decoder consists of two complex linear layers followed by five complex 2D transposed convolutions with batch normalization and complex ReLU activations. The final layer utilizes a real-valued Tanh activation to map the reconstructed real and imaginary coefficients of the output strictly back to the [−1,1] range.
Figure 5: Comparison of the ground truth versus reconstructed Mel spectrogram output from the Mel VAE.
Figure 6: Comparison of the ground truth versus reconstructed spectrogram output from the 4-Channel STFT VAE. To the left, we see the general reconstruction, from a zoom out. To the right, you see the zoomed-in reconstruction on the lower frequencies, where the model performs better, just so we can observe a bit more in detail the accuracy - or lack of it - of the magnitude and phase reconstructions.
Figure 7: Comparison of the ground truth versus reconstructed spectrogram output from the Complex VAE. To the left, we see the general reconstruction, from a zoom out. To the right, you see the zoomed-in reconstruction on the lower frequencies, where the model performs better, just so we can observe a bit more in detail the accuracy of the frequency and phase reconstructions.
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group +1