Latent Terrain: Adapting Neural Audio Autoencoders as Design Materials in NIME
Organizations: Centre for Digital Music, Queen Mary University of London London, UK · Graduate School of Media and Governance, Keio University Kanagawa, Japan · Creative Computing Institute, University of the Arts London London, UK
Abstract
Neural audio autoencoders, a deep learning method for sound synthesis, are increasingly popular in AI-enhanced NIMEs. It is timely to explore as a community how this technological opportunity has opened a domain of design in NIME. This paper focuses on one compelling technique of using autoencoders for sound synthesis: navigating their latent space as a generative sound space. We introduce Latent Terrain, a Max/MSP tool package designed to tailor latent spaces into corpus-based sound spaces for NIMEs. We describe the rationale and development process of Latent Terrain to offer insights into the use of autoencoders in a material-oriented crafting space of musical interface design. We deliver an annotated portfolio resulting from a collaborative artistic exploration of Latent Terrain with four NIME makers, to showcase the design possibilities opened by autoencoders. We reflect on our practice-based account to discuss the challenges and opportunities of enabling neural audio autoencoders as design materials for AI-enhanced NIMEs.
Figures & tables
| Autoencoder | Year | Latent dim. | Latent type 5 | Regularisation 6 | Comp. 7 | In Max 8 | Used in NIME design | |
| RAVE 1 | 2021 | Various 9 | Continuous | Various 9 | 2048 | Yes | Privato et al. (2023) ; Scurto and Postel (2023) ; Shepardson and Magnusson (2023) ; Wilson et al. (2023) ; Hein and Yip (2024) ; Nabi et al. (2024) ; Shaheed and Wang (2024) ; Shepardson et al. (2024) ; Stefani et al. (2024) ; Symons (2024) ; Privato et al. (2024) ; Vigliensoni and Fiebrink (2024) ; Visi (2024) ; Zheng et al. (2024) ; Bacon et al. (2025) ; Caspe et al. (2025) ; Khallaghi et al. (2025) ; Liu et al. (2025) ; Meyer et al. (2025) ; Mitcheltree et al. (2025) | |
| Music2Latent 2 | 2024 | 64 | Continuous | Variational | 4096 | Yes | Nistal et al. (2024) ; Tokui and Baker (2025) | |
| Stable Audio Open 3 | 2024 | 64 | Continuous | Variational | 2048 | Yes | Azimi and Zareei (2025) ; Collins (2025) ; Swift (2025) | |
| FlowDec 4 | 2025 | 128 | Discrete | RVQ Kumar et al. (2023) | 1764 | No | None | |
| 1 Realtime Audio Variational autoEncoder (RAVE) Caillon and Esling (2021) , Max/MSP version available at https://github.com/acids-ircam/RAVE | ||||||||
| 2 Music2Latent (M2L) Pasini et al. (2024) is a consistency autoencoder. \anon []Our Max/MSP implementation: https://github.com/jasper-zheng/music2latent-scripted | ||||||||
| Type of work | Autoencoder | Audio collection | Key aspect explored by the work | |
|---|---|---|---|---|
| Project 1 | A data sonification patch | RAVE | Samples of vintage music | Interactive sound space adaptation by biofeedback |
| Project 2 | An audio-visual environment | Stable Audio Open | Archival and field recordings in Hutongs | Crafting a sound space for an virtual gallery |
| Project 3 | A live performance patch | RAVE | Samples of organ instrument recordings | Autonomous micro-variations in latent space navigation |
| Project 4 | A soundscape composition | Stable Audio Open | Various (drum loops, vocals, and piano) | Neural synthesis as a resampling tool for composition |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Change Logs | Exporting trained mapping model from PyTorch, loading it in MaxMSP | ||||
| Creating mapping visualisation in MaxMSP | |||||
| Tracking the stylus position as input | |||||
| Logging stylus behaviours | |||||
| Encapsulating nn for simplicity | |||||
| Mile-stones | Time | Event | Type | Input Devices | Autoencoder |
| Oct 2024 | Soundwalking Workshop | Interactive demo | Stylus and tablet interface | RAVE | |
Autoencoders
Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.
See all 166 papers