Audio Generation

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
May 29, 2026Zhaoqing Li, Haoning Xu, Jingran Su +9Text-to-Audio Generation TTS Synthesis
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · City University of Hong Kong +3
Sound effects in media:A comparative analysis of recorded and synthetic samples in live-action and animation
May 29, 2026Nelly Garcia, Joshua ReissSound Effects Generation Audio Generation
Centre for Digital Music (C4DM),Queen Mary University of London, 327 Mile End Rd, Bethnal Green, London E1 4NS
Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer
May 29, 2026Ke Lei, Yu Zhang, Changhao Pan +4Text-to-Audio Generation Audio-Video Generation
1Zhejiang University, China · 2ByteDance
3DAE: Binaural Quality Assessment for Audio Novel View Synthesis with Spatial Maps and Benchmark
May 28, 2026Jialu Xu, Yifan ZhouSpatial Audio Audio Generation
University of Waterloo
Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts
May 27, 2026Yuyue Wang, Xihua Wang, Xin Cheng +2Text-to-Audio Generation Audio Generation
Renmin University of China
LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation
May 27, 2026Zhisheng Zhang, Xiang Li, Yixuan Zhou +3Audio Token Compression Audio Representation Learning
1Shenzhen International Graduate School, Tsinghua University, China · *Work conducted while interning at ModelBest. · 2ModelBest Inc., China
Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
May 27, 2026Jiahao Mei, Heinrich Dinkel, Yadong Niu +7Text-to-Audio Generation Audio Representation Learning
1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China
An investigation of AI integration in sound designer workflows and experiences
May 26, 2026Nelly Garcia, Joshua ReissAudio Generation Human-AI Collaboration
1Queen Mary University of London
Generative AI and Copyright Infringement: A Legal-Technical Analysis of AI Music Generation Systems Under 17 U.S.C. Title 17
May 20, 2026Zuhaib Hussain ButtText-to-Music Generation Music Generation
Gujranwala, Punjab, Pakistan
WavFlow: Audio Generation in Waveform Space
May 18, 2026Feiyan Zhou, Luyuan Wang, Shoufa Chen +6Text-to-Audio Generation Audio-Video Generation
Meta AI · Northeastern University
Voice "Cloning" is Style Transfer
May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech Generation Trust in AI
Cornell University · TogetherAI · Stanford University
Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis
May 14, 2026Shuyang Cui, Zhi Zhong, Qiyu Wu +9Controllable Music Generation Music Generation
Sony Group Corporation · Sony AI
Seconds-Aligned PCA-DAC Latent Diffusion for Symbolic-to-Audio Drum Rendering
May 13, 2026Konstantinos Soiledis, Maximos Kaliakatsos Papakostas, Dimos Makris +1Latent Diffusion Models Symbolic Music Generation
1Dept. of Music Technology and Acoustics, Hellenic Mediterranean University, Rethymno & Athens, Greece · 2Athena RC, Athens, Greece
AuDirector: A Self-Reflective Closed-Loop Framework for Immersive Audio Storytelling
May 12, 2026Yiming Ren, Xuenan Xu, Ziyang Zhang +3TTS Synthesis Human-in-the-Loop AI
Shanghai Artificial Intelligence Laboratory · Tsinghua University
Drum Synthesis from Expressive Drum Grids via Neural Audio Codecs
May 11, 2026Konstantinos Soiledis, Maximos Kaliakatsos-Papakostas, Dimos Makris +1Neural Audio Codecs Music Generation
Dept. of Music Technology and Acoustics, Hellenic Mediterranean University, Rethymno & Athens, Greece · Athena RC, Athens, Greece
Communicating Sound Through Natural Language
May 9, 2026Emanuele Rossi, Emanuele RodolàAudio-Language Models Audio Generation
Sapienza University of Rome · Sapienza University of Rome / Paradigma
Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation
May 9, 2026Shihao Cheng, Jiaxu Zhang, Quanyue Song +6Audio-Video Generation Audio-Visual Synchronization
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China · ByteDance, China · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China +2
VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing
May 7, 2026Jiacheng Xu, Heting Gao, Liufei Xie +8Singing Voice Synthesis Spoken Dialogue Systems
QinYu Team & VITA-Team
MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model
May 5, 2026Jingyao GongSpeech Foundation Models Unified Multimodal Models
Independent Researcher
JaiTTS: A Thai Voice Cloning Model
Apr 30, 2026Jullajak Karnjanaekarin, Pontakorn Trakuekul, Narongkorn Panitsrisit +5TTS Synthesis Audio Generation
Jasmine Technology Solution · Sirindhorn International Institute of Technology · Department of Linguistics, Chulalongkorn University
V.O.I.C.E (Voice, Ownership, Identity, Control, Expression): Risk Taxonomy of Synthetic Voice Generation From Empirical Data
Apr 25, 2026Tanusree Sharma, Anish Krishnagiri, Lili Dudas +2TTS Synthesis Audio Generation
Penn State University · University of Dhaka · Arizona State University
UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
Apr 24, 2026Chunyu Qiang, Xiaopeng Wang, Kang Yin +11Text-to-Music Generation Text-to-Audio Generation
1Tianjin University, Tianjin, China · 2Kling Team, Kuaishou Technology, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Materialistic RIR: Material Conditioned Realistic RIR Generation
Apr 22, 2026Mahnoor Fatima Saad, Sagnik Majumder, Kristen Grauman +1Generative Modeling Conditional Generative Modeling
University of Utah · UT Austin
NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Apr 17, 2026Liumeng Xue, Weizhen Bian, Jiahao Pan +9Speech Generation Evaluation Speech Quality Assessment
Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +4
Geo2Sound: A Scalable Geo-Aligned Framework for Soundscape Generation from Satellite Imagery
Apr 16, 2026Kunlin Wu, Yanning Wang, Haofeng Tan +6Cross-Modal Alignment Audio Generation
The Hong Kong University of Science and Technology (Guangzhou), China · University of South Carolina, USA · University of Canterbury, New Zealand +2
Acoustic and perceptual differences between standard and accented speech and their voice clones
Apr 2, 2026Tianle Yang, Chengzhe Sun, Phil Rose +1TTS Synthesis Audio Generation
Department of Linguistics, University at Buffalo, United States · Department of Computer Science and Engineering, University at Buffalo, United States · Emeritus Faculty, Australian National University, Australia
YingMusic-Singer: Controllable Singing Voice Synthesis with Flexible Lyric Manipulation and Annotation-free Melody Guidance
Mar 25, 2026Chunbo Hao, Junjie Zheng, Guobin Ma +6Singing Voice Synthesis Controllable Music Generation
AI Lab, Giant Network, China
Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Audio-Language Models
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research
RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models
Jan 31, 2026Ruinan Jin, Xinting Liao, Hanlin Yu +2Adversarial Robustness Audio Generation
The University of British Columbia · Vector Institute
SemanticAudio: Audio Generation and Editing in Semantic Space
Jan 29, 2026Zheqi Dai, Guangyan Zhang, Haolin He +5Text-to-Audio Generation Audio Editing
The Chinese University of Hong Kong · LIGHTSPEED · Shanghai Jiao Tong University +1

May 29, 2026, eess.ASJ/K move · Enter open · S save

UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion

Zhaoqing Li, Haoning Xu, Jingran Su, Yaofang Liu+8

The Chinese University of Hong Kong · The Hong Kong Polytechnic University · City University of Hong Kong+3

Text-to-Audio Generation TTS Synthesis Latent Diffusion Models Audio Editing Unified Multimodal Models Speech Editing+1

Open paper AskPDF arXiv

We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed temporal composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-matched semantic conditioning that improves instruction following over single-layer baselines; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M--732M trainable parameters, UNISON achieves results competitive with or exceeding task-specialist models across evaluated domains, while being roughly

4\times

smaller than comparable unified systems.

Momentum

Related topics

Latest papers 95