cs.SDAug 10, 2026

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

Authors: Shuyu LiKejun ZhangJiahe LeiShulei JiZihao WangJiaxing YuWanying WuLei Wang

Organizations: College of Artificial Intelligence, Zhejiang University · 3Innovation Center of Yangtze River Delta, Zhejiang University · 4The Chinese University of Hong Kong · College of Computer Science and Technology, Zhejiang University · 5Shandong University · 6Chu Kochen Honors College, Zhejiang University · 7Ant Group

Abstract

Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.

Explore similar work

CardsList
  1. Text2Score: Generating Sheet Music From Textual Prompts

    May 13, 2026Keshav Bhandari, Sungkyun Chang, Abhinaba Roy +4Music GenerationSheet Music