MultiVerse: A Creator-Centered Approach to Steering Context-Adaptive Lyrics
Authors: Alexander Wang, Chris Donahue, David Lindlbauer
Organizations: Human-Computer Interaction Institute Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Computer Science Department Carnegie Mellon University Pittsburgh, Pennsylvania, USA
Generative AI may enable new forms of context-aware creative expression by dynamically tailoring media content to its consumption context. For instance, AI systems could adapt song lyrics to the listener and their current activity. However, existing media adaptation systems primarily optimize for audience experience, often neglecting artists' intent, style, and preference. We address this challenge by introducing a novel creator-centered approach to adaptive media authoring and present MultiVerse, a system that instantiates this approach for steering adaptive lyrics. Our approach allows creators to explicitly author controls based on their intent, lyric structure, and audience context, and uses rule-based validations to ensure controls are followed. We conducted a study with 10 songwriters, comparing MultiVerse with a prompting-based workflow for composing adaptive lyrics. The comparison revealed that creators preferred to author how lyrics adapt by explicitly specifying relevant context and adaptation constraints, while recognizing trade-offs in flexibility and iteration speed. Interviews further revealed that creators viewed adaptive media as enabling new forms of audience connection, introducing a distinct creative process, favoring new compositional strategies, and reshaping notions of authorship.
Figures & tables
Figure 1. We anticipate the rise of AI-driven trends in music and multimedia like adaptive lyrics , where lyrics are dynamically tailored to different audiences at consumption time. Because final outputs are generated at consumption time, artists cannot directly review or edit each variation, running the risk of deviating from their creative intent . MultiVerse addresses this by providing tools that allow artists to iteratively author steering controls over intent, content structure, and audience context, while previewing how lyrics may adapt across audiences with simulated personas. Once the controls are refined, they can be deployed to deliver adaptive lyrics to real listeners at scale.
Figure 2. Adapt for persona Anna with no constraints.
Figure 3. Artist specifies content structure.
Figure 4. Artist specifies creator intent.
Figure 5. Artist specifies relevant audience context.
Figure 6. Artist preview output for other personas.
Figure 7. C3 unifies creator, content, and context to support creator-driven adaptive media.
Figure 8. Creator intent authoring features in MultiVerse. Global anchors affect all generations, local anchors affect one line, and word locks keep specific words unchanged.
Figure 9. Content structure authoring features in MultiVerse. Accent ensures the labeled syllable is stressed, Rhyme ensures selected words rhyme, and Cluster prevents multisyllabic words from crossing labeled boundaries.
Figure 10. Persona-based authoring in MultiVerse. Creators query listener context, bind responses to lyric variables, and preview personalized lyric adaptations.
Figure 11. Lyric generation pipeline: Creator-authored constraints are passed as modular prompts to the LLM to generate an initial set of candidates. Each candidate is then validated using rule-based tests to ensure all constraints are satisfied. All verified candidates that pass the tests are then ranked by the LLM based on persona preference to select the final output.
ID
FT
SW
Musical Practice
Genres
P1
10
5
EDM producer; pianist, drummer, and DJ; experience with AI-assisted songwriting
Progressive/Future House, EDM, Pop/Punk
P2
16
10
Graduate screen scoring student; composer for indie games and films; released album; active songwriter across genres
Electronic Pop, Alt Pop, Indie
P3
1
4
Vocal training; hobby songwriter and producer
Pop, R&B
P4
14
5
Violinist with orchestral and jazz background; writes instrumentals and lyric adaptations
Pop
P5
0
5
Collegiate a cappella performer; co-wrote and produced albums for collaborators
Bedroom Jazz Fusion
P6
4
16
Primary songwriter and lyricist for two bands; producer, engineer, and performer
Indie Rock, Synth-pop, Electronic, Experimental
Table 1. Participant musical backgrounds. FT = years of formal musical training; SW = years of songwriting experience.
Question
Mean MultiVerse
Mean Prompt
I would be happy to use this system or tool on a regular basis.
4.5 (SD = 1.3)
4.1 (SD = 2.1)
I enjoyed using the system or tool.
5.0 (SD = 1.3)
5.0 (SD = 1.6)
It was easy for me to explore many different ideas, options, designs, or outcomes using this system or tool.
5.2 (SD = 1.6)
4.6 (SD = 1.8)
The system or tool was helpful in allowing me to track different ideas, outcomes, or possibilities.
4.8 (SD = 1.5)
3.6 (SD = 1.8)
I was able to be very creative while doing the activity inside this system or tool.
4.5 (SD = 1.6)
4.8 (SD = 1.7)
The system or tool allowed me to be very expressive.
4.2 (SD = 1.8)
3.9 (SD = 1.6)
Table 2. Survey questions and mean ratings for MultiVerse and prompting.
Figure 12. Interface for prompting-based workflow.
Controllable text generation steers language models toward desired attributes, formats, audiences, and constraints while preserving useful, natural text.
We propose CoLyricist, an AI-assisted lyric writing tool designed to support the typical workflows of experienced lyricists and enhance their creative efficiency. While lyricists have unique processes, many follow common stages. Tools that fail to accommodate these stages challenge integration into creative practices. Existing research and tools lack sufficient understanding of these songwriting stages and their associated challenges, resulting in ineffective designs. Through a formative study involving semi-structured interviews with 10 experienced lyricists, we identified four key stages: Theme Setting, Ideation, Drafting Lyrics, and Melody Fitting. CoLyricist addresses these needs by incorporating tailored AI-driven support for each stage, optimizing the lyric writing process to be more seamless and efficient. To examine whether this workflow-aligned design also benefits those without prior experience, we conducted a user study with 16 participants, including both experienced and novice lyricists. Results showed that CoLyricist enhances the songwriting experience across skill levels. Novice users especially appreciated the Melody-Fitting feature, while experienced users valued the Ideation support.
Masahiro Yoshida, Bingxuan Li, Songyan Zhao +4
University of California, Los Angeles Los Angeles, California, USA
This paper examines a use of AI in creative practice as an interpretive sounding board for human-generated material, rather than the more familiar pattern of AI generation followed by human curation. Through the lens of resonance as theorized by Hartmut Rosa, I present a first-person case study of songwriting from July 2025 to March 2026, drawing on 16 original pieces in English, French, and other languages along with piano solos. I describe a configuration in which resonance is not located between user and model, but in the author's deepening contact with their own material, mediated through the model. This kind of resonance was supported rather than inhibited by AI when sounding-board behavior was cultivated through sustained calibration by the user. Two failure modes appeared when calibration was absent: sycophantic drift and magical overinterpretation. This account suggests both the potential and the risks of AI as an interpretive partner in creative practice.
Xiao Xiao
Institute for Future Technologies De Vinci Research Center, De Vinci Higher Education Paris, France · MIT Media Lab Cambridge, MA, USA
Recent advances in AI music generation have enabled users to create complete musical pieces from natural-language prompts. However, most existing systems follow a prompt-and-regenerate paradigm, making iterative refinement difficult because users must repeatedly recreate compositions instead of directly evolving existing musical ideas. We present MusiChat, a conversational vibe composing system that enables collaborative human-AI music creation through natural-language interaction and iterative refinement. At the core of MusiChat is a hierarchical controllable music generation framework that separates lyric-aligned musical structure generation from expressive surface realization, allowing flexible stylistic transformations and structure-preserving edits. The system integrates a large language model with a hybrid symbolic music engine through a memory-augmented architecture that maintains the active composition state and user history across interactions. A hybrid intent-routing mechanism further enables efficient interpretation of both precise musical edits and open-ended creative requests. Rather than regenerating compositions from scratch, MusiChat incrementally transforms an evolving musical artifact while preserving relevant musical structure and user intent. We evaluate MusiChat through objective analysis and human studies, achieving 95.31% and 100% accuracy for single- and multi-turn interactions, respectively, and obtaining like-to-dislike ratios of 2:1 for melody naturalness and 3:1 for musical quality. Our results demonstrate that MusiChat supports coherent multi-turn music authoring and interactive human-AI co-creation through a conversational interface.
Callie C. Liao, Duoduo Liao, Ellie L. Zhang
George Mason University Fairfax, VA · IntelliSky McLean, VA