People differ in how they perceive music, what associations they make with it, what they like about it, and how they respond to it. Computational models intended to simulate people in environments that include music should account for these differences. Although LLMs have been shown to reproduce aspects of human behavior, whether audio-language models (ALMs) can reproduce a specific person's responses to textual and musical stimuli remains underexplored. We introduce MusicTwins, a framework that combines persona summaries, personal memory, and post-training to generate person-specific responses. We construct persona representations from human writings, capturing background, views, personality, writing style, and a musical profile, and post-train Audio Flamingo Next through supervised fine-tuning and reinforcement learning. Evaluation on ten held-out personas shows notable improvements for Audio Flamingo Next in persona fidelity, including personality, writing style, and responses to music, supporting MusicTwins as a foundation for training audio-language models to listen like their persona.
Figures & tables
Figure 2: Two illustrative applications for MusicTwins: teaching and music production assistants. The system prompt contains the persona summary, and memory contains specific experiences related to music. The shared model is given a musical and a textual stimulus, and learns to reproduce person-specific responses.
Table 1: Persona fidelity and music understanding for Audio Flamingo Next before (Base) and after MusicTwins training. Scores are multiplied by 100; 1–5 judge ratings are first rescaled to 0–1, giving 25(score−1) . Δ is MusicTwins minus Base in score points, computed before rounding; the final column gives its paired 95% bootstrap confidence interval. Bold marks the higher point estimate. Models use personas and memory unless noted: UC, unconditioned; ‡, no memory; †, random persona per question and ten random textual memories. n counts questions; personality uses 44 items for each of ten personas. Average weights the five evaluated persona components equally. Metrics are summarized in Section 5.2 ; the bootstrap procedure is detailed in Appendix E.7 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Training objectives by whether the question asks about a fact and where the evidence for the answer lies. Here, facts include documented preferences and other personal responses.