cs.SDFeb 24, 2026
SaveMIDI-Informed Singing Accompaniment Generation in a Compositional Song Pipeline
Organizations: National Taiwan University · Taiwan AI Labs · University of Rochester
Abstract
While end-to-end lyrics-to-song models offer convenience for casual users, professional songwriters require score-to-song systems that allow them to retain authorship over the core melody. However, existing score-to-song methods are limited to short-form snippets and fail to maintain coherence in long-form generation, particularly during vocal-silent sections like intros and bridges. To address this long-form bottleneck, we propose MIDI-informed singing accompaniment generation (MIDI-SAG). Unlike conventional audio-only models, MIDI-SAG utilizes symbolic timing and chord information derived from the vocal MIDI to provide a stable musical roadmap. By incorporating structure planning, which defines temporal boundaries and semantic labels, our framework facilitates consistent generation across both vocal and non-vocal sections. We demonstrate the feasibility of this compositional pipeline by leveraging specialized pre-trained modules, enabling data-efficient training on a single GPU. Our experiments show the potential of this approach for both professional score-to-song and general lyrics-to-song tasks. While an early exploration, MIDI-SAG suggests a promising direction for structured, long-form music synthesis. Audio demos are available, and the code will be open-sourced at https://composerflow.github.io/web_revealed/.
Figures & tables
Figure 1. Architectural comparisons of score-to-song generation systems: (a) naïve chaining of SVS and SAG as done in ( Hong et al., 2024 ) , which adapts a text-to-music (TTM) model for SAG; (b) the proposed extension (cf. Section 3.2 ) that uses score-related time-varying conditions extracted from the vocal audio for SAG (named MIDI-SAG); (c) the proposed extension (cf. 3.2 , 3.3 ) that gets the conditions instead from the input MIDI score; (d) the proposed extension (cf. 3.4 ) that considers full-song MIDI and structure planning.
Figure 2. Illustration of rhythmic stability. White stripes represent the predicted beat positions from a generated accompaniment. While (a) audio-SAG loses rhythmic consistency between the non-vocal (first-half), vocal (second-half) segments, (b) MIDI-SAG yields stable beat and coherent content.
Figure 3. Overview of the implemented compositional song generation pipeline. It performs score-to-song generation when the melody score is provided by a user (i.e., without the lyrics-to-melody module), and performs lyrics-to-song generation when the melody is generated by the lyrics-to-melody module. For lyrics-to-song, the system sequentially maps lyrics to a full song through: (1) melody composition (CSL-L2M ( Chai and Wang, 2025 ) ), (2) SVS (FastSpeech ( Ren et al., 2020 ) ), (3) melody harmonization (AccoMontage2 ( Yi et al., 2022 ) ), and (4) the proposed MIDI-SAG, which adapts MuseControlLite ( Tsai et al., 2025 ) to incorporate symbolic and acoustic conditions.
| Condition type | Training | Inference |
| beat | All-in-One ( Kim and Nam, 2023 ) | From generated vocal MIDI |
| pitch | RMVPE ( Wei et al., 2023 ) | RMVPE ( Wei et al., 2023 ) |
| chord | Chord detection ( Park et al., 2019 ) | Accomontage2 ( Yi et al., 2022 ) |
| structure | All-in-One ( Kim and Nam, 2023 ) | User-provided or by LLM |
| text prompt | Audio Flamingo 3 ( Goel et al., 2025 ) | User-provided or by LLM |
| reference audio | Mask one structure; others as reference | From previous song-structure segment |
Table 1. Conditioning signals used during training and inference times for the proposed MIDI-SAG module.
| SAG Method | duration | Rhythm F1 | Key Acc. | CLAP | Coherence | Musicality | Memorability | Clarity | Naturalness |
| TTM with no vocal input | 47s | 0.2061 | 0.0650 | 0.3221 | 2.9056 | 2.8372 | 2.6813 | 2.7236 | 2.6762 |
| Audio-SAG (Figure 1 (a)) (simulating Melodist ( Hong et al., 2024 ) ) | 47s | 0.2216 | 0.1850 | 0.3064 | 2.9052 | 2.8348 | 2.6605 | 2.6613 | 2.6195 |
| MIDI-SAG 47s detected MIDI+Beat (Figure 1 (b)) | 47s | 0.6342 | 0.5550 | 0.2453 | 3.3032 | 3.1942 | 3.0425 | 3.0998 | 2.9353 |
| MIDI-SAG 47s (Figure 1 (c)) | 47s | 0.9534 | 0.7700 | 0.2387 | 3.5473 | 3.4055 | 3.2921 | 3.3306 | 3.1434 |
| MIDI-SAG 90—120s (Figure 1 (d)) | 90–120s | 0.9134 | 0.8200 | 0.2820 | 3.6480 | 3.3957 | 3.4355 | 3.4341 | 3.2500 |
Table 2. Result of Experiment 1, comparing different SAG settings for score-to-song generation, using the same SVS module. The scores are the higher the better for all metrics. Coherence, Musicality, Memorability, Clarity, Naturalness are evaluated using SongEval ( Yao et al., 2025 ) . The vocal audio starts with two silent bars as the intro.
| AnyAccomp | FastSAG | Our MIDI-SAG w/o ground truth vocal MIDI | |
| Training data (hrs) | 8k | 3k | 2.5k |
| Base generation length | 10s | 10s | 47s |
| Longer generation | |||
| Text control capability | |||
| APA ( Grachten and Nistal, 2025 ) | 0.457 | 0.000 | 0.595 |
Table 3. Result of Experiment 2, comparing our MIDI-SAG and existing SAG baselines. For fairness, we use only vocal audio input (i.e., without ground-truth vocal MIDI) here.
| Model | Total Parameters | Training Data (hrs) | Audiobox | SongEval | |||||||||
| CLAP | PER | CE | CU | PC | PQ | Coherence | Musicality | Memorability | Clarity | Naturalness | |||
| Suno v4.5 | — | — | 0.417 | 0.290 | 7.339 | 7.766 | 5.333 | 8.036 | 4.198 | 4.011 | 4.174 | 4.034 | 3.939 |
| ACE-Step | 3.5B | 100K | 0.368 | 0.238 | 7.209 | 7.642 | 5.820 | 7.948 | 3.449 | 3.214 | 3.203 | 3.216 | 3.162 |
| DiffRhythm | 1.3B | 60K | 0.323 | 0.325 | 7.530 | 7.791 | 6.336 | 8.189 | 3.740 | 3.419 | 3.595 | 3.512 | 3.354 |
| LeVo | 2B | 110K | 0.229 | 0.617 | 7.565 | 7.674 | 4.993 | 8.295 | 3.392 | 3.272 | 3.198 | 3.265 | 3.155 |
| Ours | 1.3B | 2.5K | 0.282 | 0.213 | 7.590 | 7.712 | 6.294 | 8.240 | 3.648 | 3.396 | 3.436 | 3.434 | 3.250 |
Table 4. Objective evaluation results of Experiment 3 for lyrics-to-song generation. / indicates the higher/lower the better.
| Model | Overall Preference | Lyrics Adherence | Musicality | Voice Naturalness | Structure Clarity |
|---|---|---|---|---|---|
| Suno v4.5 | 4.297 0.194 | 4.375 0.188 | 4.344 0.201 | 4.188 0.195 | 4.297 0.184 |
| ACE-Step | 2.953 0.249 | 3.656 0.244 | 3.172 0.271 | 2.672 0.280 | 3.266 0.252 |
| DiffRhythm | 2.469 0.231 | 2.922 0.192 | 2.859 0.235 | 2.703 0.238 | 2.688 0.239 |
| LeVo | 2.297 0.275 | 2.188 0.239 | 2.641 0.256 | 2.484 0.262 | 2.531 0.235 |
| Ours | 2.656 0.287 | 3.672 0.243 | 2.703 0.242 | 3.109 0.270 | 2.516 0.262 |
Table 5. Subjective evaluation results of Experiment 3 for lyrics-to-song generation (mean confidence interval); higher values indicate better performance. Bold denotes the highest value, and underlining indicates the second highest.
| Setting | Chord F1 | Key Acc | Rhythm F1 |
| w/ all conditions | 0.9006 | 0.79 | 0.8339 |
| w/o chord | 0.3908 | 0.21 | 0.7870 |
| w/o rhythm | 0.8914 | 0.79 | 0.4279 |
| w/o structure | 0.8957 | 0.79 | 0.8317 |
| w/o audio | 0.9027 | 0.84 | 0.8442 |
| w/o vocal pitch contour | 0.5930 | 0.69 | 0.4319 |
Table 6. Ablation study on conditions for MIDI-SAG for score-to-song generation.
| Model | Vocal MIDI generation | Score-to- song | Lyrics-to- song | Max length |
| Melodist ( Hong et al., 2024 ) | 10s | |||
| MelodyLM ( Li et al., 2024 ) | 12s | |||
| VersBand ( Zhang et al., 2025c ) | 12s | |||
| Ours | 90–120s | |||
| Diffrhythm ( Ning et al., 2025 ) | 285s | |||
| ACE-Step ( Gong et al., 2025 ) | 280s |
Table 7. Comparison of song generation systems. The length limit shown for our method is due to the limitation of CSL-L2M ( Chai and Wang, 2025 ) . In contrast, under singing accompaniment generation or score-to-song settings, our method is able to generate longer backing tracks.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4. The data-preprocessing pipeline to curate data for fine-tuning Stable Audio Open to implement our MIDI-informed singing accompaniment generation (MIDI-SAG) model.
Figure 5. Augmenting Stable Audio Open for singing accompaniment and audio continuation. The architecture utilizes the MuseControlLite framework to integrate multi-modal conditioning signals, enabling precise singing-accompaniment alignment and seamless long-form audio continuation.
| Module | Time (s) |
| Lyrics-to-melody | 10 |
| Melody harmonization | 0.2 |
| Singing voice synthesis | 3 |
| Singing accompaniment generation | 40 |
Table 8. Inference latency per module for a 120-second song.
| Method | FD | KL | CLAP | Smoothness |
| MuseControlLite | 111.58 | 0.2160 | 0.3622 | -0.2734 |
| MuseControlLite w/ unfrozen self-attention layers | 109.62 | 0.1794 | 0.3961 | -0.3529 |
Table 9. Ablation study demonstrating the effectiveness of unfreezing self-attention layers for audio continuation.
| Stage | Component | Base model / source | Upstream training (prior work) | Our additional training (this work) |
| 1 | Lyrics Vocal MIDI | CSL-L2M ( Chai and Wang, 2025 ) | 300 hours lyrics–melody pairs; trained with 1 V100 for 24 hours | None |
| 2 | SVS (MIDI vocals) | FastSpeech ( Ren et al., 2020 ) | None | 10 h licensed internal (2 singers); GPU: 1 3090 for 8 days; trained from scratch |
| 3 | Backing generator backbone | Stable Audio Open ( Evans et al., 2025 ) | 7.3k hours licensed data; 36224 A100 GPU hours (VAE + Diffusion Transformer) | None |
| 3b | MIDI-SAG conditioning module (adapter) | MuseControlLite | None | 2.5k h Mandarin pop; GPU: 1 3090 for 9 days |
| 4 | Harmony / chord extraction or harmonization | AccoMontage2 ( Yi et al., 2022 ) | None; template matching and dynamic programming | None |
| 5 | Beat / tempo tracking (for silent spans) | Heydari and Duan (2022) | None | 2.5k h Mandarin pop; GPU: 1 3090 for 12 hours |
Table 10. Component-level provenance and training effort. We explicitly separate upstream pretraining from incremental training performed in this paper.
Figure 6. The input lyrics are the same as those used by other end-to-end song generation models, but our method supports Mandarin only due to the constraint of CSL-L2M ( Chai and Wang, 2025 ) . The text control of our method could be either a single global style prompt or different local style prompts for different segments.