Fusing Memory and Attention: A study on LSTM, Transformer and Hybrid Architectures for Symbolic Music Generation
Organizations: Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, India · IMC University of Applied Sciences Krems, Krems, Austria
Abstract
Machine learning techniques, such as Transformers and Long Short-Term Memory (LSTM) networks, play a crucial role in Symbolic Music Generation (SMG). Existing literature indicates a difference between LSTMs and Transformers regarding their ability to model local melodic continuity versus maintaining global structural coherence. However, their specific properties within the context of SMG have not been systematically studied. This paper addresses this gap by providing a fine-grained comparative analysis of LSTMs versus Transformers for SMG, examining local and global properties in detail using 17 musical quality metrics on the Deutschl dataset. We find that LSTM networks excel at capturing local patterns but fail to preserve long-range dependencies, while Transformers model global structure effectively but tend to produce irregular phrasing. Based on this analysis and leveraging their respective strengths, we propose a Hybrid architecture combining a Transformer Encoder with an LSTM Decoder and evaluate it against both baselines. We evaluated 1,000 generated melodies from each of the three architectures on the Deutschl dataset. The results show that the hybrid method achieves better local and global continuity and coherence compared to the baselines. Our work highlights the key characteristics of these models and demonstrates how their properties can be leveraged to design superior models. We also supported the experiments with ablation studies and human perceptual evaluations, which statistically support the findings and provide robust validation for this work.
Figures & tables
| Model | Architecture | Local Structure Capability | Global Structure Capability | Key Limitations | Dataset (s) |
| Music Transformer ( Huang et al., 2018 ) | Transformer with relative positional self-attention | Maintains local timing grid through relative attention | Strong: Generates minute-long compositions with coherent long-term structure; captures motifs and ABA form | Quadratic complexity for long sequences; limited to 6 years of training data | JSB Chorales, Piano-e-Competition |
| Performance RNN ( Simon and Oore, 2017 ) | LSTM-based | Strong: Generates plausible short-term structure; maintains local continuity | Weak: Loses long-term structure; forgets primer quickly; drifts to unrelated material | Poor long-range dependency modeling; cannot extend user primers coherently | Piano-e-Competition |
| CNN-LSTM for Raga ( Singh and Arora, 2024 ) | CNN + LSTM with tonic normalization | Strong: Extracts local features; F1-score 0.89 with tonic normalization | Limited evaluation on global coherence | Struggles with closely related Ragas; explainability restricted to 12 classes | Hindustani Classical Music dataset |
| Melody Diffusion ( Li and Sung, 2023 ) | Transformer-based diffusion model with BERT-like encoder | Moderate: Hits@1 = 72.41% | Strong: Attention captures long-distance dependencies | Monophonic only; quadratic attention limits long sequences; no polyphony | Chord-conditioned melody generation dataset |
| Multi-Instrument Synthesis ( Hawthorne et al., 2022 ) | Encoder-decoder Transformers + GAN + DDPM | Challenges in segment transitions for smaller models without context encoding | Strong: Flexible multi-instrument generation with context modeling | Audio artifacts, loudness issues; GAN-based inversion degrades quality; computationally expensive | MAESTROv3, Slakh2100, MusicNet |
| VQ-VAE + Discrete Diffusion ( Zhang et al., 2024 ) | Vector Quantized VAE with discrete diffusion models | Moderate: VQ-VAE compresses symbolic music into discrete codebook indexes | Strong: Diffusion models trained on discrete latent space achieve 72.36% accuracy in composer style generation | Limited to symbolic music; lacks real-time controllability; computational overhead of two-stage generation process | Classical composer datasets |
| Metric (with Theoretical Foundation) | Description | Range |
| Pitch Variance (Melodic contour analysis ( Morris, 1993 ) ) | Measures statistical variability between adjacent pitches, quantifying melodic smoothness versus angularity | 0 (no variance) to high |
| Pitch Range (Registral analysis theory ( Morris, 1993 ) ) | Distance between highest and lowest pitches, representing the melodic ambitus in traditional music theory | 0 to maximum interval |
| Rhythmic Variance (Rhythmic complexity theory ( London, 2012 ) ) | Statistical variation in note duration values, reflecting temporal complexity | 0 (uniform) to high |
| Note Density (Temporal density analysis ( London, 2012 ) ) | Number of note onsets per temporal unit, corresponding to surface rhythm density | 0 to high |
| Rest Ratio (Phrase structure analysis ( Caplin, 1998 ) ) | Proportion of silence relative to sounding events, relating to phrase structure theory | 0 (no rests) to 1 (only rests) |
| Interval Variability (Species counterpoint theory ( Fux and Edmunds, 1965 ) ) | Average fluctuation in melodic intervals, measuring step-wise versus leap-wise motion | 0 (constant) to high |
| Metric (with Theoretical Foundation) | Description | Range |
| Pitch Entropy (Music expectation theory ( Huron, 2008 ) ) | Shannon entropy of pitch class distribution, quantifying pitch unpredictability | 0 (predictable) to |
| Rhythmic Entropy (Music expectation theory ( Huron, 2008 ) ) | Shannon entropy applied to rhythm patterns, measuring temporal unpredictability | 0 (predictable) to |
| Motif Diversity Index (Motivic analysis theory ( Schoenberg, 2006 ) ) | Variety of recurring melodic patterns, based on motivic segmentation analysis | 0 (single motif) to high |
| Harmonic Complexity (Tonal pitch space theory ( Lerdahl, 2001 ) ) | Richness of implied harmonic structures derived from tonal pitch space theory | 0 (simple) to high |
| Contour Variability (Contour analysis theory ( Morris, 1993 ) ) | Statistical variance in melodic shape patterns extending contour analysis | 0 (similar) to high |
| Tonal Drift (Key analysis theory ( Krumhansl, 2001 ) ) | Shift in tonal center over time, measuring key stability using key-finding principles | 0 (stable) to high |
| Metric | LSTM | Transformer | Hybrid |
| Pitch Variance | 3.57 | 3.63 | 7.11 |
| Pitch Range | 4.24 | 4.33 | 5.21 |
| Rhythmic Variance | 0.12 | 0.10 | 0.25 |
| Note Density | 1.08 | 1.14 | 0.92 |
| Rest Ratio | 0.07 | 0.09 | 0.06 |
| Interval Variability | 1.69 | 1.69 | 3.94 |
| Metric | LSTM | Transformer | Hybrid |
| Pitch Entropy | 2.53 | 2.43 | 2.70 |
| Rhythmic Entropy | 1.33 | 1.20 | 1.94 |
| Motif Diversity Index | 0.76 | 0.74 | 0.86 |
| Harmonic Complexity | 0.52 | 0.51 | 0.90 |
| Contour Variability | 23.61 | 24.16 | 45.51 |
| Tonal Drift | 4.30 | 3.58 | 6.41 |
| Metric | LSTM | Transformer | Hybrid |
| Musicality | 6.97 | 6.92 | 7.03 |
| Coherence | 6.81 | 6.92 | 6.98 |
| Originality | 6.62 | 6.81 | 7.14 |
| Naturalness | 7.02 | 6.70 | 7.14 |
| Overall Quality | 6.83 | 6.94 | 7.17 |
| Dimension | ICC (Type 2) | F(8,160) | p-value | Agreement Level |
| Musicality | 0.072 | 4.228 | <0.001 | Moderate |
| Coherence | 0.066 | 3.671 | <0.001 | Moderate |
| Originality | 0.073 | 4.286 | <0.001 | Moderate |
| Naturalness | 0.062 | 3.738 | <0.001 | Moderate |
| Overall Quality | 0.081 | 4.963 | <0.001 | Good |