Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrence

Yungang Yi; Weihua Li; Matthew Kuo; Catherine Shi; Quan Bai

cs.SDFeb 23, 2026

Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrence

Authors: Yungang Yi, Weihua Li, Matthew Kuo, Catherine Shi, Quan Bai

Organizations: Auckland University of Technology, Auckland, New Zealand · University of Tasmania, Tasmania, Australia

Abstract

For computational efficiency, modern language models are typically trained on independently sampled fixed-length sequences. Symbolic music language models largely inherit this paradigm, despite musical structure naturally unfolding over complete compositions rather than isolated excerpts. Fragmenting compositions into independent training instances therefore prevents continuous conditioning over the complete work. We present a practical framework for whole-piece training of symbolic music language models via Full-Horizon Compressed Recurrence (FHCR). FHCR preserves the full temporal horizon of recurrent memory while reducing the dimensionality of its key-value (KV) representation, making continuous whole-piece training practical under limited GPU memory. To directly assess functional long-range dependence, we introduce KV-Reset Context Utilization (KRCU), an evaluation-time diagnostic. On the MAESTRO symbolic piano dataset, KRCU shows that full-horizon models utilize context far beyond the local segment window, whereas reducing the temporal extent of recurrent memory substantially weakens this measurable long-range dependence. FHCR preserves long-range context utilization while substantially reducing recurrent memory cost. These findings show that preserving the temporal extent of recurrent history is important for efficient whole-piece modeling, and that memory cost can instead be reduced through KV representation compression. The project demos and generated music samples are available at https://wholemusic.github.io.

Figures & tables

Figure 1: Whole-piece training through streamed recurrent segments. A complete musical composition remains a single continuous training instance while computation proceeds sequentially over recurrent segments. Segmentation is introduced solely for efficient computation rather than defining independent optimization samples.

Figure 2: Full-Horizon Compressed Recurrence (FHCR). Temporal-memory pruning reduces recurrent cost by discarding older historical states, shortening the accessible temporal horizon. FHCR instead preserves the complete recurrent horizon while reducing the number of recurrent KV heads through grouped-query attention.

Figure 3: Temporal-memory configurations used in the experiments. Bar length indicates the recurrent history retained at each Transformer depth, with all horizons aligned to the current position. The full-horizon reference retains the complete available history at every layer, whereas selective retention, progressive allocation, and two-scale allocation reduce the temporal extent or depth-wise availability of recurrent memory.

Item	Value
Transformer layers $L$	18
Hidden size $d$	1024
Query heads $H$	16
Head dimension $d_{\mathrm{head}}$	64
FFN size $d_{\mathrm{ff}}$	4096
Segment length $s$	1024

Table 1: Key model and recurrence configurations. FHCR reduces the number of KV heads while preserving the full recurrent horizon.

Figure 4: Long-range context utilization in the full-horizon MHA reference. Increase in negative log-likelihood when the accessible recurrent history is restricted to the most recent 16k, 8k, 4k, or 1k tokens. Targets are grouped by their absolute positions within the composition. Positive

\Delta

NLL indicates that removed earlier history contributed to prediction.

Figure 5: Comparison of modeling quality, long-range context utilization, and memory cost across recurrent-memory configurations. Left: best validation perplexity (lower is better). Middle: KRCU-AUC following the 8k KV reset (higher indicates a larger and more persistent predictive effect of pre-reset recurrent information). Right: peak GPU memory during training (lower is better). FHCR closely matches the full-horizon MHA reference in perplexity, exhibits a strong KRCU response, and substantially reduces peak GPU memory, whereas temporal-memory-pruning configurations generally show much weaker long-range context utilization.

Setting	Tokens/s	Peak GPU Mem (GB)	Time to best checkpoint (h)	Best Val PPL
Full-horizon MHA reference	7,618	15.5	6.06	5.98
Perceiver-AR-like temporal pruning	11,753 (+54.3%)	6.3 (-59.1%)	3.92 (-35.3%)	6.54 (+9.4%)
Selective retention (sliding-window)	11,061 (+45.2%)	7.8 (-49.6%)	4.17 (-31.1%)	6.51 (+9.0%)
Selective retention (uniform)	11,257 (+47.8%)	7.8 (-49.3%)	4.10 (-32.3%)	6.52 (+9.1%)
Selective retention (random)	11,160 (+46.5%)	8.0 (-48.0%)	4.13 (-31.9%)	6.53 (+9.3%)
Progressive ascending	10,540 (+38.4%)	7.0 (-54.5%)	4.38 (-27.7%)	6.15 (+2.9%)

Table 2: Efficiency metrics for the full-horizon MHA reference, temporal-memory-pruning baselines, and FHCR on MAESTRO. Percentages in parentheses are relative to the full-horizon MHA reference.

Figure 6: KV-Reset Context Utilization (KRCU) across recurrent-memory configurations. Recurrent KV states are reset at the 8k-token boundary (red dashed line). The vertical axis shows the increase in NLL relative to uninterrupted recurrent inference. Persistent positive

\Delta

NLL indicates that information accumulated before the reset boundary continued to contribute to later predictions.

Model	KRCU-AUC
Full-horizon MHA	0.024804
Perceiver-AR-like temporal pruning	0.000657
Selective retention (sliding-window)	0.001356
Selective retention (uniform)	0.001652
Selective retention (random)	0.001436
Progressive ascending	0.002376

Table 3: KRCU-AUC after recurrent KV reset at the 8k-token boundary. Higher values indicate a larger and more persistent prediction penalty following removal of pre-reset recurrent information. All models are evaluated over the same 22 post-reset windows.

Long-Context Language Modeling

Long-context language modeling studies how language models can use, retain, and efficiently process information across very long sequences.

See all 189 papers

Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music Generation
Apr 7, 2026Boyu Cao, Lekai Qian, Dehan Li +3Symbolic Music Generation Music Generation
South China University of Technology, School of Future Technology
How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models
Oct 2, 2026Amir FathiLong-Context Modeling Sequence Modeling
Department of Computing Science University of Alberta Edmonton, AB, Canada
Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation
Feb 28, 2026Jinhan Xu, Xing Tang, Houpeng Yang +7Symbolic Music Generation Diffusion Models
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China 430070 · School of Computer Science, Hubei University of Technology, Wuhan, China 430068 · School of Information Engineering, Hubei University of Economics, Wuhan, China 430205; Hubei Key Laboratory of Digital Finance Innovation, Wuhan, China 430205

Apr 7, 2026J/K move · Enter open · S save

Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music Generation

Boyu Cao, Lekai Qian, Dehan Li, Haoyu Gu+2

South China University of Technology, School of Future Technology

Symbolic Music Generation Music Generation Autoregressive Generation

Open paper AskPDF arXiv

Generating long sequences with structural coherence remains a fundamental challenge for autoregressive models across sequential generation tasks. In symbolic music generation, this challenge is particularly pronounced, as existing methods are constrained by the inherent severe error accumulation problem of autoregressive models, leading to poor performance in music quality and structural integrity. In this paper, we propose the Anchored Cyclic Generation (ACG) paradigm, which relies on anchor features from already identified music to guide subsequent generation during the autoregressive process, effectively mitigating error accumulation in autoregressive methods. Based on the ACG paradigm, we further propose the Hierarchical Anchored Cyclic Generation (Hi-ACG) framework, which employs a systematic global-to-local generation strategy and is highly compatible with our specifically designed piano token, an efficient musical representation. The experimental results demonstrate that compared to traditional autoregressive models, the ACG paradigm achieves reduces cosine distance by an average of 34.7% between predicted feature vectors and ground-truth semantic vectors. In long-sequence symbolic music generation tasks, the Hi-ACG framework significantly outperforms existing mainstream methods in both subjective and objective evaluations. Furthermore, the framework exhibits excellent task generalization capabilities, achieving superior performance in related tasks such as music completion.

Abstract

Long-Context Language Modeling

Explore similar work

Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music Generation

How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models

Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation