Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSP

Tornike Karchkhadze; Shlomo Dubnov

cs.SDApr 8, 2026

Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSP

Authors: Tornike Karchkhadze, Shlomo Dubnov

Organizations: University of California San Diego

Abstract

We present a framework for real-time human-AI musical co-performance, in which a latent diffusion model generates instrumental accompaniment in response to a live stream of context audio. The system combines a MAX/MSP front-end-handling real-time audio input, buffering, and playback-with a Python inference server running the generative model, communicating via OSC/UDP messages. This allows musicians to perform in MAX/MSP - a well-established, real-time capable environment - while interacting with a large-scale Python-based generative model, overcoming the fundamental disconnect between real-time music tools and state-of-the-art AI models. We formulate accompaniment generation as a sliding-window look-ahead protocol, training the model to predict future audio from partial context, where system latency is a critical constraint. To reduce latency, we apply consistency distillation to our diffusion model, achieving a 5.4x reduction in sampling time, with both models achieving real-time operation. Evaluated on musical coherence, beat alignment, and audio quality, both models achieve strong performance in the Retrospective regime and degrade gracefully as look-ahead increases. These results demonstrate the feasibility of diffusion-based real-time accompaniment and expose the fundamental trade-off between model latency, look-ahead depth, and generation quality that any such system must navigate.

Figures & tables

Figure 1: Real-time human–AI co-performance system architecture. A musician performs live while MAX/MSP captures the audio stream and communicates with a remote GPU server via OSC/UDP. The server runs a diffusion-based generative model that produces a complementary accompaniment (e.g., bass for a drummer), which is returned and mixed with the live performance in real time.

Figure 2: Sliding-window protocol for continuous/streaming accompaniment generation. At each step, the window advances by $T\cdot r$ , where $r$ controls the step size, and the model predicts a new audio segment conditioned on the available context. Green regions indicate audio that is given (either observed from the performer or generated in previous steps), while red regions denote the segment to be generated. The horizontal axis represents time, with $t=0$ marking the current playback position. We illustrate three regimes corresponding to look-ahead depth $w\in\{-1,0,1\}$ : (left, $w<0$ ) Retrospective prediction on already-observed content; (middle, $w=0$ ) Immediate prediction without look-ahead, insufficient for uninterrupted playback under non-negligible inference latency; (right, $w>0$ ) explicit Look-ahead, where audio is generated ahead of playback time, enabling continuous real-time streaming.

Figure 3: Latent diffusion model for accompaniment generation. The input mixture is encoded into a latent representation by encoder $E$ , then concatenated with a noise sample and a stem-selection one-hot vector. A SongUNet-based diffusion model iteratively denoises the target stem latent, conditioned on the mixture context. Decoder $D$ reconstructs the waveform. For streaming operation, the model is trained and inferred in inpainting mode, where the portion of the target latent corresponding to already-generated content is kept fixed while only the future segment is denoised.

Figure 4: Consistency distillation. A pretrained diffusion model (teacher) generates denoised targets by applying $k$ ODE steps along the diffusion trajectory. A student model is trained to directly predict these targets from noisy inputs, enforcing consistency across noise levels. An EMA of the student parameters provides stable training targets. The training objective combines the CD loss with a DSM loss. Latent context conditioning and instrument label inputs are omitted from the figure for clarity.

Figure 5: The RTAP MAX/MSP patch. The main panel controls streaming parameters ( $r$ , $w$ , $T$ ), instrument selection (bass, drums, guitar, piano), and playback ( Play / Stop/Reset ). Fade crossfades between chunks to suppress boundary clicks; Package size sets OSC chunk size. The Internal / Live toggle selects between buffer playback and live microphone input; Internal mode adds a Source file selector (inactive here). Waveform displays and output meters provide real-time monitoring. The lower-left panel configures the server: inference mode (Diffusion or CD), credentials (host, ports, conda environment, project path), and CUDA device. In the screenshot, the patch runs in remote Live mode: microphone inputs provide context for drums, guitar, and piano; the first buffer shows the predicted bass stem; the highlighted window is the current step — live-recorded for the other instruments and bass-predicted on the previous step — with the next prediction already in flight.

Figure 6: Comparison of our diffusion model and consistency distillation (CD) model against paper baselines across three evaluation metrics. The x-axis represents the net look-ahead $T\cdot r\cdot w$ (seconds), partitioned into three zones: Retrospective ( $w{=}{-1}$ , green), Immediate ( $w{=}0$ , gray), and Look-ahead ( $w{=}1$ , salmon). Points in the Immediate zone are spread by step size $r$ .

	Config (1)		Config (2)		Config (3)		Config (4)
Client	Win 10, SD		Win 10, SD		Mac M2, local		Win 10, SD
Server	Win 10, RTX 2070, local		Linux, RTX A6000, Paris		Mac M2, MPS, local		Mac M2, MPS, SD, remote
Model	Diff.	CD	Diff.	CD	Diff.	CD	Diff.	CD
MAX/MSP $\to$ server	17	17	188	188	20	20	107	107
CAE encode	40	40	52	52	55	55	55	56
Sampling ( $N$ fwd. passes)	1175 (10)	130 (2)	480 (10)	88 (2)	1072 (10)	146 (2)	1072 (10)	146 (2)

Table 1: Mean per-stage timings (ms) at

r{=}0.25

across four deployment configurations. OSC chunk size is 4,410 samples, giving 15 chunks per send. RT: real-time constraint

d<T{\cdot}r=1500

ms satisfied (

\checkmark

) or not (

\times

Model	$r$	$T{\cdot}r$	MAX $\to$ server	enc. $+$ sample	dec.	server $\to$ MAX	Total	RT
Diff.	0.250	1500	188	532	72	189	981	$\checkmark$
	0.125	750	145	532	67	145	889	$\times$
	0.0625	375	145	532	64	145	886	$\times$
	$1/64^{\dagger}$	94	145	532	64	145	886	$\times$
CD	0.250	1500	188	140	72	189	589	$\checkmark$
	0.125	750	145	140	67	145	497	$\checkmark$

Table 2: Full-cycle inference timings (ms) for both models across step ratios

r

, measured on a dedicated NVIDIA RTX A6000 GPU with no competing processes. MAX

\to

server: OSC transfer of a

T{\cdot}r

audio chunk from MAX/MSP to the Python server; enc.

+

sample: CAE encoding plus iterative sampling (10 forward passes for Diffusion, 2 for CD); dec.: CAE decoding to waveform; server

\to

MAX: OSC return of the generated

T{\cdot}r

audio chunk back to MAX/MSP. RT indicates whether the real-time constraint

d<T{\cdot}r

is satisfied (

\checkmark

) or not (

\times

). † Theoretical minimum step (

r_{\min}{=}1/64

, one latent time step).

Music Accompaniment Generation

See all 12 papers

Streaming Generation for Music Accompaniment
Oct 25, 2025Yusong Wu, Mason Wang, Heidi Lei +5Music Generation Conditional Generative Modeling
Mila, Quebec Artificial Intelligence Institute, Universit´e de Montr´eal · MIT - Massachusetts Institute of Technology · Canada CIFAR AI Chair
Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation
Jun 23, 2026Baisen Wang, Chenxi Bao, Qisong HanText-to-Music Generation Controllable Music Generation
Work done in ZhuoLab. · DynamiX · Cardiff University
LiveBand: Live Accompaniment Generation in the Audio Domain
Jun 2, 2026Marco Pasini, Javier Nistal, Ben Hayes +3Music Generation Autoregressive Generation
Sony Computer Science Laboratories, Paris, France · Johannes Kepler University, Linz, Austria · Queen Mary University of London, UK

Oct 25, 2025J/K move · Enter open · S save

Streaming Generation for Music Accompaniment

Yusong Wu, Mason Wang, Heidi Lei, Stephen Brade+4

Mila, Quebec Artificial Intelligence Institute, Universit´e de Montr´eal · MIT - Massachusetts Institute of Technology · Canada CIFAR AI Chair

Music Generation Conditional Generative Modeling Audio Generation Music Accompaniment Generation

Open paper AskPDF arXiv

Music generation models can produce high-fidelity coherent accompaniment given complete audio input, but are limited to editing and loop-based workflows. We study real-time audio-to-audio accompaniment: as a model hears an input audio stream (e.g., a singer singing), it has to also simultaneously generate in real-time a coherent accompanying stream (e.g., a guitar accompaniment). In this work, we propose a model design considering inevitable system delays in practical deployment with two design variables: future visibility

t_f

, the offset between the output playback time and the latest input time used for conditioning, and output chunk duration

k

, the number of frames emitted per call. We train Transformer decoders across a grid of

(t_f,k)

and show two consistent trade-offs: increasing effective

t_f

improves coherence by reducing the recency gap, but requires faster inference to stay within the latency budget; increasing

k

improves throughput but results in degraded accompaniment due to a reduced update rate. Finally, we observe that naive maximum-likelihood streaming training is insufficient for coherent accompaniment where future context is not available, motivating advanced anticipatory and agentic objectives for live jamming.

Abstract

Music Accompaniment Generation

Explore similar work

Streaming Generation for Music Accompaniment

Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation

LiveBand: Live Accompaniment Generation in the Audio Domain