Agentic Music Post-Production

Jaywon Koo; Junghyun Koo; Marco A. Martinez-Ramirez; Wei-Hsiang Liao; Seungheon Doh; Qiyu Wu; Vicente Ordonez; Yuki Mitsufuji

cs.SDOct 21, 2026

Agentic Music Post-Production

Authors: Jaywon Koo, Junghyun Koo, Marco A. Martinez-Ramirez, Wei-Hsiang Liao, Seungheon Doh, Qiyu Wu, Vicente Ordonez, Yuki Mitsufuji

Organizations: Rice University · Sony Corporate Technology Center America, Inc. · Sony Group Corporation

Abstract

We introduce MixAgent, an agentic pipeline for music post-production that follows the workflow of audio engineers to build interpretable and editable mixing plans guided by a reference song. A grouping agent first routes related input tracks, such as drum recordings, into subgroups. A planning agent then uses retrieved examples to choose and order audio effects, such as volume control and reverb, for tracks, subgroups, and the full mix. To refine the mix, a detector identifies what needs correction, and a critic proposes numerical parameter adjustments from tracks to the full mix. In our experiments, prompting recent multimodal models to listen and revise often worsens the mix, making reliable correction a central challenge. We therefore build MixCritic, a dataset of 45,657 repair critiques derived from parameters fitted to professional mixes, and train the critic with supervised fine-tuning followed by preference optimization on rendered audio outcomes. On MedleyDB, learned refinement reduces error by 19% relative to the initial mix, with preference optimization reducing it by 8.4% over supervised training alone. Retrieved examples further improve effects selection, and a human listening test shows that MixAgent gets higher quality ratings than mixes produced by Qwen3-Omni, Gemini 3.8 Flash and GPT-4o-audio.

Figures & tables

Figure 1: Comparison of music post-production workflows. (a) Neural mixing systems predict final mixes or audio effects settings without explicit critique–revision loops. (b) Prior LLM-based approaches focus on audio effects chain prediction and iterative mix refinement. (c) MixAgent introduces an agentic framework that integrates hierarchical planning, execution, and iterative refinement into a unified music post-production pipeline.

Figure 2: Overview of MixAgent. (a) The grouping agent assigns tracks to subgroups. (b) Using retrieved examples, the planner selects audio effects chains sequentially at the track, subgroup, and full-mix levels. (c) After parameter initialization, the detector flags elements for improvement, and the critic provides perceptual, tool-specific, or numerical feedback. Refinement proceeds from tracks to subgroups to the full mix.

Figure 3: Data generation for MixCritic. (a) From a fixed tool plan, we obtain parameters through gradient-based optimization against the professional mix with GRAFX (green) and LLM generation (pink). Replacing the optimized parameters of randomly selected tracks with LLM outputs creates corrupted mixing states. (b) The teacher LLM compares the selected track’s corrupted and optimized parameters to generate A1–A3 critiques. The critic agent is trained to repair one corrupted track’s parameters at a time.

Form	Critic output (abridged)
A1	The vocal sounds heavily squashed and lifeless… It needs to breathe naturally with smooth dynamics, a cleaner noise floor, and a touch of high-frequency air.
A2	{"tools": [{"tool": "noisegate", "comment": Raise the threshold significantly so it only engages on actual noise, and reduce the wet mix to make the processing transparent and eliminate pumping artifacts. } ]}
A3	{"tool": "noisegate", "parameters": {"log_threshold": 1.28, "log_ratio": 1.04, "log_knee": 2.45, "z_alpha_pre": 4.35}, "drywet": 0.599} …

Table 1: Critic output examples. A1 provides high-level semantic feedback, A2 names tools and revision directions, and A3 provides explicit numerical parameter values.

Qwen3-Omni
	MedleyDB					AlbumDB
	Reconst.	Quality		Similarity		Reconst.	Quality		Similarity
Method	MSS $\downarrow$	TuneJury $\uparrow$	AAPQ $\uparrow$	FxEnc++ $\uparrow$	AF $\downarrow$	MSS $\downarrow$	TuneJury $\uparrow$	AAPQ $\uparrow$	FxEnc++ $\uparrow$	AF $\downarrow$
Random	7.04	-2.18	5.96	0.51	8.7e4	7.90	-2.26	5.84	0.46	401.6
Single-pass	3.73	-0.55	6.38	0.65	0.36	4.34	0.44	6.44	0.78	0.14
Self-Refine	4.57	-1.22	5.78	0.63	2.86	4.50	0.17	6.09	0.76	0.21

Table 2: Results on MedleyDB and AlbumDB. Bold and underline mark the best and second-best reported scores across all methods for each metric and dataset. denotes off-the-shelf LLM baselines without task-specific training; marks MixAgent with a trained detector and critic. Single-pass uses no audio-feedback revision.

	Final output			Applied revisions
Configuration	MRS $\downarrow$	TuneJury $\uparrow$	FxEnc++ $\uparrow$	Improved (%) $\uparrow$	Mean $\Delta$ MRS $\uparrow$	Count
No refinement (planner only)	2.17	-0.72	0.71	—	—	—
Oracle feedback and parameter fitting (GT access)
Oracle A1	1.87	-0.41	0.78	53.6	+0.0331	250
Oracle A2	1.78	-0.44	0.80	60.9	+0.0374	243
Oracle A3	0.90	0.51	0.93	—	—	—

Table 3: Critic training and feedback form on MedleyDB. Improved is the percentage of revisions that lower full-mix MRS, Mean

\Delta

MRS is the average reduction per revision, and Count is the number of corrections applied across all clips. denotes prompted Qwen3-Omni critic without fine-tuning and marks trained detector and critic (trained critics use Qwen3.5-4B). Gray oracle entries use GT.

Figure 4: Human evaluation of mixing quality.

Examples	MRS $\downarrow$	FxEnc++ $\uparrow$	Tools/track
No examples	2.13	0.70	1.55
Random (3)	1.71	0.75	1.74
Text (3)	1.10	0.94	3.75
Text (3) + audio (1)	1.08	0.94	3.77
GRAFX	0.89	0.98	4.38

Table 4: Audio effects chain planning on 100 randomly selected MedleyDB clips. Parentheses denote number of retrieved in-context examples.

Retrieval and learned refinement
Configuration	MRS $\downarrow$	MSS $\downarrow$	TuneJury $\uparrow$	FxEnc++ $\uparrow$
MixAgent (full)	1.75	2.83	0.21	0.73
w/o learned refinement	2.17 $+$ 0.42	3.70 $+$ 0.87	$-0.72$ $-$ 0.93	0.71 $-$ 0.02
w/o retrieval	2.39 $+$ 0.64	3.65 $+$ 0.82	0.28 $+$ 0.07	0.70 $-$ 0.03
w/o both	2.81 $+$ 1.06	4.52 $+$ 1.69	$-0.82$ $-$ 1.03	0.65 $-$ 0.08
Refinement order

Table 5: Component ablations on MedleyDB.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Audio effect	Parameters (safe range)	Levels
Gain / panning	left and right gain ( $-7.8$ to $+2.2$ dB each), with panning from unequal channel gains	S, B, M
Parametric EQ	10 bands (low shelf, 8 peaking, high shelf) with frequency (0.1–10.4 kHz), gain ( $-6.1$ to $+4.3$ dB), and $Q$ (0.45–3.3)	S, B, M
Compressor	threshold ( $-30$ to $-12$ dB), ratio (1.3–3.3:1), makeup ( $\pm 3$ dB), attack, release, expander threshold and ratio	S, B, M
Multiband compressor	3 bands (crossovers 41–279 Hz and 2.8–11.2 kHz), per-band threshold ( $-57$ to $-3$ dB), ratio (1.3–17.7:1), attack, release	M
Noise gate	threshold ( $\approx-41$ to $-28$ dB), ratio, knee, smoothing	S, B, M
Distortion	drive (0.2–0.7 dB)	S, B, M

Appendix

Table 6: Audio effects library. S, B, and M denote track, subgroup, and full mix. Ranges are the safe ranges enforced after every edit, converted to physical units. The multiband compressor and limiter start bypassed.

Setting	Value
Critic backbone	Qwen3.5-4B + frozen RLAT
Audio adapter	32 queries, 32 tokens per audio input
Trainable parameters (adapter + LoRA)	89M + 99M
SFT learning rate (stage 1 / 2)	$10^{-4}$ / $5\cdot 10^{-5}$
SFT effective batch size (stage 1 / 2)	32 / 16
SFT stage 1 length	$\approx$ 14 epochs

Appendix

Table 7: Training and inference hyperparameters.

Statistic	Value
Source clips / songs	165 / 106
Supervised critique examples (train, A1–A3)	45,657
Validation examples	2,200
Feedback forms	A1, A2, A3
Preference pairs (DPO)	47,857
Detector training states	40,264

Appendix

Table 8: MixCritic statistics. Example counts are totals across the three feedback forms.

	MedleyDB					AlbumDB
	Reconst.	Quality		Similarity		Reconst.	Quality		Similarity
Method	MSS $\downarrow$	TuneJury $\uparrow$	Audiobox PQ $\uparrow$	FxEnc++ $\uparrow$	AF $\downarrow$	MSS $\downarrow$	TuneJury $\uparrow$	Audiobox PQ $\uparrow$	FxEnc++ $\uparrow$	AF $\downarrow$
Diff-MST	4.31	-0.70	6.49	0.72	0.19	4.72	1.02	7.15	0.73	0.11
MST ∗	3.08	-0.18	7.04	0.79	0.08	3.60	1.50	7.70	0.83	0.04
StemFX ∗	3.31	0.28	–	0.65	1.08	3.55	1.35	–	0.79	0.21
MixAgent	2.83	0.21	7.03	0.73	0.20	3.81	0.70	7.41	0.79	0.11

Appendix

Table 9: Comparison with dedicated mixing models. Lower is better for MSS and AF; higher is better for TuneJury, Audiobox PQ, and FxEnc++. Audiobox PQ is reference-free and computed at

-12

LUFS. ∗ Gray rows start from the professional mix separated into four sources and thus have privileged access to GT.

	Audio effects chain F1 $\uparrow$			GT-assisted parameter fitting				Plan size
Retrieved examples	Track	Subgroup	Full mix	MRS $\downarrow$	FxEnc++ $\uparrow$	TuneJury $\uparrow$	PQ $\uparrow$	Tools/track
No examples	0.33	0.38	0.49	2.13	0.70	-0.12	7.03	1.55
Random (3)	0.37	0.42	0.69	1.71	0.75	0.09	7.05	1.74
Text (1) + audio (1)	0.54	0.56	0.61	1.12	0.94	0.25	7.28	3.65
Text (3)	0.56	0.59	0.79	1.10	0.94	0.28	7.29	3.75
Text (3) + audio (1)	0.57	0.59	0.83	1.08	0.94	0.29	7.26	3.77

Appendix

Table 10: Audio effects chain planning on 100 MedleyDB clips. Audio effects chain F1 measures agreement with the pseudo-GT plan of the professional mix. Audio scores are computed after fitting each plan’s parameters to GT, so they measure what the chosen audio effects can achieve rather than agent performance. PQ is the reference-free Audiobox-Aesthetics production-quality score; TuneJury and PQ are computed at

-12

LUFS. GRAFX uses the pseudo-GT plan itself (F1

=1

by construction) and serves as an upper bound. MixAgent uses Text (3) + audio (1).

Panel of Figure 5 — Figure 5: Mixing consoles for MedleyDB clip sample_129. Each input is labeled with its instrument and index, and input $k$ is the same track in both panels. Both consoles use the same three subgroups: double bass, drums, and piano. GRAFX is shown after pruning near-inactive processors. MixAgent shows every tool in its final plan.

Panel of Figure 6 — Figure 6: Mixing consoles for AlbumDB clip sample_010. Both consoles use the same three subgroups: bass, drums (kick, overheads, snares, toms), and electric guitars. Layout as in Figure 5 .

Panel of Figure 7 — Figure 7: Mixing consoles for MedleyDB clip sample_110. The GRAFX console has no subgroups: all eight tracks feed the full mix directly. MixAgent groups the four guitars and the two vocals into subgroups and gives drums and bass their own subgroups. Layout as in Figure 5 .

Panel of Figure 8 — Figure 8: Mixing consoles for AlbumDB clip sample_013. The GRAFX console groups the two acoustic guitars and the two violins, and sends bass, cello, viola, and lead vocal to the full mix directly. MixAgent places the two violins, cello, viola, and bass in one subgroup, the two acoustic guitars on another, and the lead vocal in its own subgroup. Layout as in Figure 5 .

Target	$N$	Prec.	Recall	F1	AUROC
Track	9,779	0.797	0.822	0.809	0.893
Audio effect	33,329	0.700	0.849	0.767	0.874

Appendix

Table 11: Detector validation at threshold 0.50.

N

counts decisions.

Critic	Track	Subgroup	Full mix	Final
Untrained A1	1.50	1.82	2.12	2.26
SFT A1	1.46	1.73	1.99	2.08
DPO A1	1.47	1.81	1.96	2.09
Untrained A2	1.71	2.13	2.34	2.44
SFT A2	1.53	1.97	2.19	2.24
DPO A2	1.60	1.88	1.89	1.98

Appendix

Table 12: MRS after each processing stage (lower is better). Track, subgroup, and full mix targets are the corresponding GT-fitted outputs. The final target is the real mix. Untrained rows use prompted Qwen3-Omni as the critic.

Figure 9: MRS (lower is better) with a maximum of one, three, or five refinement passes per stage for A1, A2, and A3, from left to right. Parentheses show the score change from the preceding plotted pass budget.

Audio Editing

Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.

See all 53 papers

RIME: Enabling Large-Scale Agentic Music Post-Production
Jul 21, 2026Noah Schaffer, Nikhil SinghAudio Editing Multimodal Large Language Models
Dartmouth College
MixJudge: Learning a Language-Queryable Critic for Vocal Mixing
Oct 22, 2026Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh +1Audio-Language Models Audio Editing
Independent Researcher · Music Informatics Group, Georgia Institute of Technology · Music and AI Lab, National Taiwan University +1
Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects
Aug 5, 2026Yisu Zong, Jinjie Shi, Joshua ReissControllable Music Generation Audio Diffusion Models
Centre for Digital Music, Queen Mary University of London, UK

Jul 21, 2026J/K move · Enter open · S save

RIME: Enabling Large-Scale Agentic Music Post-Production

Noah Schaffer, Nikhil Singh

Dartmouth College

Audio Editing Multimodal Large Language Models Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully formed from a musician's mind. Despite the promise of music generation models for one-shot output, such fine-grained iterative refinement workflows are a complementary problem and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as agentic post-production, wherein individual aspects of a track are targeted, refined, and combined into a final version. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We observe that there is a language for modifying recorded music that is dense, consistent, and learnable. To leverage this, we introduce the Rule-based Instructions for Music Editing (RIME) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use POEMS and RIME to generate 15,000 pairs of edit instructions and ground-truth audio, then use this data to evaluate existing multimodal LLMs as agents on this task, revealing persistent limitations in current models. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning on synthetic data. We see RIME as a step towards iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering.

Abstract

Audio Editing

Explore similar work

RIME: Enabling Large-Scale Agentic Music Post-Production

MixJudge: Learning a Language-Queryable Critic for Vocal Mixing

Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects