Fish Audio Drama 3 Technical Report

Dawei Han; Shidong Li; Shijia Liao; Jiahua Liu; Songting Liu; Qingzheng Wang; Yuxuan Wang; Xin Chen

cs.SDOct 13, 2026

Fish Audio Drama 3 Technical Report

Authors: Dawei Han, Shidong Li, Shijia Liao, Jiahua Liu, Songting Liu, Qingzheng Wang, Yuxuan Wang, Xin Chen,

Abstract

We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.

Figures & tables

Figure 1: Fish Audio Drama 3 turns text into directed performances. It casts voices from references, follows natural-language direction, revises a take when the next turn gives a note, stages multi-character scenes, edits existing takes, and synthesizes long-form audio, streaming speech as the text arrives.

Figure 2: Fish Audio Drama 3 architecture. The backbone reads the system prompt, the target text, and then its own audio frames, each entered as a fused embedding of its codes 0 and 1. The codebook-MTP decoder runs over every position on top of the backbone’s hidden state. Its four shared layers take that state, the predicted code 0, and the input frame’s codes 2–9, which bypass the backbone, and MTP Layers 1–9 then predict the remaining codes one at a time. Codes are predicted wherever the next token is an audio frame, every layer attends causally along time, and the audio tokenizer decoder turns the ten codes into a waveform.

Figure 3: Drama 3 training pipeline. A pre-trained Qwen3.5 LLM goes through three pre-training stages, then instruction SFT and preference RL.

Figure 4: Per-frame sequential path after the backbone step. (a) S2’s four-layer Fast AR runs ten passes, one that takes in the backbone’s state and one for each of the nine acoustic codes, and reads its weights on every pass. (b) Drama 3’s codebook-MTP decoder runs its four shared and nine per-codebook layers once. Boxes are layer passes drawn to scale; each chip is the code emitted by the layer above it.

Metric	Drama 3	S2
Layer passes in sequence after the backbone step	13	40
Weights read per frame (GB)	$\approx$ 6.0	$\approx$ 15.6
Backbone without routed experts	2.9	7.3
Eight active routed experts	1.0	–
Codebook-MTP decoder or Fast AR	2.1	8.3
Layers whose KV cache grows with the context (backbone and decoder)	23	36

Table 1: Decoding cost of Drama 3 and S2. Weights read per frame are computed from the tensor sizes of the checkpoints.

Change	Kernels or time	Gain	GPU, concurrency
Auxiliary decode kernels removed	1,429 $\to$ 1,278 kernels per step	+1.7%	B300, 32
Gated DeltaNet decode fused; sampler counter folded in	$-$ 100 kernels per step	+4.2%	B300, 1
Gated DeltaNet prefill launches merged	23 $\to$ 13 kernels per layer	+4.4%	B300, 128
First-frame sampling captured into CUDA graphs †	30.9 $\to$ 14.7 ms host time per round	+15.3%	H200, 32
Execution plan built from host-side lengths	9.75 $\to$ 0.86 ms GPU idle per round	$\approx$ +7%	B300, 128

Table 2: Kernel launches and waits removed below CUDA graphs and PDL. Gains are in the per-stream decode rate.

Figure 5: Shape-driven kernel optimization loop. An engineer writes the contract for a round; a coding agent captures the serving shapes, profiles, writes candidate kernels, and tests them, first on the operator interface and then inside the engine. A candidate that fails returns to profiling, and one that passes is deployed. WER: word error rate.

Figure 6: Schematic timelines of streaming on one GPU (a) and with prefill–decode disaggregation (b). On one GPU, every prefill pauses all running streams, and the long prefill of a cold request, one whose reference audio is not cached, causes a playback stall. With disaggregation, the prefill GPU runs the same prefills, each finished prefill writes its KV and recurrent state into the decode GPU, and the streams play without gaps.

Style	Pairs	$\boldsymbol{\Delta}$ SIM	$\boldsymbol{\Delta}$ WER (points)	SIM prefers default (%)	Reward model prefers styled (%)
Style	Pairs	$\boldsymbol{\Delta}$ SIM	$\boldsymbol{\Delta}$ WER (points)	SIM prefers default (%)	With direction	Without direction
Whisper	1,439	$-0.310$	$+0.24$	$99.7$	$62.2$	$10.6$
Laughing	1,423	$-0.193$	$+0.79$	$93.9$	$64.4$	$27.1$
Sad	1,432	$-0.178$	$+0.34$	$96.1$	$54.3$	$13.1$
Confused	1,372	$-0.107$	$-0.61$	$89.3$	$26.2$	$16.3$
Happy	1,436	$-0.063$	$-0.16$	$77.9$	$38.5$	$29.5$

Table 3: Styled versus default human readings in Expresso.

\Delta

SIM and

\Delta

WER are means over pairs, styled minus default,

\Delta

WER in percentage points. SIM is to one default-style recording per speaker. SIM prefers default is the share of pairs in which the default reading has the higher SIM. Reward model prefers styled is the share of pairs in which our reward model gives the styled reading the higher score; with direction, the style is written into the script as a direction, and without direction, the script is the sentence alone.

Model	test-zh CER (%) $\downarrow$		test-en WER (%) $\downarrow$		Flagged (%) $\downarrow$
Model	First take	Final	First take	Final	test-zh	test-en
SFT checkpoint	1.66	0.84	3.49	1.60	2.82	3.40
Fish Audio Drama 3	1.00	0.83	2.20	1.54	1.24	1.84

Table 4: SFT checkpoint and Drama 3 on Seed-TTS-Eval. First take is the output before the regeneration of Section 5.1 , Final the output after it, and Flagged the share of first takes that fail the signal checks. Bold : better model.

System	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$
Fish Audio Drama 3, one reference	1.85	0.605	4.31
Fish Audio Drama 3, three references	1.85	0.633	4.31
OmniVoice ( Zhu et al., 2026 )	1.30	0.729	4.28
Qwen3-TTS ( Hu et al., 2026 )	1.60	0.704	4.41
CosyVoice 3 ( Du et al., 2025 )	1.59	0.694	4.28

Table 5: Cloning from one or three references on LibriSpeech-PC test-clean. The other systems use one reference and are quoted from OmniVoice. Scored as in OmniVoice, with HuBERT-large ( Hsu et al., 2021 ) WER, UTMOS ( Saeki et al., 2022 ) , and SIM to the first reference. Bold : best per column.

Language	Metric	Drama 3	OmniVoice	FireRedTTS3-Base	Fish Audio S2	ElevenLabs v4	Inworld TTS-2	Soniox TTS RT v2
Arabic	CER	0.76	1.39	1.75	3.50	0.71	1.39	2.30
Cantonese	CER	22.70	17.71	40.32	30.67	19.74	24.14	–
Chinese	CER	0.63	1.01	0.91	0.73	0.66	0.97	1.32
Czech	WER	3.53	2.86	3.17	2.84	2.13	3.95	8.36
Dutch	WER	0.59	1.36	1.15	0.99	1.00	1.53	2.01
English	WER	1.13	1.56	2.12	1.62	1.56	1.56	1.94

Table 6: MiniMax Multilingual Testset, error rate (%). ElevenLabs v4 ( ElevenLabs, 2026 ) , Inworld TTS-2 ( Inworld AI, 2026 ) , and Soniox TTS RT v2 ( Soniox, 2026 ) are run by us through their public APIs on the same items. OmniVoice, FireRedTTS3-Base, and Fish Audio S2 are quoted from their papers. Bold : lowest per row.

English
System	Insertion			Deletion			Substitution
System	WER $\downarrow$	SIM $\uparrow$	MAE ${}_{\text{D}}$ $\downarrow$	WER $\downarrow$	SIM $\uparrow$	MAE ${}_{\text{D}}$ $\downarrow$	WER $\downarrow$	SIM $\uparrow$	MAE ${}_{\text{D}}$ $\downarrow$
Fish Audio Drama 3	3.03	0.905	0.208	5.79	0.838	0.174	2.08	0.866	0.181
VoiceCraft-X ( Zheng et al., 2025 )	5.74	0.85	0.166	9.71	0.80	0.172	7.29	0.81	0.155
LEMAS-Edit ( Zhao et al., 2026 )	6.04	0.81	0.158	9.29	0.80	0.167	6.09	0.80	0.167
ECPA ( Ren et al., 2026 )	4.50	0.82	–	6.91	0.77	–	4.13	0.78	–

Table 7: Ming-Freeform basic split under the CosyEdit protocol. WER in %. Baselines are quoted from CosyEdit2; – marks a metric not reported. Bold : best per column within each language.

System	Basic		Full
System	English	Chinese	English	Chinese
Fish Audio Drama 3	3.49	3.69	3.43	2.35
Cartesia Sonic-3, infill ( Cartesia, 2026 )	4.58	8.64	4.68	6.53

Table 8: Ming-Freeform WER (%) against Cartesia Sonic-3. Each cell averages deletion, insertion, and substitution, with the new audio pasted into the source recording. Cartesia is run by us through its public API. Bold : lowest per column.

Metric	System	Iter. 0	Iter. 1	Iter. 2	Iter. 3
Emotion $\uparrow$	Fish Audio Drama 3	55.5	57.2	57.5	58.2
Emotion $\uparrow$	Step-Audio-EditX ( Yan et al., 2025b )	53.6	66.8	68.7	69.3
Style $\uparrow$	Fish Audio Drama 3	76.5	79.4	80.5	81.8
Style $\uparrow$	Step-Audio-EditX ( Yan et al., 2025b )	45.6	62.7	64.2	63.8
Paralinguistic $\uparrow$	Fish Audio Drama 3	1.32	2.32	–	–
Paralinguistic $\uparrow$	Step-Audio-EditX ( Yan et al., 2025b )	1.91	2.89	–	–

Table 9: Step-Audio-Edit benchmark. Step-Audio-EditX is quoted from the Prompt-Fixed setting of its paper, with the judge scores reported there; Drama 3 is judged by our own run of the official Gemini-2.5-Pro protocol. Emotion and style are accuracies (%). Bold : better system.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Structure
Control	Syntax	Examples
Speaker token	<\|speaker:N\|> , $N=0,\dots,15$	<\|speaker:0\|> Ready? <\|speaker:1\|> Always.
Reference block	[Reference N] Text: <\|speaker:S\|> transcript Speech: {{audio}}	Figure 11
Line separator	<\|audio_separator\|>	The first line. <\|audio_separator\|> The second line.
Infill markers	<\|fim_suffix\|>{{after}} <\|fim_prefix\|>{{before}}	Figure 14
Streaming look-ahead	chunk<\|endofturn\|>next chunk	It was late<\|endofturn\|>and the hall was empty.

Appendix

Table 10: Drama 3 control syntax.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 267 papers

Fish Audio S2 Technical Report
Mar 9, 2026Shijia Liao, Yuxuan Wang, Songting Liu +11Multi-Speaker TTS TTS Synthesis
StepAudio 3 Gen Technical Report
Sep 11, 2026Bin Lin, Bo Zhao, Boyang Wang +68TTS Synthesis Zero-Shot TTS
Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis
Sep 21, 2026Chee-En Yu, Yi-Cheng Lin, Sung-Feng Huang +4TTS Synthesis Audio Reasoning
Graduate Institute of Electrical Engineering, National Taiwan University, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NVIDIA Research 4 Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan

Mar 9, 2026J/K move · Enter open · S save

Fish Audio S2 Technical Report

Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng+10

Multi-Speaker TTS TTS Synthesis Controllable Speech Generation Streaming TTS Synthesis

Open paper AskPDF arXiv

We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 ms.Our code and weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/s2-pro). We highly encourage readers to visit https://fish.audio to try custom voices.

Abstract

TTS Synthesis

Explore similar work

Fish Audio S2 Technical Report

StepAudio 3 Gen Technical Report

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis