Multimodal Co-Generation With Continuous Bitstream Diffusion

Georgios Batzolis; Georgios Kontos; Bruno Trentini; Pranam Chatterjee; Micha Livne; Mark Girolami; Luca Ambrogioni

cs.CLOct 9, 2026

Multimodal Co-Generation With Continuous Bitstream Diffusion

Authors: Georgios Batzolis, Georgios Kontos, Bruno Trentini, Pranam Chatterjee, Micha Livne, Mark Girolami, Luca Ambrogioni

Organizations: Radboud University

Abstract

We show that continuous bitstream diffusion (CoBit), originally developed for language modeling, provides a unified framework for joint and conditional multimodal generation using the same training and sampling protocol. Across image-text, speech-text and protein sequence-structure modeling, models are trained separately from scratch, but each uses a single input projection, transformer backbone, bitwise prediction head and binary score-matching objective shared across its modalities. Within each setting, a single checkpoint supports joint generation and the conditional directions included in training by changing which parts of the bitstream are observed. We first validate the approach on MNIST-Sum, a controlled image-text task pairing raw binary images with equations encoded as text, achieving near-perfect conditional accuracy and high joint consistency without learned tokenizers. We then apply the same protocol at larger scale to speech-text and protein sequence-structure modeling. Trained without text pretraining, a single speech-text model performs speech synthesis, recognition, continuation, and unconditional co-generation. Jointly generated speech and text achieve 3.5% word error rate between the generated text and automatic transcriptions of the generated speech. In protein modeling, a single CoBit checkpoint supports forward folding, inverse folding, and joint sequence-structure generation. For jointly generated proteins spanning 100-500 residues, computational refolding of CoBit's sequences yields structures that agree with the generated backbones at a mean self-consistency TM-score of 0.911. Together, these results show that CoBit provides a common generative framework for substantially different modalities and forms of cross-modal dependence, without introducing a domain-specific objective or sampling procedure.

Figures & tables

Method	Data	Generated state	Objective	Shared I/O	Directions
UniDiffuser	image–text	continuous latents	noise prediction	✗	joint, cond., marginal
Show-o	image–text	discrete tokens	AR + masked diffusion	✓	cond., mixed-modality
Spirit-LM	speech–text	discrete tokens	autoregressive	✓	set by prompting
DPLM-2	protein	discrete tokens	discrete diffusion	✓	joint, cond., marginal
CoBit (this work)	all three †	fixed binary codes	binary score matching	✓	joint, cond.

Table 1: Representative multimodal generators. Shared I/O: one input layer and one output head serve all modalities of a model. † One model per pair, one shared training and sampling protocol.

Figure 1: MNIST-Sum, a controlled image–text evaluation. (a) Four binarised MNIST digits and their equation form one flat sequence of 3,388 bits (3,136 pixel, 84 text and 168 marker bits); the zoom-ins show bits from the displayed pair. One denoiser serves both conditional directions by observing one part of the stream and denoising the other. (b) With nothing observed, the same weights co-generate an image and equation from noise; a pair is correct only when the generated equation matches the generated image. Rates use $n{=}4{,}096$ samples per direction; the displayed examples illustrate, but do not estimate, those rates.

Figure 2: Text-to-speech generation with CoBit-MLS. Given text and speaker tokens (extracted from a held-out example), the model is tasked with generating the speech part of the sequence. The similarity between the reference and generated waveforms is measured in terms of frame-level mel-cepstral distortion (MCD), aligned using dynamic time warping (DTW). CoBit generates the same text, but with different speech pacing and a longer pause between the two sentences. (Left) Temporal alignment of waveform frames. CoBit introduces silence between the two sentences. (Right) Pairwise MCD cost matrix and the optimal DTW path. The horizontal segment between 1.5 and 2 seconds corresponds to inter-sentence silence.

Direction	$n$	Steps (NFE)	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SpkSim $\uparrow$	FSD $\downarrow$	GenPPL $\downarrow$
CoBit-LibriTTS — one set of weights (134M, 800 speech tokens)
text $\rightarrow$ speech	2,294	512 (1024)	11.7	8.1	3.31	0.19	—	—
speech $\rightarrow$ text	2,294 / 2,813	512 (1024)	23.3 / 38	15.3 / 24.1	—	—	—	—
speech continuation	1,936	512 (1024)	—	—	2.82	0.21	2,373 / 40.5	152.1
joint	1,024	256 (256)	19.9	13.3	3.02	—	—	509.6 / 377.2
CoBit-MLS — one set of weights (633M, 500 speech tokens)

Table 2: All text-audio generative directions from a single set of weights per model. Each conditional row is quoted at the best step count for that direction, evaluated on LibriSpeech-PC. Generated audio is transcribed with Whisper-large. Per-direction guidance and churn are listed in Appendix C.1 ; co-generation uses

\gamma=0.175

. Text-to-speech and speech continuation evaluation is on test-clean; the two scores for speech-to-text refer to test-clean and test-other, respectively. For FSD, the two separate metrics refer to FSD-wlm and FSD-e2v, respectively. For co-generation, generative perplexity is reported for both the text (left) and the audio (right) streams.

Figure 3: Backbone quality from denoised bits: (a) scTM by length, (b) strict success by length, and (c) fold similarity versus coordinate error. Colours denote length. 100 backbones (20 per length) each receive one ProteinMPNN redesign and ESMFold refold. Intervals are 95%. Shading marks both strict criteria.

Figure 4: Selected CoBit backbones, rendered with PyMOL ( Schrödinger, LLC, 2026 ) . Highest scTM among 20 samples at each displayed length with scRMSD $\leq 2$ Å; scores use ProteinMPNN/ESMFold. pLDDT is refold confidence.

ID	Model	Scope	$n$	$D_{\rm RMSD}$	$D_{\rm TM}$
Our evaluation: joint bitstream generator
1	CoBit	G	100	36.0	75.0
Published evaluation: Lu et al. (2026)
Joint generators: discrete structural codes
2	DPLM-2 (650M)	P	100	31.0	48.0
3	ESM3 Open (sequence first)	P	100	17.0	19.0

Table 3: Backbone designability (%) with one ProteinMPNN redesign and ESMFold. Training data and evaluation settings differ across studies. Published PMPNN1 baselines: Lu et al. (2026) ; G: general bitstream framework; P: protein-specific generator.

Figure 5: Visual representation of the rates from Table 4.4 . CoBit captures how proteins fold overall, but is less accurate on fine structural detail.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

	Seen tuples	Held-out tuples	CI (seen)
Image $\rightarrow$ text, exact string match	97.56	97.85	$[97.04,97.99]$
Text $\rightarrow$ image, all four digits	99.95	99.93	$[99.82,99.99]$
Joint from noise, mutually consistent	99.46	—	$[99.19,99.65]$

Appendix

Table 4: MNIST-Sum task accuracy (%),

n{=}4{,}096

per direction and split. “Held-out” uses digit tuples absent from training. Joint generation is counted correct only when the generated equation is arithmetically valid and names the four digits in the generated image.

The task is evaluating the relevance and likelihood of the predicted text continuation, given the text prompt.

You should also consider whether the meaning of the text continuation is making sense.

The text prompt is: ” prompt ”, and the text continuation is: ” content ”

You must give an overall rating from 1 to 5. The rating guideline is as below:

1: The text continuation is very unlikely and irrelevant to the text prompt.

2: The text continuation is unlikely and marginally relevant to the text prompt.

3: The text continuation is moderately likely and relevant to the text prompt.

Appendix

Table 5: LLM-as-a-Judge prompt for thematic relevance evaluation of speech continuation. GPT-4o is used as the judge model.

Direction	Steps (NFE)	$\gamma$	gs
CoBit-LibriTTS — one set of weights (134M, 800 speech tokens)
text $\rightarrow$ speech	512 (1024)	0.23	6
speech $\rightarrow$ text	512 (1024)	0.37	5
speech continuation	512 (1024)	0.15	5
CoBit-MLS — one set of weights (633M, 500 speech tokens)
text $\rightarrow$ speech	512 (1024)	0.27	5

Appendix

Table 6: Sampler operating points for text-audio experiments. Steps, stochasticity and guidance settings were chosen based on validation performance on 500 samples from LibriTTS dev-clean.

	CoBit-LibriTTS			CoBit-MLS
$\gamma$	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$
Steps=256 (NFE=512); gs=5/gs=6
0.11	11.4 / 12.3	8.2 / 8.7	3.29 / 3.27	7.2 / 7.4	5.7 / 6.1	3.96 / 3.94
0.13	12.2 / 12.6	8.8 / 8.8	3.3 / 3.27	6.4 / 7.9	5 / 6.4	3.95 / 3.96
0.15	11.4 / 11.9	8.2 / 8.7	3.32 / 3.27	6.8 / 7	5.5 / 5.4	3.97 / 3.95
0.17	12.2/ 12	8.9 / 8.5	3.3 / 3.27	5.5 / 6.9	4.3 / 5.1	3.95 / 3.95

Appendix

Table 7: Validation results for text-to-speech transfer, computed on 500 samples from LibriTTS dev-clean. Scores with guidance scales 5 and 6 are reported for each

\gamma

-steps combination.

	CoBit-LibriTTS		CoBit-MLS
$\gamma$	WER $\downarrow$	CER $\downarrow$	WER $\downarrow$	CER $\downarrow$
Steps=256 (NFE=512); gs=5/gs=6
0.11	20.1 / 20.5	13.5 / 13.5	8.2 / 8.2	4.8 / 4.9
0.13	20.6 / 20.9	14 / 13.8	8.1 / 8.1	4.9 / 5
0.15	20.2 / 20.6	13.5 / 13.8	8.3 / 8.1	4.8 / 4.8
0.17	20.5 / 20.2	13.3 / 13.4	8.4 / 8.5	4.9 / 4.9

Appendix

Table 8: Validation results for speech-to-text transfer, computed on 500 samples from LibriTTS dev-clean. Scores with guidance scales 5 and 6 are reported for each

\gamma

-steps combination.

	CoBit-LibriTTS				CoBit-MLS
$\gamma$	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	GenPPL- Speech $\downarrow$	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	GenPPL- Speech $\downarrow$
Steps=256 (NFE=512); gs=5/gs=6
0.11	24.7 / 25.4	16.9 / 17.5	2.48 / 2.4	138.1 / 123.4	5.3 / 5.6	2.9 / 3.2	3.89 / 3.93	77.9 / 79.5
0.13	23.7 / 23.8	16 / 16.2	2.48 / 2.42	122.7 / 121.5	5.6 / 5.2	3.4 / 2.9	3.89 / 3.91	77.1 / 76
0.15	23.8 / 23.5	16.1 / 16.4	2.46 / 2.43	123.7 / 116.4	5.6 / 5.6	3.1 / 3.3	3.91 / 3.91	76 / 75.4
0.17	22.2 / 23.3	15 / 16	2.46 / 2.41	117.9 / 117.6	5.5 / 5.4	3 / 3.1	3.89 / 3.89	78.2 / 74

Appendix

Table 9: Validation results for speech continuation, computed on 500 samples from LibriTTS dev-clean. Scores with guidance scales 5 and 6 are reported for each

\gamma

-steps combination.

Model	Params	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	GenPPL-Text $\downarrow$	GenPPL-Speech $\downarrow$
CoBit (ours; from scratch, no text pretraining)
CoBit-LibriTTS	134M	19.9	13.3	3.02	509.6	377.2
CoBit-MLS	633M	3.5	1.8	3.77	252.4	133.3
Autoregressive System (SmolLM+Speaker $\rightarrow$ SparkTTS)
Cascade	640M	3.2	3.1	4.23	122	1,648

Appendix

Table 10: Joint generation performance, computed by generating

1,024

samples. Speech perplexity for the cascade system is high, as SmolLM produces multilingual sequences or web/code artifacts that are not transferable to speech. We use the same NFE as in Table 2 ; 256 for CoBit-LibriTTS and 512 for CoBit-MLS.

		Text-to-Speech				Speech-to-Text
Model	Params	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SpkSim $\uparrow$	WER $\downarrow$ Clean	WER $\downarrow$ Other
CoBit (ours; from scratch, no text pretraining)
CoBit-LibriTTS	134M	11.7	8.1	3.31	0.19	23.3	38
CoBit-MLS	633M	5.3	4.1	4	0.27	8.2	22
Autoregressive SLMs/Hybrids
GPA-v1.5 ( Cai et al., 2026 )	505M	3.1	3.1	4.37	0.4	3.4	6.3

Appendix

Table 11: Text-to-speech and Speech-to-text performance on LibriSpeech-PC. For text-to-speech, evaluation is conducted on the test-clean partition. For each speaker in the dataset, one sample was held out to extract speaker features. Speech-to-text evaluation is conducted on test-clean and test-other partitions separately. As in Table 2 , Steps

=

512 for both CoBit-LibriTTS and CoBit-MLS.

Model	Params	UTMOS $\uparrow$	FSD-wlm $\downarrow$	FSD-e2v $\downarrow$	SpkSim $\uparrow$	GenPPL- Speech $\downarrow$	GPT-4o $\uparrow$
CoBit (ours; from scratch, no text pretraining)
CoBit-LibriTTS	134M	2.82	2,373	40.5	0.21	152.1	1.11
CoBit-MLS	633M	3.89	2,386	13	0.34	83.6	1.44
Trained on MLS-English (same corpus as CoBit-MLS, excluding LibriTTS)
Flow-SLM-S ( Chou et al., 2025 )	395M	2.47	3,391	35.7	0.21	242.8	1.1
Flow-SLM-L	1.25B	3.48	3,476	35.4	0.41	131.7	1.57

Appendix

Table 12: Continuation-based evaluation for text-audio experiments against published SLMs. All figures are evaluated using publicly available checkpoints. GPT-4o refers to LLM-as-a-Judge score with the prompt shown in Table 5 . We use Steps

=

512 for both CoBit models. The last row reports the tokenizer performance ceiling.

Model	Sentiment	Speaker	Gender	Bg.- Domain	Bg.- Random	Room	Avg.
CoBit (ours; from scratch, no text pretraining)
CoBit-LibriTTS	86	88.5	88.5	66	71	71	78.5
CoBit-MLS	90.5	93	93.5	73.5	80	75.5	84.3
Trained on MLS-English (same corpus as CoBit-MLS, excluding LibriTTS)
Flow-SLM-S	85	87	92.5	77	86	69.5	82.83
Flow-SLM-L	92.5	99.5	99	77	84.5	77	88.25

Appendix

Table 13: Speech continuation performance across the consistency tasks of the SALMON benchmark for generation-based evaluation. Each model generates a continuation for each prompt, and accuracy scores are computed for each task by comparing the cosine similarity of the generated continuation with the positive and negative examples. Different embedding models are used for each task, specified in Appendix C.1 . Steps

=

512 is used for both CoBit-LibriTTS and CoBit-MLS, with the operating points selected for speech continuation.

Figure 6: Text-to-speech generation with CoBit-MLS. Given text and speaker tokens (extracted from a held-out example), the model is tasked with generating the speech part of the sequence. The top mel-spectrogram corresponds to the ground truth waveform, the middle one to the StableCodec reconstruction, and the bottom mel-spectrogram to the generated audio by the 633M parameter bitstream diffusion model. The tokenizer introduces visible artifacts, especially at high frequencies. CoBit generation spreads the utterance across a longer time span, without leading and trailing silence, replicating similar frequency patterns. The only exception is the gap between 1.7 and 1.9 seconds, which corresponds to the boundary between sentences.

Figure 7: Speech continuation with CoBit-MLS. Given speaker and speech tokens extracted from a 3-second audio prompt, the model is tasked with generating its own continuation. The top mel-spectrogram corresponds to the ground truth continuation, and the bottom mel-spectrogram to the continuation generated by the 633M parameter bitstream diffusion model. The generated spectrogram blends well with the audio prompt, introducing minimal artifacts at the prompt-continuation boundary. Speech continuation is scored not only in terms of acoustic consistency, but also for the semantic relevance and intelligibility of the generated content, shown on the right.

Steps	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	GenPPL-Text $\downarrow$	GenPPL-Speech $\downarrow$
4	367	250.5	1.28	20,101.9	1,132.5
8	267.8	181.4	1.31	16,791.8	708.2
16	173.4	120.5	1.51	9,002.5	303.4
24	100	70.4	1.94	5,377.7	440.5
32	68	47.4	2.3	3,665.2	391.4
48	41.7	28.6	2.88	1,857.4	359.5

Appendix

Table 14: CoBit-MLS joint generation performance at different step counts, computed by generating

1,024

samples. All rows inject stochasticity at

\gamma=0.175

. Unconditional generation does not use classifier-free guidance, so each integration step corresponds to a single function evaluation.

	Text-to-Speech				Speech-to-Text
Steps	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SpkSim $\uparrow$	WER $\downarrow$ Clean	WER $\downarrow$ Other
4	184.2	133.3	1.51	0.05	15.4	34.9
8	32	21.1	3.03	0.19	9	23.9
16	6.6	4.5	3.85	0.25	8.6	23
24	6.7	4.8	3.96	0.26	8.4	22.8
32	5.7	4	3.99	0.26	8.3	22.7

Appendix

Table 15: CoBit-MLS cross-modal performance at different step counts, computed by generating

1,024

samples. All rows inject stochasticity at

\gamma=0.175

. Classifier-free guidance with gs=5 is used throughout the table.

Steps	UTMOS $\uparrow$	FSD-wlm $\downarrow$	FSD-e2v $\downarrow$	SpkSim $\uparrow$	GenPPL-Speech $\downarrow$	SALMON $\uparrow$
4	1.44	6,001	156.2	0.05	375.1	74.58
8	1.82	3,409.6	42.1	0.14	262.6	80.08
16	3.12	2,236.3	8.9	0.3	232	84.41
24	3.53	2,176.6	7.1	0.33	200.9	84.41
32	3.7	2,117.6	6.8	0.34	178.3	84.25
48	3.83	2,127.4	7.5	0.34	150.6	85.08

Appendix

Table 16: CoBit-MLS speech continuation performance at different step counts, computed by generating

1,024

samples. All rows inject stochasticity at

\gamma=0.175

. Classifier-free guidance with gs=5 is used throughout the table.

Direction	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SpkSim $\uparrow$	FSD $\downarrow$	GenPPL $\downarrow$	SALMON $\uparrow$
Deterministic Sampler
text $\rightarrow$ speech	5.9	4.1	3.97	0.26	—	—	—
speech $\rightarrow$ text	8.6 / 23.4	4.7 / 14	—	—	—	—	—
speech continuation	—	—	3.85	0.33	2,130.4 / 8.2	136.9	85.41
joint	11.3	7.3	3.54	—	—	413.9 / 204.2	—
Stochastic sampler, $\gamma$ =0.175, gs=5

Appendix

Table 17: CoBit-MLS scores with a deterministic sampler across all four tasks. Integration steps are fixed to 256 and the conditional task use classifier-free guidance with strength 5.

	Text-to-Speech				Speech-to-Text
gs	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SpkSim $\uparrow$	WER $\downarrow$ Clean	WER $\downarrow$ Other
1	8.6	6	3.77	0.25	9.3	23.8
2	5.8	4.9	3.99	0.26	8.2	22.1
3	6.5	5.2	4.01	0.26	8.1	22.2
4	5.4	4.5	4.03	0.27	8.1	22.2
5	5.4	5.2	4.03	0.27	8.2	22.4

Appendix

Table 18: CoBit-MLS cross-modal performance at different guidance scales. All rows use 256 integration steps (256 function evaluations at gs=1; 512 for gs

>

1) and stochasticity at

\gamma=0.175

gs	UTMOS $\uparrow$	FSD-wlm $\downarrow$	FSD-e2v $\downarrow$	SpkSim $\uparrow$	GenPPL-Speech $\downarrow$	SALMON $\uparrow$
1	3.88	1,953.7	5.1	0.33	172.1	83.41
2	3.94	2,054.1	7.8	0.34	116	85
3	3.92	2,073.1	9.1	0.34	116	85
4	3.92	2,140.3	10	0.34	96.1	84.83
5	3.91	2,108.4	10.6	0.34	94.7	84.41

Appendix

Table 19: CoBit-MLS speech continuation performance at different guidance scales. All rows use 256 integration steps (256 function evaluations at gs=1; 512 for gs

>

1) and stochasticity at

\gamma=0.175

	FSD-wlm			FSD-e2v
gs	Mean Term	Covariance Term	Total	Mean Term	Covariance Term	Total
1	684.5	1,269.1	1,953.7	0.19	4.86	5.05
2	748.7	1,305.4	2,054.1	0.3	7.51	7.81
3	756.3	1,316.8	2,073.1	0.33	8.78	9.11
4	792.7	1,347.6	2,140.3	0.36	9.65	10.01
5	777.2	1,331.3	2,108.5	0.37	10.24	10.61

Appendix

Table 20: Breakdown of FSD-wlm and FSD-e2v metrics across the mean term

\|\mu-\mu^{\prime}\|^{2}

and the covariance term

\text{trace}\{\Sigma+\Sigma^{\prime}-2(\Sigma\Sigma^{\prime})^{\frac{1}{2}}\}

for guidance sweeps. All rows use 256 integration steps (256 function evaluations at gs=1; 512 for gs

>

1) and stochasticity at

\gamma=0.175

	WER $\downarrow$	CER $\downarrow$	UTMOS $\uparrow$	SpkSim $\uparrow$
CoBit-MLS, Steps=512, $\gamma=0.27$ , gs=5
held-out features	5.3	4.1	4	0.27
ground-truth features	5.2	3.6	4.03	0.32
Held-out vs. ground truth samples	—	—	—	0.65

Appendix

Table 21: Effect of speaker tokens on text-to-speech performance. Both CoBit-MLS rows are obtained from text-to-speech generation.

	Token Agreement(%)	Position RMSE(%)	Sequence RMSE(%)	Cosine Similarity
Paired	7.62	5.08	5.69	0.785
Shuffled	0.26 ( $\pm$ 0.03)	10.58 ( $\pm$ 0.03)	10.99 ( $\pm$ 0.03)	0.222 ( $\pm$ 0.005)

Appendix

Table 22: Speaker conditioning realization for CoBit-MLS co-generation. Comparisons are drawn between BiCodec-global representations CoBit-MLS samples directly (speaker positions in its generated sequence) and BiCodec-global representations extracted from the waveforms CoBit-MLS generates. RMSE and cosine similarity metrics are computed on BiCodec-global’s latent space by transforming each FSQ code into its underlying latent vector. The first row computes paired sample correspondence (generated and realized representations for the same sequence); the second row gives a shuffled baseline for shuffled examples, averaged over 100 draws. Cosine similarity is averaged across 32 sequence positions and 1024 samples.

	BiCodec-global		CAM++
Model	Entropy(bits)	Cosine Similarity (sampled)	Cosine Similarity (realized)
CoBit (ours; from scratch, no text pretraining)
CoBit-LibriTTS	—	—	0.439
CoBit-MLS	8.64	0.27	0.295
Autoregressive System (SmolLM+Speaker $\rightarrow$ SparkTTS)
Cascade	—	—	0.29

Appendix

Table 23: Co-generation speaker diversity, measured in terms of speaker-token entropy and cosine similarity of the sampled BiCodec-global latents and the extracted CAM++ embeddings. Matched population statistics for the training set are computed using 100 random draws.

Figure 8: Speaker consistency and diversity distribution for co-generation, in terms of latent space cosine similarity. Top left: Per-sample embedding similarity between BiCodec-global FSQ components sampled by CoBit-MLS (Steps=512, $\gamma=0.175$ ) during co-generation and BiCodec-global FSQ components extracted from the generated waveforms (decoded StableCodec tokens). Top right: Embedding similarity of BiCodec-global FSQ components across the 1,024 BiCodec-global sequences sampled by CoBit-MLS. Both plots are averaged over sequence positions. Bottom: realized speaker diversity among the generated populations of CoBit-MLS, CoBit-LibriTTS (Steps=256, $\gamma=0.175$ ) and the SmolLM-Speaker-SparkTTS autoregressive cascade (Appendix C.8 ), measured in terms of embeddings from the speaker verification CAM++ checkpoint.

Figure 9: (Left) SpkSim and (Right) UTMOS score distributions for text-to-speech for the 39 speakers in LibriSpeech-PC test-clean. For 10 text prompts, we generate audio with CoBit-MLS using held-out speaker tokens from all speakers in the test set. We use Steps=512, $\gamma=0.27$ and gs=5, as selected for standard text-to-speech evaluation.

Figure 10: Breakdown of speech-to-text WER and CER for CoBit-MLS, using the selected setting in Table 6 .

Figure 11: LLM-as-a-Judge score distribution for CoBit and autoregressive SLMs. CoBit-LibriTTS almost always receives the lowest possible score. A large portion of CoBit-MLS samples perform moderately better, but neither of the model can consistently achieve scores of 3 or higher.

Figure 12: Protein representation and denoising. The displayed equation describes the deterministic, unguided DDIM core. (a) Residue-aligned sequence and LFQ bits. (b) Task conditioning and successive DDIM updates with the same denoiser.

Component	Setting
Transformer	20 layers; width 1024; 16 heads; FFN 4096; RoPE; SwiGLU
Optimisation	AdamW; $\beta=(.9,.95)$ ; weight decay .01; gradient clip 1
Learning rate	$1.1\!\times\!10^{-4}$ ; 10k warm-up; cosine decay to $10^{-6}$
Batch / precision	Effective batch 512; accumulation 2; 8 GPUs; FP32; TF32 allowed
Regularisation	Dropout .1; EMA .9999; self-conditioning probability .5
Noise / loss	Log-normal warm-up, then entropy-derived (App. A ); EDM-weighted bit loss

Appendix

Table 24: Training configuration of the reported protein run.

Task and sample count	Metric	Estimate [95% interval]
Inverse: 128 targets, 512 sequences	AAR (%) $\uparrow$	44.10 [40.21, 48.21]
First sequence: 128 refolds	scTM $\uparrow$	.739 [.704, .770]
	scRMSD (Å) $\downarrow$	6.104 [5.251, 7.020]
	$D_{\rm TM}$ / $D_{\rm RMSD}$ (%) $\uparrow$	25.00 / 23.44
Forward: 183 targets, 732 predictions	TM $\uparrow$	.554 [.516, .591]
	lDDT $\uparrow$	.541 [.508, .575]

Appendix

Table 25: Task results under the reported sampling configuration; brackets give 95% intervals. Joint rows use ProteinMPNN redesign.

Figure 13: Backbone evaluation by length for the 100 reported samples (20 per length), each with one ProteinMPNN redesign and ESMFold refold. Left to right: scTM, scRMSD and mean C

\alpha

pLDDT of the refold. Points show every sample and boxes show the median and interquartile range, with whiskers extending to the most extreme values within 1.5 interquartile ranges. pLDDT is predicted refold confidence.

Refolded sequence	$n$	scTM $\uparrow$	scRMSD (Å) $\downarrow$	$D_{\rm TM}$	$D_{\rm RMSD}$
CoBit	100	.911	3.060	69.0	41.0

Appendix

Table 26: Direct refolding of CoBit’s sequences for the 100 pairs reported in the main text. Success rates are percentages.

Figure 14: CFG sweep at

G=50

: inverse success rates on 128 targets and mean forward TM on 183 targets. Curves show point estimates. CFG 0 costs 99 branch evaluations; positive CFG costs 198.

Figure 15: Quality–cost trade-offs across all settings. Lines join non-dominated point estimates. Diamonds mark the reported configuration. Each joint setting uses 100 pairs with matched length/seed assignments and one ProteinMPNN redesign per backbone. Conditional target sets are unchanged.

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

CoBit: Language Modeling with Bitstream Diffusion
May 7, 2026Georgios Batzolis, Mark Girolami, Luca AmbrogioniDiffusion Language Model Decoding Diffusion Language Models
Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.
Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
Oct 6, 2026Mathias Ollu, Nikos KomodakisLanguage Modeling Diffusion Language Models
Ecole Polytechnique · Archimedes, Athena RC · University of Crete +1
MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation
Jun 15, 2026Kyeongmin Yeo, Yunhong Min, Minhyuk SungLatent Diffusion Models Multimodal Generation
KAIST

May 7, 2026J/K move · Enter open · S save

CoBit: Language Modeling with Bitstream Diffusion

Georgios Batzolis, Mark Girolami, Luca Ambrogioni

Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.

Diffusion Language Model Decoding Diffusion Language Models Continuous Diffusion Language Models Diffusion Model Inference Acceleration Diffusion Models

Open paper AskPDF arXiv

Diffusion language models (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregressive models in sample quality and diversity. Recent continuous flow and diffusion approaches have narrowed this gap. In this work, we further close the autoregressive gap by modeling text as a continuous diffusion process over fixed-width binary bitstreams. We refer to the resulting model as CoBit (Continuous Bitstream Diffusion). Our approach represents semantic tokens as analog bit sequences and uses a matched-filter residual parameterization to isolate contextual learning from analytic independent-bit posteriors. Crucially, we adopt a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile, concentrating stochasticity in high-information regions while remaining nearly deterministic elsewhere. On LM1B, our 130M-parameter model reaches a generative perplexity (GenPPL) of 59.76 at matched real-data entropy (4.31) using 256 neural function evaluations (NFEs), outperforming prior DLM baselines and reaching the autoregressive reference. On OpenWebText (OWT), our sampler establishes a new continuous-DLM Pareto frontier, achieving GenPPL 27.06 at entropy 5.26 using 4x fewer steps than previous 1024-NFE baselines. Scaling the same recipe to a 462M-parameter model (CoBit-M) further improves the OWT GenPPL-entropy frontier over the 130M model (CoBit-S) and over medium-scale continuous and discrete DLM baselines, reaching GenPPL 19.5 at entropy 5.40, near real-data entropy (5.44), and approaching pretrained GPT-2 Medium over the high-quality region. As an additional benefit, bitstream diffusion removes the O(V) vocabulary scaling bottleneck of standard DLMs: by predicting O(log V) bitwise logits via semantic bit-patching, it lowers memory and raises throughput, a scalable paradigm as vocabulary sizes grow.

Abstract

Multimodal Generation

Explore similar work

CoBit: Language Modeling with Bitstream Diffusion

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation