Mixture of Decoders for Diverse Dialog Response Generation

Wenchao Du

cs.CLSep 30, 2026

Mixture of Decoders for Diverse Dialog Response Generation

Authors: Wenchao Du

Organizations: Microsoft Corporation Redmond, WA 98052

Abstract

Mixture modeling is a long established machine learning technique for learning large sets of multi-modal data. While it is known that sequence-to-sequence models for dialog response generation suffer from the problem of low diversity, we hypothesize that it is because sequence-to-sequence models tend to learn a degenerate uni-modal distribution of responses. We then propose to incorporate a mixture of decoders into sequence-to-sequence models and try to make each decoder learn specialized topics in order to improve the diversity of generated responses. Our model is developed under the framework of conditional variational autoencoder (CVAE). We evaluate our approach on an open domain chat corpus and show improvement over strong baselines in quantitative measures and human evaluation.

Figures & tables

Figure 1: Sequence-to-sequence model learns a degenerate unimodal distribution of responses, as illustrated by the red lines.

Figure 2: The graphical representation of our mixture of decoders model, where each decoder is a VAD. The figure shows the first two recurrent steps for two decoders. Circles are variables. Attention mechanism is left out for simplicity.

	MoD	MoS
Output		✓
Bottleneck	✓
Attention	✓
RNN Decoder	✓

Table 1: Parameter sharing in different layers in mixture of decoders (MoD) and mixture of softmax (MoS). Check mark indicates each mixture uses its own parameters.

	MoD	MoS
Order	$O(KEH)$	$O(KEV)$
K = 3	$<9M$	$>18M$
K = 5	$<11M$	$>30M$

Table 2: Comparison of the order and numeric values of the number of parameters used by MoD and MoS .

	Human	Seq2Seq	VAD	MoS	MoD	MoD (without z)	MoD (Supervised)
Length	10.35	8.93	7.88	10.53	12.34	9.24	14.01
BLEU	N/A	0.135	0.126	0.159	0.169	0.152	0.157
ROUGE-L	N/A	0.155	0.153	0.164	0.170	0.158	0.160
Cos. Sim. to targets	1.0000	0.0481	0.0603	0.0611	0.0699	0.0820	0.0715
Cos. Sim. to persona	0.1246	0.0614	0.0480	0.0491	0.0668	0.0515	0.0810
Distinct uni-gram	6327	353	566	424	617	590	661

Table 3: Evaluation results on Persona dataset.

Context	Sorry to hear that. Making crafts takes my stress away. I enjoy it.
Persona	I love animals and have two dogs.
Baseline	What do you do for a living?
MoD	Do you like dogs? I love them.
Context	I grew up with Morgans. I love horses.
Persona	I also enjoy cooking meals with food from our garden.
Baseline	I do not have any pets. Do you have any pets?

Table 4: Positive examples of generated responses from the MoD model with better reference to persona.

Panel of Figure 3 — Figure 3: Quantitative results by varying the number of mixtures. X-axis is for number of mixtures and y-axis for metrics. The number of mixtures are experimented with K = 1, 3, 4, 5.

Model	Win	Loss	Tie
Ours vs VAD	$47.0\pm 12.1\%$	$25.0\pm 10.3\%$	$28.0\pm 19.8\%$
Ours vs MoS	$42.0\pm 12.2\%$	$30.0\pm 6.6\%$	$28.0\pm 11.0\%$

Table 5: Human evaluation results.

	Example 1	Example 2	Example 3
Mix. 1	What kind of music do you like	Do you like movies ? I love horror movies	Do you like to watch TV
Mix. 2	What city are you from	Yes I do . I am a full time student	What are you majoring in
Mix. 3	Hi how are you today	I am great . How are you	That sounds like a lot of fun
Mix. 4	I am a huge fan of red hot chili peppers . They are my favorite	I am a huge fan of French fries . You	Do you like to cook ? I love it
Mix. 5	What do you do for a living	What do you like to do in your spare time	I do not have a lot of free time . What about you

Table 6: Examples of generated responses from each mixture in supervised MoD.

	Seq2Seq	VAD	MoS	MoD	MoD (Supervised)
BLEU	0.58	0.49	0.47	0.46	0.22
Distinct uni-gram	22.8	22.3	23.4	23.8	24.5
Distinct bi-gram	59.7	58.7	70.1	75.4	86.9
Distinct tri-gram	76.4	74.0	98.0	106.7	122.9

Table 7: Diversity evaluation of hypotheses sets.

Explore similar work

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following
Oct 1, 2026Scott Geng, Yufei Zhang, Joseph Lee +3Large Language Model Generation Diversity
University of Washington · Meta Superintelligence Labs
Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responses
May 27, 2026Kerui Peng, Feifei Li, Xingyu Fan +1Large Language Model Generation Responses
WeChat, Tencent Inc., Beijing, China
Where You Inject Diversity Matters: A Unified Framework for Diverse Generation
Jun 9, 2026Cheng Zhang, Rui Xin, Chudi ZhongDiversity Unified Framework
UNC Chapel Hill · University of Washington

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

Scott Geng, Yufei Zhang, Joseph Lee, Jerry Li+2

University of Washington · Meta Superintelligence Labs

Large Language Model Generation Diversity Decoding Instruction

Open paper AskPDF arXiv

We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations that scales with model capability. Across open-ended domains (in-the-wild chat, creative writing, planning for image generation, and protein design), Gacha Decoding significantly outperforms existing generation diversity approaches at equal quality (up to 2.4x Vendi over the next-best prior approach), reaching the same number of high-quality modes with over an order of magnitude fewer samples (11.0x) and discovering novel modes that no other approach surfaces. Our key insight is to treat diversity as an instruction-following problem: rather than relying on the LM's token entropy, we combine its instruction-following capability with randomness from an external RNG tool to scalably identify and realize distinct modes of the response space. This approach of "planning with dice" enables Gacha to invert the long-observed tension between diversity and model capability. As the underlying LM becomes a better instruction follower, diversity under Gacha Decoding consistently improves--even as its token entropy and diversity under prior approaches decline. Together, our results highlight that instruction following, rather than token entropy alone, can drive generation diversity.

Abstract

Explore similar work

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responses

Where You Inject Diversity Matters: A Unified Framework for Diverse Generation