Data Selection for Multi-turn Dialogue Instruction Tuning
Authors: Bo Li, Shikun Zhang, Wei Ye
Organizations: National Engineering Research Center for Software Engineering, Peking University · School of Computer Science, Peking University · PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications
Instruction-tuned language models increasingly rely on large multi-turn dialogue corpora, but these datasets are often noisy and structurally inconsistent, with topic drift, repetitive chitchat, and mismatched answer formats across turns. We address this from a data selection perspective and propose MDS (Multi-turn Dialogue Selection), a dialogue-level framework that scores whole conversations rather than isolated turns. MDS combines a global coverage stage that performs bin-wise selection in the user-query trajectory space to retain representative yet non-redundant dialogues, with a local structural stage that evaluates within-dialogue reliability through entity-grounded topic grounding and information progress, together with query-answer form consistency for functional alignment. MDS outperforms strong single-turn selectors, dialogue-level LLM scorers, and heuristic baselines on three multi-turn benchmarks and an in-domain Banking test set, achieving the best overall rank across reference-free and reference-based metrics, and is more robust on long conversations under the same training budget. Code and resources are included in the supplementary materials.
Figures & tables
Figure 1: Comparison of turn-level quality between a single-turn instruction dataset (Alpaca) and a multi-turn dialogue dataset (Banking). We randomly sample 1,000 examples from each dataset and score the assistant responses for helpfulness, relevance and clarity on a 1–5 scale using GPT-4o as the evaluator.
MT-Eval
ConsistentChat
TopDial
Avg. Rank
L-E
G-E
Ent-F1
Cos
L-E
G-E
Ent-F1
Cos
L-E
G-E
Ent-F1
Cos
Backbone: LLaMA3-8B-Instruct
Backbone
8.04
7.44
0.569
0.831
8.11
6.73
0.222
0.808
7.12
6.68
0 .158
0.499
5.00
All Data
8 .09
7.41
0.567
0.845
8.42
7 .20
0 .310
0.794
6.61
6.20
0.139
0.409
5.67
Random Data
8.00
7.41
0.558
0.842
8 .46
7 .20
0.306
0.783
6.58
6.20
0.140
0.409
6.92
SuperFiltering
8.08
7.44
0.568
0 .846
8.38
7.11
0.301
0.788
6.98
6.27
0.156
0.458
4.92
Table 1: Main results of MDS and baseline selection methods on Baize dataset and three multi-turn benchmarks. L-E and G-E denote LLM-EVAL and G-EVAL , respectively; Ent-F1 and Cos denote entity-level F1 and embedding cosine similarity, respectively. All reported scores are averaged over 5 runs for each method. Bold numbers denote the best score and underlined numbers denote the second-best score in each column; the rightmost column reports the average rank over all 12 metrics, where lower is better.
Banking Test
ConsistentChat
G-E
Ent-F1
G-E
Ent-F1
Backbone
6.28
0.184
6.70
0.222
All Data
6.58
0.354
7.12
0.288
Random Data
6.42
0.313
7.22
0.290
SuperFiltering
6.44
0.305
7.12
0.283
Rethinking
6.62
0.333
7.18
0.282
Table 2: Domain-specific selection performances on the Banking corpus.
MT-Eval
TopDial
G-E
Ent-F1
G-E
Ent-F1
MDS
7.52
0.584
6.48
0.173
Global-only
7.38
0.580
6.30
0.157
Local-only
7.44
0.576
6 .46
0 .162
w/o Binning
7.44
0.570
6.26
0.145
w/o Form Filter
7.42
0.574
6 .46
0.148
Table 3: Ablation on the components of MDS using LLaMA3-8B. All rows are variants of MDS.
Figure 2: Performance on short (turns 1–3) vs. long (turns >10 ) queries on MT-Eval. We show Ent-F1 and cosine similarity averaged over turns in each bucket. Blue bars denote MDS, green bars adapted single-turn selectors (Rethinking, ZIP), and orange bars dialogue-level baselines (CC-Score, DialScore).
All Selected
Top 20% by H(d)
ESC
ESC
HAR
ENR
MDS
0.599
0.614
0.514
0.714
shuffle level
Pair
0.596
0.602
0.497
0.707
Block(k=2)
0.596
0.606
0.504
0.708
Block(k=4)
0.596
0.603
0.498
0.707
Table 4: Order-perturbation analysis on the same 10K dialogues selected by MDS. We report the turn-weighted Entity Sequence Consistency score (ESC) on the full set ( All Selected ), and additionally report ESC together with two interpretable components on the Top 20% high-history-dependency subset ranked by H(d) : History Anchoring Rate (HAR) and Entity Novelty Rate (ENR).
Figure 3: Error-type gaps on difference sets between MDS and each baseline selector. Each column compares MDS against one baseline. For example, in the DialScore column, a cell value is the percentage-point gap Δ=p(MDS-only)−p(DialScore-only) for that error type. Detailed statistics are provided in the Appendix D .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
#Dialogues
Avg.turn
Baize
54,456
3.95
Banking
66,948
5.01
MT-Eval
130
7.30
ConsistentChat
1,000
7.73
TopDial
1,321
5.11
Banking Test
1,000
4.98
Appendix
Table 5: Statistics of the dialogue selection pools and evaluation benchmarks. #Dialogues denotes the number of multi-turn dialogues in each dataset, and Avg. turn denotes the average number of user–assistant turns per dialogue.
Figure 4: Prompt used for GPT-based multi-turn dialogue error-type classification. The judge assigns a primary error label and an optional set of additional error labels from a fixed taxonomy, and returns brief evidence by referencing specific turns.
Error type
CC
DialScore
Rethinking
SuperFiltering
MDS-only
B-only
MDS-only
B-only
MDS-only
B-only
MDS-only
B-only
No Error
87.5
82.5
87.0
79.5
86.0
80.5
86.5
77.0
Topic Drift
9.5
10.0
5.0
9.5
6.0
9.0
6.0
11.5
Repetition
1.0
2.5
0.5
2.5
3.0
2.5
3.0
2.0
Form Mismatch
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
Contradiction
1.0
4.0
4.5
5.0
3.5
2.0
2.5
2.0
Appendix
Table 6: Error-type distribution on difference sets . For each baseline selector B , we compare 1K dialogues sampled from MDS-only ( D\textscMDS∖DB ) versus B-only ( DB∖D\textscMDS ). Values are percentages; higher No Error indicates cleaner dialogues.
Figure 5: Prompt used for LLM-EVAL.
Figure 6: Prompt used for G-EVAL.
MT-Eval
ConsistentChat
TopDial
L-E
G-E
Ent-F1
Cos
L-E
G-E
Ent-F1
Cos
L-E
G-E
Ent-F1
Cos
100
8.10
7.46
0.561
0.847
8.46
7.16
0.310
0.791
6.94
6.40
0.156
0.453
500
8.08
7.48
0.568
0.843
8.48
7.20
0.305
0.792
6.84
6.38
0.145
0.440
1000(default)
8.16
7.52
0.584
0.857
8.52
7.26
0.316
0.797
7.12
6.48
0.173
0.465
1500
8.10
7.44
0.561
0.844
8.48
7.16
0.302
0.795
6.94
6.40
0.162
0.457
2000
8.04
7.46
0.576
0.846
8.46
7.16
0.307
0.799
6.88
6.40
0.161
0.450
Appendix
Table 7: Ablation on bin granularity on Baize with LLaMA3-8B-Instruct . We report performance under four metrics (L-E, G-E, Ent-F1, and Cos), where higher is better for all metrics. Bold numbers indicate the best score in each column.
Figure 7: Local-stage structured scoring prompt used in MDS. We query Qwen3-8B with a single-turn QA pair and require a JSON -only output that simultaneously extracts question/answer entities ( q_entities , a_entities ) and predicts a discrete form/style compatibility score ( style_match_score ∈{0,1,2} ) with a brief rationale. This one-pass, multi-signal design enables efficient local scoring by avoiding separate calls for entity statistics and form assessment.
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · WeChat, Tencent Inc, Beijing, China · University at Buffalo, New York, United States