We propose a multi-task learning approach for multi-party dialogue intent recognition that leverages an auxiliary task that models turn-taking dynamics. Specifically, we introduce turn-transition entropy, a self-supervised target computed from the sequence of speaker transitions, which quantifies the predictability of interaction patterns. Experiments on multiple pre-trained models demonstrate that incorporating this auxiliary task improves intent recognition performance, outperforming existing approaches which ignore multi-party interaction dynamics. We find that our proposed continuous target can be learned as a single-task objective, suggesting that it is an actual signal carrying useful information.
Figures & tables
Figure 1: Examples of low- and high-entropy turn transitions. The top conversation follows a structured pattern of speaker transitions, presenting a lower turn-transition entropy. The bottom conversation exhibits more varied and less organized speaker transitions, leading to a higher entropy 2 2 2 We define the entropy computation in Section 3.3 . . Note that both examples have the same number of speakers and turn shifts.
Figure 2: Overview of our multitask learning approach architecture. Input representation includes model token embeddings; segment IDs to distinguish contextual from target utterances (i.e. latest turn utterance, to classify into any intent); and speaker embeddings to discriminate context uttered by the speaker producing the target utterance.
Figure 3: Entropy distributions computed on the training sets of MIntRec2.0 and MPGT.
MPGT m
MIntRec2.0
Model
Method
F1
Acc.
F1
Acc.
BERT
CTX TEXT Zhang et al. (2024) 4 4 4 Performance metrics reported in the original paper.
- -
- -
53.61
59.04
Label Space Reduction Castillo-López et al. (2025b)
71.38
91.73
41.92
41.66
baseline (ours)
88.81
95.19
55.07
60.33
TT Entropy Aux. Task (ours)
92.80
96.09
55.87
60.22
Norm. TT Entropy Aux. Task (ours)
92.05
96.69
56.01
60.35
Table 1: Performance evaluation on the intent classification task. Scores correspond to the average macro F1 and accuracy over 5 runs. Scores in bold highlight the best performance on the same backbone model.
Method
F1
Acc.
He et al. (2021c)
70.90
79.15
Qamar et al. (2023)
72.49
80.72
baseline (ours)
73.54
81.00
TT Entropy Aux. Task (ours)
74.31
81.83
Norm. TT Entropy Aux. Task (ours)
74.78
81.95
Table 2: Evaluation on MEETInt with RoBERTa LARGE . Scores correspond to the average macro F1 and accuracy over 5 runs. Scores in bold highlight the best performance.
MPGT
MIntRec2.0
Model
random
real
random
real
BERT
0.680
0.274
1.165
0.379
RoBERTa
0.677
0.205
1.161
0.428
DeBERTa
0.682
0.220
1.162
0.330
Table 3: Mean Absolute Error (MAE) on the regression target learned as a single task on real and random entropy values. Scores correspond to the average MAE over 5 runs. A lower MAE suggests an ease on the continuous variable learning.
Figure 4: Mean Absolute Error values along training epochs on MPGT validation set across all models.
Model
MPGT
MIntRec2.0
BERT
-8.84
-0.70
RoBERTa
-5.53
-0.29
DeBERTa
-0.41
-2.15
Table 4: Absolute difference in macro-F1 score of the classification task when replacing the auxiliary regression targets with randomly shuffled values during multitask training ( F1random−F1real ). Negative values indicate a classification performance degradation when training with random regression targets.
Dataset
H
p -value
ϵ2
MPGT
51.57
7.11×10−9
0.112
MIntRec2.0
155.15
3.45×10−19
0.021
Table 5: Relationship analysis between the intent classes and the auxiliary variable. Kruskal-Wallis test results and effect sizes across datasets. H corresponds to the Kruskal–Wallis statistic, and ϵ2 is the effect size
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Property
Entropy
Norm. entropy
turn index
0.69 *
0.46 *
MPGT
context length
0.08
0.01
speaker count
0.66 *
0.20 *
turn index
0.77 *
0.54 *
MIntRec2.0
context length
0.03
0.02
speaker count
0.89 *
0.24 *
Appendix
Table 6: Spearman correlations between entropy measures and utterance properties. An asterisk ( ∗ ) indicates p<0.01 .
Figure 5: Prompt template used on the LLM baseline for the intent classification task. Highlighted text in blue varies among dataset examples.
hyperparameter
value MPGT
value MIntRec2.0
eval_monitor
macro F1
macro F1
train_batch_size
16
16
eval_batch_size
16
16
test_batch_size
16
16
wait_patience
5
5
num_train_epochs
60
60
Appendix
Table 7: Set of hyperparameters used on BERT, RoBERTa, and DeBERTa fine-tuning baseline experiments.
hyperparameter
search range
step
weight λ
[0.01,0.20]
0.01
layer l
[3,21]
1
Appendix
Table 8: Optuna hyperparameter search space for the average weight λ and layer number l on the multi-task architecture.
hyperparameter
value MPGT
value MIntRec2.0
BERT
eval_monitor_main
macro F1
macro F1
eval_monitor_aux
MAE
MAE
train_batch_size
16
16
eval_batch_size
16
16
test_batch_size
16
16
Appendix
Table 9: Set of hyperparameters used on BERT, RoBERTa, and DeBERTa fine-tuning multi-task method experiments after Bayesian optimization.
train
dev
test
labels
MPGT
.5K
.1K
.2K
8
MIntRec2.0
9.9K
1.8K
3.2K
30
MEETInt
81.9K
21.0K
31.8K
12
Appendix
Table 10: Number of examples per dataset split and number of labels on each corpus.
MPGT
MIntRec2.0
Model
Input Representation
F1
Acc.
F1
Acc.
BERT
w/o context
86.84
95.04
53.08
58.67
only target speaker
87.92
95.79
55.18
60.45
all speakers w/o speaker emb.
87.69
94.69
54.74
59.94
all speakers w. speaker emb.
90.65
95.49
55.36
60.57
RoBERTa
w/o context
81.85
93.83
55.44
59.85
Appendix
Table 11: Performance evaluation on the intent classification task using different input representation strategies. Scores correspond to the average macro F1 and accuracy over 5 runs. Scores in bold highlight the best performance on the same backbone model.
MPGT
MIntRec2.0
Model
Method
F1
Acc.
F1
Acc.
RoBERTa BASE
Turn Modeling He et al. (2021c)
82.00
94.26
49.76
57.68
Turn Aware Speaker Graph Qamar et al. (2023)
83.30
95.19
52.31
57.70
RoBERTa LARGE
Turn Modeling He et al. (2021c)
73.64
92.92
51.40
56.40
Turn Aware Speaker Graph Qamar et al. (2023)
81.28
94.29
52.82
57.92
Appendix
Table 12: Performance evaluation on the intent classification task. Comparison between RoBERTa BASE and RoBERTa LARGE on prior work baselines. Scores correspond to the average macro F1 and accuracy over 5 runs.
Figure 6: Distribution of the turn-transition entropy values across classes for MPGT.
Dialogue models are inherently reactive, responding to the current user turn without anticipating upcoming intents, which leads to redundant interactions in multi-intent settings. We address this limitation by introducing a lightweight intent-transition prior derived from dialogue data and injected into the system prompt at inference time. We instantiate this prior using a Temporal Bayesian Network (T-BN) trained on per-turn intent annotations in MultiWOZ 2.2. The T-BN achieves Recall@5 = 0.787 and MRR = 0.576 on 1,071 held-out USER-turn pairs. In a ground-truth replay over 200 dialogues, BN-guided generation improves Coverage AUC from 0.742 to 0.856 and reduces the number of turns required to reach 75% intent coverage from 3.95 to 2.73. These results show that lightweight intent-transition guidance enables more proactive and efficient dialogue behavior without modifying the underlying language model.
Yang Luo
Department of Computer Science University of Hong Kong
Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.
Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.
Kratika Bhagtani, Mrinal Anand, Yu Chen Xu +1
School of Electrical and Computer Engineering, Purdue University, United States · Ishiki Labs Inc.