DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations

Wenchao Du; Fangxin Shang; Long Wu; Runze Cui; Qing Yang; Yehui Yang

cs.CLOct 26, 2026

DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations

Authors: Wenchao Du, Fangxin Shang, Long Wu, Runze Cui, Qing Yang, Yehui Yang

Organizations: AI Lab, Qifu Technology, Beijing, China · University of Chinese Academy of Sciences

Abstract

In high-stakes domains such as finance, critical information evolves through multi-party interaction. Recovering these changes and their sources is fundamental to reliable speech understanding. Existing speech benchmarks focus on transcription, local understanding, and turn-level state prediction, leaving joint decision evolution across speakers insufficiently evaluated. We introduce DecisionTraceBench, a speech state-tracking benchmark for Decision-Critical Information State Tracking (DCIST). It jointly evaluates decision event sequences, speaker attribution, scope, and temporal evidence, using trajectory-level F1 to measure complete event-chain recovery. Our dataset comprises Chinese financial dialogues involving two to four speakers, with 100.39 hours of synthetic audio and 8.61 hours of human-performed recordings. On a fixed cohort of human-performed recordings, five cascaded systems and two direct-audio models achieve DCIST scores of 5.73% to 21.35%. Ablations with speaker-labeled reference scripts and no temporal localization requirements yield a maximum score of only 38.72%. Audio perturbation experiments further show that overall transcription error rates do not adequately reflect complete trajectory recovery. These findings reveal limitations in jointly understanding information provenance and state evolution, providing a benchmark for evaluating decision-process understanding and decision traceability in high-stakes speech applications.

Figures & tables

Figure 1: Full-trajectory speech state tracking. Systems recover an event chain from audio without manual transcripts or gold speaker labels. Evaluation jointly checks required-event coverage, attributes and relations, and temporal evidence. Start, Pause, Progress, and Finalize correspond to start , hold , advance , and settle .

Action	Trigger	State after
start	First valid candidate or explicit initiation	active
advance	Substantive progress or resumption	active
hold	Actual suspension due to an explicit obstacle	held
settle	Explicit, source-supported scoped conclusion	settled

Table 1: Event actions and resulting states.

Figure 2: Minimum required chains and additional nodes. S1 handles documentation and S2 performs technical verification. Omitting a required suspension or resumption fails the trajectory despite the correct terminal value. Supported refinements are exempt; an invalid insertion fails the trajectory. All other attributes, evidence intervals, and predecessor relations are assumed correct.

Subset	Audio	Hours	Tracks / events
Synthetic	2,246	100.39	5,319 / 18,227
Human	26	8.61	308 / 562
Total	2,272	109.01	5,627 / 18,789

Table 2: Dataset size. Hours are playback duration; totals use unrounded values. A/B variants count as separate recordings.

Input pipeline	Trajectory model	N	DCIST	TP
FunASR	Astra	25	21.35	82
FunASR	Kimi-K3	25	9.80	38
FunASR	Qwen3.8-Max	25	5.75	18
VibeVoice	Kimi-K3	25	11.09	39
VibeVoice	Qwen3.8-Max	25	5.73	15
Direct audio	Gemini-4-Argon	25	18.82	72

Table 3: Full-trajectory recovery (DCIST, %) on a fixed cohort of 25 unperturbed human recordings, 16 dialogue families, and 293 gold trajectories. TP counts complete matches. VibeVoice uses native full-length transcription; Gemini receives full-length audio directly. All systems use the same minimum-chain protocol and time threshold. Appendix C.4 specifies the direct-audio configuration; Appendix F reports all 21 Holm-corrected comparisons and the retained cascade intervals.

ASR	Reasoner	DCIST	No time
FunASR	DeepSeek	32.71	51.98
FunASR	Qwen	28.33	46.47
VibeVoice	DeepSeek	22.61	74.59
VibeVoice	Qwen	21.64	67.70

Table 4: Synthetic results on 2,224 shared recordings (5,269 gold trajectories). Scores are percentages. The no-time column disables temporal checks only.

Model	Labeled	TP	Unlabeled	TP	$\Delta$ (pp)
Astra	38.72	133	28.99	100	+9.73
Kimi-K3	22.72	78	23.58	77	-0.86
DeepSeek V4 Flash	14.04	41	7.31	18	+6.73
Qwen3.8-Max	18.89	55	13.79	40	+5.10

Table 5: Reference-script results: 26 scripts, two input conditions, four models, and 208 predictions. Scores are percentages;

\Delta

is labeled minus unlabeled in percentage points. Both conditions disable time checks but retain strict speaker matching. TP counts complete trajectories.

FunASR ( $N=26$ )
Condition	CER	WER	cpWER	DCIST
Original	17.12	21.69	31.18	9.68
B2	18.77	23.68	33.15	7.64
VibeVoice ( $N=22$ )
Original	9.01	11.10	13.29	9.66
B2	10.41	12.57	14.21	6.34

Table 6: Same-route Kimi-K3 comparison with B2 keyword noise. All metrics are percentages. FunASR and VibeVoice share 26 and 22 completed recordings across conditions, respectively. Within each frontend, the API route, prompt, and output budget are held constant. Transcription metrics use the reading scripts as references.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Utterance (English translation)	Event
S1: Start checking this equipment at CNY 3 million.	start: 3.0M
S2: An invoice is missing. Pause this; we cannot finalize it yet.	hold: 3.0M
S2: The invoice and serial numbers now check out. Depreciation gives 2.9 million. Resume verification.	advance: 2.9M
S2: Yes, record the equipment at 2.9 million.	settle: 2.9M

Appendix

Table 7: A required chain in an established technical-verification context. M denotes million CNY.

Task family	Existing evaluation or representation	Full-trajectory extension
MultiWOZ, ProPara, OpenPI ( Budzianowski et al., 2018 ; Dalvi et al., 2018 ; Tandon et al., 2020 )	Dialogue belief states or entity changes in procedural text	Joint recovery of spoken evolution, source identity, scope, and acoustic evidence
SpokenWOZ and spoken DST ( Si et al., 2023 ; Lee et al., 2023 ; Lee and Pfister, 2026 )	Turn-level slot updates, including synthetic-speech and audio-verification studies	Event actions, authority scope, and time evidence evaluated over the full chain
MEET consensus model ( Esfandiari-Baiat and Edlund, 2026 )	Incremental decision history with provenance and time metadata	A unified raw-audio prediction benchmark with joint trajectory scoring
M3-SLU / HEAR ( Kwon et al., 2025 ; Lee et al., 2026a )	Speaker binding, provenance questions, and voice-swap controls	Attribution as part of effective decision events and their scoped consequences
SpeechEE ( Wang et al., 2024 )	Spoken event types, triggers, and arguments	The continuous suspension, resumption, and resolution of a proposition
Audio Arena ( Arcada Labs, 2026 )	Correction, memory, and state maintenance in interactive speech	Traceable record recovery from an existing multi-speaker conversation

Appendix

Table 8: Structured comparison of prediction targets.

Statistic	Synthetic	Human-recorded
Scenario categories	14	10
Two / three / four speakers	448 / 1,357 / 441	5 / 14 / 7
A/B pairs (recordings)	790 (1,580)	10 (20)
Standalone scripts	666	6
Dialogue families	1,456	16
Minimum / median / maximum duration (s)	95.47 / 154.50 / 272.39	495.50 / 997.53 / 3,264.00

Appendix

Table 9: Recording distribution. Speaker counts refer to script roles. Both A/B variants are counted as audio, but grouped into one family.

Stage	Model / implementation	Configuration
Script generation	DeepSeek V4 Flash	Temperature 0.45; generates colloquial scripts conditioned on predesigned gold trajectories
Cross-review	GLM5.3-Flash; Qwen3.8-Max	Temperature 0.05; respective output budgets 18,000 and 9,000 tokens
Speech synthesis	CosyVoice2-0.5B ( Du et al., 2024 )	24 kHz, mono, 16-bit; speed 1.0; half precision; turn-level synthesis
Full-length transcription	FunASR	Punctuation-based segmentation; processing batch configured for 300 s; no supplied speaker count
Native transcription	VibeVoice-ASR	Non-sampling decoding, SDPA; 32,768-token budget for synthetic, original human, and B2; 65,536 for historical A1
Chunked transcription	VibeVoice with FunASR speaker alignment	Continuous non-overlapping 300-s and 60-s variants; 16,384 tokens per chunk; uncertain identities retained

Appendix

Table 10: Generation and inference settings. Different frontends, chunk lengths, and auxiliary inputs define distinct conditions.

ASR	Reasoner	Completed / total	Format errors	DCIST	No time
FunASR	DeepSeek	2232/2246	66	32.65	51.88
FunASR	Qwen	2245/2246	95	28.17	46.25
VibeVoice	DeepSeek	2239/2246	53	22.60	74.44
VibeVoice	Qwen	2241/2246	65	21.66	67.57

Appendix

Table 11: Synthetic results over all completed outputs per system, with coverage reported separately. DeepSeek denotes DeepSeek V4 Flash and Qwen denotes Qwen3.8-Max. Complete malformed answers remain scored; incomplete outputs are not imputed as zeros.

ASR	Reasoner	Time	Legacy	+ Amount	+ Semantics	+ Extras
FunASR	DeepSeek	Yes	15.42	15.62	31.90	32.71
FunASR	DeepSeek	No	24.16	24.45	50.55	51.98
FunASR	Qwen	Yes	6.47	6.75	28.10	28.33
FunASR	Qwen	No	10.38	10.92	46.08	46.47
VibeVoice	DeepSeek	Yes	10.95	10.97	22.18	22.61
VibeVoice	DeepSeek	No	36.18	36.21	72.96	74.59

Appendix

Table 12: Sequential scoring decomposition on the same 2,224 recordings, keeping predictions and gold fixed. Columns cumulatively introduce monetary equivalence, reviewed expression mappings, and extra-node adjudication; the last column is the reported score. Increments depend on this order and are not independent causal effects.

Condition	Region	Mask duration	Changed samples
Original	No added noise	—	Reference
A1	Outside protected gold evidence	2,270.24 s	7.32% overall
A2	Expanded evidence-external speech	6,705.04 s	21.62%; one 11.58% exception
B2	Critical short words	2,541.69 s	8.20% overall

Appendix

Table 13: Frozen perturbation conditions, each with 26 full-length recordings and approximately 4 dB noise. Changed-sample proportions are measured against original waveforms. The other 25 A2 recordings range from 21.01% to 22.89%.

Frontend	Condition	N	CER (%)	WER (%)	cpWER (%)	DCIST (%)
FunASR	Original	26	17.12	21.69	31.18	12.41
FunASR	A1	26	19.20	24.17	34.59	7.91
FunASR	A2	26	23.67	29.51	43.00	10.19
VibeVoice	Original	23	9.10	11.13	13.33	10.10
VibeVoice	A1	23	10.09	12.34	14.23	11.22

Appendix

Table 14: Historical Kimi-K3 evidence-external noise experiments: 26 common FunASR recordings and 23 common VibeVoice recordings. Transcription metrics aggregate corpus-level errors against reading scripts. These routes differ from the B2 control.

Kimi-K3 condition	N	Automatic (%)	Fixed (%)	Difference
FunASR A1	26	7.91	9.73	+1.82
FunASR A2	26	10.19	8.94	-1.25
VibeVoice A1	23	11.22	12.46	+1.24

Appendix

Table 15: Automatic-speaker control. Fixed attribution uses speaker predictions from the unperturbed recording, not human gold identities. Differences are percentage points.

Comparison (first minus second)	Difference (pp)	Raw $p$	$p_{\rm Holm}$
F+Astra $-$ F+Kimi	+11.55	0.0029	0.0498
F+Astra $-$ F+Qwen	+15.59	0.0020	0.0371
F+Astra $-$ V+Kimi	+10.26	0.0078	0.1172
F+Astra $-$ V+Qwen	+15.62	0.0137	0.1777
F+Astra $-$ G-Argon	+2.53	0.0703	0.7031
F+Astra $-$ G-Flash	+14.75	0.0010	0.0205

Appendix

Table 16: All 21 pairwise main-system comparisons on the fixed 25-recording cohort in 16 families. F/V denote FunASR/VibeVoice; G-Argon/G-Flash denote Gemini-4-Argon/Gemini-3.8-Flash. Two-sided exact family-level sign-permutation tests enumerate 65,536 assignments. Holm correction covers all 21 contrasts jointly; bold adjusted values pass 0.05 before rounding. Differences are recording-macro DCIST percentage points.

System / contrast	Estimate (% / pp)	Pointwise 95% CI
F+Astra	21.35	[9.70, 34.62]
F+Kimi	9.80	[3.08, 19.52]
F+Qwen	5.75	[1.16, 11.19]
V+Kimi	11.09	[4.46, 19.17]
V+Qwen	5.73	[2.30, 9.46]
G-Argon	18.82	[8.55, 30.71]

Appendix

Table 17: Family-clustered 95% confidence intervals for all seven main systems and their 21 pairwise differences on the fixed 25-recording cohort (16 families; 20,000 bootstrap draws). F/V denote FunASR/VibeVoice; G-Argon/G-Flash denote Gemini-4-Argon/Gemini-3.8-Flash. System estimates are percentages; differences and their intervals are percentage points. These are pointwise, not simultaneous, intervals. Main-system significance is determined by the 21-contrast Holm correction in Table 16 , not by interval exclusion of zero.

Model	Labeled 95% CI	Unlabeled 95% CI	Difference 95% CI	$p_{\rm Holm}$
Astra	[26.06, 52.53]	[16.77, 42.44]	[4.18, 15.14]	0.019
Kimi-K3	[13.44, 34.36]	[16.21, 31.47]	[-5.90, 4.24]	0.766
Qwen3.8-Max	[16.36, 21.23]	[7.42, 20.30]	[-1.11, 11.20]	0.285
DeepSeek V4 Flash	[9.31, 19.02]	[3.25, 11.60]	[3.05, 10.56]	0.019

Appendix

Table 18: Family-clustered intervals and two-sided paired tests for 26 scripts in 16 families. Differences are labeled minus unlabeled, in percentage points. Holm correction jointly covers the four within-model label contrasts.

Kimi-K3 comparison	N	Families	Difference	95% CI	$p_{\rm Holm}$
FunASR: B2 $-$ Original	26	16	-2.04	[-8.15, 2.40]	0.625
VibeVoice: B2 $-$ Original	22	16	-3.32	[-5.51, -0.96]	0.039
FunASR: A1 $-$ Original	26	16	-4.50	[-9.08, -0.81]	0.047
FunASR: A2 $-$ Original	26	16	-2.22	[-6.05, 0.85]	0.562
VibeVoice: A1 $-$ Original	23	15	+1.13	[-1.71, 4.72]	0.609
FunASR A1: Fixed $-$ Automatic	26	16	+1.82	[-0.11, 4.32]	0.469

Appendix

Table 19: Paired acoustic and automatic-speaker controls. Holm correction is separate for two B2 comparisons, three historical A comparisons, and three speaker controls. Only jointly completed recordings enter each contrast. Differences and intervals are percentage points. Fixed speaker labels are still automatic.

System / script input	Fields	Value recall (%)	Chain recall (%)	Value-only hits (%)
FunASR + Astra	266	62.41	30.45	51.20
FunASR + Kimi-K3	266	52.63	14.29	72.86
FunASR + Qwen3.8-Max	266	32.71	6.77	79.31
VibeVoice + Kimi-K3	266	51.50	14.66	71.53
VibeVoice + Qwen3.8-Max	266	35.71	5.26	85.26
Astra (labeled)	280	68.93	46.43	32.64

Appendix

Table 20: Terminal-value projections versus full trajectories from the same predictions. Both recall denominators are gold fields with non-null terminal values: 266 fields for the common 25 audio recordings, 280 for all 26 scripts. The last column divides value-correct but chain-incorrect fields by all value-correct fields. Full-chain recall is not macro-F1 DCIST.

System / script input	Terminal-value recall 95% CI	Full-chain recall 95% CI
FunASR + Astra	[47.77, 74.80]	[13.08, 45.08]
FunASR + Kimi-K3	[37.96, 64.91]	[4.09, 25.82]
FunASR + Qwen3.8-Max	[20.93, 44.24]	[1.90, 11.30]
VibeVoice + Kimi-K3	[32.14, 66.99]	[5.29, 25.25]
VibeVoice + Qwen3.8-Max	[25.16, 45.87]	[2.17, 8.66]
Astra (labeled)	[55.00, 79.80]	[30.70, 60.36]

Appendix

Table 21: Family-clustered 95% confidence intervals for terminal-value and full-chain recall on the same non-null-terminal-value subsets as Table 20 .

System / script input	Field pairs	Both correct	Joint (%)	95% CI	Changed subset
FunASR + Astra	87	21	24.14	[0.00, 41.18]	3/35
FunASR + Kimi-K3	87	13	14.94	[0.00, 29.47]	0/35
FunASR + Qwen3.8-Max	87	3	3.45	[0.00, 7.81]	1/35
VibeVoice + Kimi-K3	87	5	5.75	[0.00, 13.13]	0/35
VibeVoice + Qwen3.8-Max	87	1	1.15	[0.00, 4.00]	0/35
Astra (labeled)	102	41	40.20	[13.89, 61.07]	14/40

Appendix

Table 22: A/B joint recovery. Audio systems share nine complete families and 87 field pairs; scripts cover ten families and 102 pairs. A pair succeeds only if both trajectories match. The changed subset has differing gold structure or attributes after ignoring evidence times and renamable IDs (35 and 40 pairs). Intervals resample paired families; fields unique to one variant are excluded.

Stratum	Audio	Tracks	F+A	F+K	F+Q	V+K	V+Q
2 speakers	5	40	20.00	6.98	12.00	12.68	10.68
3 speakers	13	163	20.24	5.68	3.64	8.77	4.99
4 speakers	7	90	24.36	19.46	5.21	14.25	3.57
$<15$ min	9	68	5.84	1.39	2.22	3.02	5.25
15– $<30$ min	12	163	34.88	18.21	9.25	19.25	6.98
$\geq 30$ min	4	62	15.62	3.49	3.23	4.76	3.04

Appendix

Table 23: DCIST (%) by speaker count and duration on the same 25 recordings. F/V denote FunASR/VibeVoice; A/K/Q denote Astra/Kimi-K3/Qwen3.8-Max. Each stratum macro-averages recording F1. Counts refer to script roles and full audio duration; the middle duration bin excludes 30 minutes.

Nodes	Tracks	F+A	F+K	F+Q	V+K	V+Q
1	131	56/131 (42.75)	31/131 (23.66)	15/131 (11.45)	29/131 (22.14)	10/131 (7.63)
2	103	18/103 (17.48)	6/103 (5.83)	1/103 (0.97)	9/103 (8.74)	2/103 (1.94)
3+	59	8/59 (13.56)	1/59 (1.69)	2/59 (3.39)	1/59 (1.69)	3/59 (5.08)

Appendix

Table 24: Full-chain recall by required gold-node count: hits/trajectories (%). F/V denote FunASR/VibeVoice; A/K/Q denote Astra/Kimi-K3/Qwen3.8-Max. These are gold-trajectory recall rates, not stratified DCIST.

System / script input	N	Macro F1	Micro F1	All nodes	Exempt nodes	Extra (%)
FunASR + Astra	25	21.35	27.06	732	193	26.37
FunASR + Kimi-K3	25	9.80	13.36	483	73	15.11
FunASR + Qwen	25	5.75	6.36	476	50	10.50
VibeVoice + Kimi-K3	25	11.09	13.66	493	76	15.42
VibeVoice + Qwen	25	5.73	5.23	491	66	13.44
Astra / labeled	26	38.72	43.18	744	203	27.28

Appendix

Table 25: Auxiliary metrics for the common audio cohort (25 recordings) and scripts (26 per condition). Macro F1 is DCIST. F1 and extra-node proportions are percentages. Node denominators include parseable outputs; no nodes are invented for unparseable answers, which remain subject to the format-error rule.

Field / evidence	Original input and prediction	B2 input and prediction	Located mismatch
Financing amount; R0023; 674.97–680.23 s	“Set financing at 8.4 million”; 9.0M start, 8.4M settlement	“Set financing at 2.4 million”; 9.0M start, 2.4M settlement	Wrong settlement value; other required attributes, edges, and time pass
Service fee; R0023; 713.93–720.30 s	“Set the fee rate at 0.8%”; 1% start, 0.8% settlement	“Set the fee rate at 1.8%”; 1% start, 1.8% settlement	Wrong settlement value despite matching remaining requirements
Requested amount and term; R0013; 263.43–267.55 s	“Five million, six months”; prediction contains both	Corrupted phrase corresponding to “five million user apply for him”; prediction retains 5M, omits six months	Missing term in a composite value; correct amount alone is insufficient

Appendix

Table 26: Traceable Kimi-K3 examples, translated into English for presentation. The first two use VibeVoice, the third FunASR. Times are intervals cited by the original predictions. Each original trajectory matches and its B2 counterpart fails. Actual input errors are preserved rather than replaced with scripted text. M denotes million CNY.

System	50%	80%	90%	No time	Bidirectional	ID permutation
FunASR + Astra	21.74	21.35	20.59	23.43	1.24	21.77
FunASR + Kimi-K3	9.80	9.80	9.80	10.27	0.00	11.65
FunASR + Qwen	5.95	5.75	5.75	5.95	0.00	5.75
VibeVoice + Kimi-K3	11.09	11.09	11.09	12.14	3.99	11.09
VibeVoice + Qwen	5.73	5.73	5.73	5.73	0.21	5.73

Appendix

Table 27: DCIST (%) under frozen-prediction diagnostics. The first three thresholds divide overlap by predicted duration. Bidirectional coverage additionally requires 50% of the gold window at the 80% threshold. ID permutation is globally consistent per recording. Changes are tested separately, retaining semantic and extra-node judgments; 80% is the official score.

System	Valid windows	At most 1 s	Median (s)	Matched nodes	Median ratio (%)
FunASR + Astra	732	3	6.73	118	25.36
FunASR + Kimi-K3	483	2	5.10	46	15.54
FunASR + Qwen	476	1	5.06	23	21.54
VibeVoice + Kimi-K3	493	0	10.20	50	48.56
VibeVoice + Qwen	489	1	10.04	23	29.51

Appendix

Table 28: Predicted evidence-window distribution. The duration ratio is predicted/gold window length (%) for required nodes in fully matched trajectories. Valid windows have non-negative starts and positive duration.

Frontend	Audio	Tracks	Retained	Lost	Gained	Neither	Lost with absent anchor
FunASR	26	308	13	26	13	256	26/26
VibeVoice	22	255	12	16	7	220	13/16

Appendix

Table 29: Paired Kimi-K3 trajectory outcomes for the same-route baseline and B2. Lost means baseline matched and B2 did not; gained is the converse. The final column counts lost trajectories with at least one linked normalized anchor form present before but absent after noise, not proven single-word causes.

Error type	Criterion	With speaker labels	Without speaker labels
Action error	start / advance / hold / settle differs from GT	39 (22.29%)	45 (21.63%)
Content error	A number, term, state, or other content in value differs from GT	94 (53.71%)	100 (48.08%)
Speaker error	speaker_id differs from GT	38 (21.71%)	72 (34.62%)
Scope error	scope_id differs from GT	70 (40.00%)	92 (44.23%)
Settlement-result error	result is missing, extra, or has an incorrect value	18 (10.29%)	28 (13.46%)
Missing node	A required GT event node is not recovered	33 (18.86%)	42 (20.19%)

Appendix

Table 30: Astra’s trajectory error distributions under the two reference-script conditions. Entries are the number (percentage) of trajectories involving each error, among 175 unmatched trajectories with speaker labels and 208 without speaker labels.

State Tracking

State tracking studies how AI systems maintain and update information about entities, tasks, or environments across sequential observations and actions.

See all 99 papers

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue
Aug 27, 2026Yen-Ju Lu, Yuzhe Wang, Yaohan Guan +8Audio-Language Model Evaluation Audio-Language Models
Center for Language and Speech Processing, Johns Hopkins University
MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue
Oct 14, 2026Puneet Mathur, Dinesh ManochaTurn-Taking in Dialogue Full-Duplex Spoken Dialogue Systems
University of Maryland College Park, USA
TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Aug 25, 2026Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16Spoken Dialogue Systems Multi-Turn Dialogue Evaluation
Sesame AI · Mundo AI · Oto +4

Aug 27, 2026J/K move · Enter open · S save

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He+7

Center for Language and Speech Processing, Johns Hopkins University

Audio-Language Model Evaluation Audio-Language Models Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.

Abstract

State Tracking

Explore similar work

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue