Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

Xinyi Liu; Hooshang Nayyeri; Dilek Hakkani-Tur; Emine Yilmaz; Joo-Kyung Kim; Yifei Zhang; Charith Peris; Hari Thadakamalla

cs.CLAug 20, 2026

Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

Authors: Xinyi Liu, Hooshang Nayyeri, Dilek Hakkani-Tur, Emine Yilmaz, Joo-Kyung Kim, Yifei Zhang, Charith Peris, Hari Thadakamalla

Organizations: Amazon · University of Illinois Urbana-Champaign · University College London

Abstract

Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions. We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes. For each scenario, we keep the task and user needs fixed while varying whether the same concern is conveyed explicitly in words or primarily through prosody, and evaluate decisions under transcript, audio, and concern-state access. Using Hear2Act, we evaluate two audio-capable LLMs. Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15.3%. In contrast, when models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state. This contrast, however, largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance. Together, these results show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do not reliably carry it into action without an explicit intermediate representation.

Figures & tables

Figure 1: Hear2Act overview. (1) Each scenario combines a surface request, prioritized hidden concerns, and candidates with verifiable satisfaction signatures. (2) The assistant interacts under different feedback and access conditions. (3) Matched rollouts are compared on task outcomes and interaction behavior.

Benchmark	P	C	M	N	O
MultiWOZ, SGD ( Budzianowski et al., 2018 ; Rastogi et al., 2020 )	$\times$	$\times$	✓	$\triangle$	✓
SpokenWOZ ( Si et al., 2023 )	✓	$\times$	✓	$\triangle$	✓
StyleTalk, ParaS2S ( Lin et al., 2024 ; Yang et al., 2026 )	✓	✓	$\times$	$\times$	$\times$
MULTI-Bench, HumDial-EIBench ( Deng et al., 2025 ; Wang et al., 2026b )	✓	$\times$	✓	$\times$	$\times$
Hear2Act (ours)	✓	✓	✓	✓	✓

Table 1: Benchmark positioning. P: prosodic input, C: matched control of prosodic access with fixed lexical content, M: multi-turn task decisions, N: task-grounded hidden user need, and O: verifiable trajectory and outcome.

\triangle

marks structured user goals conveyed lexically rather than hidden needs.

Figure 2: Illustrative Hear2Act trajectory under Prosody-mediated feedback. Three representative candidates are shown from the full 11-candidate set. Transcript-only access may confirm prematurely, while ground-truth concern-state access supports further elicitation and selection of the best-fitting option.

Benchmark artifact
SGD-seeded domains	48
Scenarios per domain	10
Benchmark scenarios	480
Candidate options per scenario	11
Hidden concern layers per scenario	3
Feedback realizations	2

Table 2: Hear2Act benchmark and rollout coverage. The 480 model-independent scenarios expand to 54,240 evaluation rollouts across models, access conditions, renderers, and interventions.

	Qwen2.5-Omni			Qwen2-Audio			Average
Input / representation	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$
Direct input
Transcript only ( $T$ )	15.4	22.9	12.3	13.7	20.1	12.0	14.6	21.5	12.2
Audio only ( $A$ )	17.3	22.7	13.3	14.7	22.4	12.2	16.0	22.6	12.8
Audio + transcript ( $A+T$ )	15.9	24.0	14.6	14.7	23.3	13.3	15.3	23.7	14.0
Textualized prosodic representations

Table 3: Spoken-assistant results under Prosody-mediated feedback with Qwen3-TTS.

T

A

S

, and

\hat{S}

denote transcript, audio, ground-truth state, and audio-inferred state; audio-derived representations are textualized and paired with the transcript. Average is computed across the two audio-capable LLMs. Bold/underline indicate the best/second-best value per column. See Table 4 for Explicit-lexical results and Appendix B for VoxCPM2.

	Qwen2.5-Omni			Qwen2-Audio			Average
Input / representation	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$
Direct input
Transcript only ( $T$ )	56.2	45.3	26.9	48.3	42.1	26.3	52.3	43.7	26.6
Audio only ( $A$ )	55.3	45.7	26.4	48.0	41.3	26.0	51.7	43.5	26.2
Audio + transcript ( $A+T$ )	52.2	45.4	26.5	47.6	41.0	25.9	49.9	43.2	26.2
Textualized prosodic representations

Table 4: Spoken-assistant use of concern information under Explicit lexical feedback with Qwen3-TTS (Qwen2.5-Omni

n{=}480

, Qwen2-Audio

n{=}1{,}440

per condition). Notation follows Table 3 . Results are similar because the concern is explicit in the transcript. Bold/underline mark column-wise highest/next-highest values.

	Prosody-mediated feedback						Explicit lexical feedback
	1st% $\uparrow$		OptSat% $\uparrow$		SvcSat% $\uparrow$		1st% $\uparrow$		OptSat% $\uparrow$		SvcSat% $\uparrow$
Model	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$
Claude Opus 4.6	49.7	79.1	37.0	41.1	24.7	26.2	74.9	81.9	44.6	43.0	25.9	26.4
Kimi K2.5	27.9	69.2	27.7	36.1	14.4	19.6	69.4	74.9	37.1	36.2	17.7	17.7
GLM-5	22.4	60.1	26.1	33.4	13.0	15.8	67.3	73.5	36.8	35.7	15.5	16.0
Qwen3-32B	21.1	53.1	31.0	35.5	10.4	13.7	64.3	64.8	43.1	37.7	12.9	12.6

Table 5: Effect of ground-truth concern-state access on text LLMs. Results use 480 matched scenarios with two runs each (

n{=}960

per cell).

T

is transcript only;

T{+}S

adds turn-level ground-truth state tags. Bold marks the larger value in each pair; action composition appears in Figure 3 .

Figure 3: Change in assistant action composition with concern-state access. Points show the

T{+}S{-}T

change, in percentage points, in the share of recommend , ask , and clarify decision turns under Prosody-mediated (red circles) and Explicit lexical (blue triangles) feedback; means are macro-averaged across models.

Figure 4: Outcome tracks label fidelity, not asking frequency. Optimal-solution rate on the 48-scenario intervention subset under Prosody-mediated feedback, pooled over five text LLMs. All label conditions have similar ask shares (

44

–

45\%

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Service family	#	Domains
Travel & Events	5	Budget airline tickets, vacation rental properties, restaurant reservations, concert ticket purchases, and wedding venue selection.
Home & Property	8	Home cleaning services, home renovation contractors, home security systems, landscaping contractors, lawn care contractors, solar panel installation, kitchen appliance upgrades, and mattress replacement.
Finance & Insurance	9	Credit card applications, mortgage lender comparison, investment portfolio allocation, retirement planning advisors, tax preparation services, car insurance policies, health insurance plans, pet insurance policies, and business insurance coverage.
Health & Wellness	7	Dermatologist appointments, pediatrician selection, mental health therapists, meditation apps, gym membership options, fitness tracker devices, and prescription eyeglasses.
Education & Career	5	College major selection, online coding bootcamps, language learning platforms, professional development courses, and laptop purchase for students.
Family & Lifestyle	6	Children’s daycare centers, dog training classes, online dating platforms, wine club memberships, meal delivery subscriptions, and video game purchases.

Appendix

Table 6: Domain inventory. Hear2Act covers 48 consumer-service domains grouped into eight families. Each domain is expanded into ten scenario seeds, yielding 480 benchmark scenarios.

	Subdomain situation (who / pressure)	Opening request
1	Last-minute emergency travel for family medical situation with extremely limited budget	“I need to fly to see my sick grandmother tomorrow but only have $200—what are my cheapest options?”
2	College student planning spring break trip with friends on tight budget	“Can you help me find the cheapest flights for four college students going to Miami for spring break?”
3	Budget-conscious family of five planning annual vacation	“What’s the most affordable way to fly my family of five to Orlando for our Disney World trip?”
4	Digital nomad seeking flexible travel dates for extended European backpacking	“I want to backpack through Europe for 3 months—which budget airlines offer the best multi-city deals?”
5	Job interview candidate needing quick affordable travel for an unexpected opportunity	“I have a job interview in Seattle next week and need the cheapest flight possible from Chicago.”
6	Retiree on fixed income wanting to visit grandchildren regularly	“As a senior on a fixed income, what budget airline options exist for regular visits to see my grandkids?”

Appendix

Table 7: One expanded domain. Ten subdomain seeds for budget airline tickets illustrate variation in user situation and decision pressure before scenario instantiation.

	Prosody-mediated feedback			Explicit lexical feedback
Input / representation	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$
Direct input
Transcript only ( $T$ )	12.5	22.5	11.9	53.2	44.4	28.8
Audio only ( $A$ )	16.1	22.6	14.1	53.9	45.4	28.2
Audio + transcript ( $A+T$ )	16.9	25.6	14.8	48.4	44.9	28.3
Textualized prosodic representations

Appendix

Table 8: Full spoken-assistant condition grid for Qwen2.5-Omni with VoxCPM2 (

n{=}480

per condition).

T

A

S

, and

\hat{S}

denote transcript, audio, ground-truth concern state, and audio-inferred concern state. Audio-derived representations are supplied as text alongside the transcript. Bold/underline mark the highest/next-highest value per column.

Figure 5: Concern-state access increases hidden-concern disclosure. Bars show mean revealed concerns, dots show per-model means, and in-bar percentages show rollouts revealing all three (

T

: transcript only;

T{+}S

: transcript plus turn-level ground-truth concern-state tags). The effect is concentrated under Prosody-mediated feedback, where concerns remain lexically implicit.

Contrast	Prosody-mediated	Explicit lexical
Text LLMs, pooled over five models ( $\Delta=\text{+State}-\text{Base}$ )
1st%	+36.7 [+34.3, +38.9]	+6.4 [+4.7, +8.1]
OptSat%	+7.3 [+5.8, +8.7]	$-$ 1.6 [ $-$ 2.6, $-$ 0.8]
SvcSat%	+4.1 [+2.8, +5.4]	+0.3 [ $-$ 0.3, +1.0]
Qwen2.5-Omni-7B, 1st% diagnostic contrasts
Ground-truth state on transcript ( $(T+S)-T$ )	+29.0 [+23.6, +34.4]	$-$ 6.5 [ $-$ 12.9, +0.0]

Appendix

Table 9: Scenario-level paired bootstrap 95% CIs for key diagnostic contrasts (2,000 joint scenario resamples). Point estimates correspond to Tables 5 , 3 , and 4 , and Figure 4 .

T

A

S

, and

\hat{S}

denote transcript, audio, ground-truth concern state, and audio-inferred textual state, respectively. The label-fidelity block uses 48 intervention scenarios.

Concern status	Qwen3-TTS	VoxCPM2
Concern resolved ( $O^{+}$ )	0.93	0.90
Concern unresolved ( $O^{-}$ )	0.91	0.93
Overall	0.92	0.92
$\kappa$ (annotators)	0.96	0.90

Appendix

Table 10: Speech-rendering validation. Human listeners recover resolved versus unresolved concern status from both renderers with

0.92

accuracy (

n{=}100

per renderer; balanced classes), confirming that the intended prosodic contrast remains perceptible after rendering.

\kappa

denotes inter-annotator agreement.

	Qwen3-TTS		VoxCPM2
	Human	Model	Human	Model
Resolved ( $O^{+}$ )	0.93	0.86	0.90	0.88
Unresolved ( $O^{-}$ )	0.91	0.84	0.93	0.64
Overall	0.92	0.85	0.92	0.76
$\kappa$ , annotators	0.96		0.90
Agreement, model	0.87		0.78

Appendix

Table 11: Qwen2.5-Omni-7B as a concern-cue reader : accuracy against the intended concern status on the audited clips (100 per renderer, balanced 50/50; protocol of Section 4.4 ). Human values average the two annotators. Bottom block: inter-annotator

\kappa

; raw model–annotator agreement; model–annotator

\kappa

(all averaged over the two annotators).

Concern state	Delivery labels
Resolved: genuine acceptance	satisfied, warm, enthusiastic, relieved
Unresolved: reluctant acceptance	underwhelmed, lukewarm, hesitant, flat
Unresolved: voiced concern	concerned
Unresolved: rejection	frustrated, disappointed, impatient, firm

Appendix

Table 12: State-to-delivery mapping. Each concern state is mapped to graded delivery labels used for speech realization.

Explore similar work

VoiceLongMemEval: Do Assistants Remember How You Sounded?
Sep 1, 2026Ramit Pahwa, Parivesh Priye, Apoorva BeeduAudio-Language Model Evaluation Long-Context Language Model Evaluation
Rivian VW Tech
Prosody-to-Text: Predicting text from low-pass filtered speech
Oct 8, 2026David Porteš, Aleš HorákSpeech Prosody Automatic Speech Recognition
Natural Language Processing Centre Faculty of Informatics, Masaryk University Botanicka 68a, 602 00 Brno, Czech Republic
Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems
Jun 30, 2026Ashish Hallur, Thomas Thebaud, Georgi Tinchev +2Voice Agent Evaluation Speech Generation Evaluation
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Amazon Inc., Seattle, WA, USA

Sep 1, 2026J/K move · Enter open · S save

VoiceLongMemEval: Do Assistants Remember How You Sounded?

Ramit Pahwa, Parivesh Priye, Apoorva Beedu

Rivian VW Tech

Audio-Language Model Evaluation Long-Context Language Model Evaluation Audio Reasoning Long-Term Conversational Memory

Open paper AskPDF arXiv

With the growing scale of multi-agent architectures and large language models, deployed AI assistants are increasingly tasked with reasoning over long, continuous, multi-session conversation histories. Current benchmarks evaluate this dialogue history as information retrieval over long horizon, temporal reasoning, or knowledge updates, while crucially ignoring the fundamental dynamics of human-agent interaction, i.e. how they said it. To address this gap, we present VoiceLongMemEval (VLME) benchmark, where every answer depends on paralinguistic metadata (emotion labels, prosody descriptors, and voice events) attached to conversational turns, which is otherwise unrecoverable from the words alone. Every item passes a three-stage adversarial gate, ensuring that a strong language model fails when given only the transcript. Evaluating leading frontier and open-weight models reveals a pervasive affect gap; providing text-track paralinguistic metadata yields a 0.09 to 0.38 accuracy boost (0.61 to 0.69 when prompted with evidence hints), while standard ASR pipelines systematically discard this signal. Additionally, audio-native models successfully extract these cues directly from speech (0.354 to 0.412 vs. 0.325 blind). Code and dataset will be made available upon acceptance.

Abstract

Explore similar work

VoiceLongMemEval: Do Assistants Remember How You Sounded?

Prosody-to-Text: Predicting text from low-pass filtered speech

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems