Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

Xinyi Liu; Hooshang Nayyeri; Dilek Hakkani-Tur; Emine Yilmaz; Joo-Kyung Kim; Yifei Zhang; Charith Peris; Hari Thadakamalla

cs.CLAug 20, 2026

Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

Authors: Xinyi Liu, Hooshang Nayyeri, Dilek Hakkani-Tur, Emine Yilmaz, Joo-Kyung Kim, Yifei Zhang, Charith Peris, Hari Thadakamalla

Organizations: Amazon · University of Illinois Urbana-Champaign · University College London

Abstract

Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions. We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes. For each scenario, we keep the task and user needs fixed while varying whether the same concern is conveyed explicitly in words or primarily through prosody, and evaluate decisions under transcript, audio, and concern-state access. Using Hear2Act, we evaluate two audio-capable LLMs. Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15.3%. In contrast, when models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state. This contrast, however, largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance. Together, these results show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do not reliably carry it into action without an explicit intermediate representation.

Figures & tables

Figure 1: Hear2Act overview. (1) Each scenario combines a surface request, prioritized hidden concerns, and candidates with verifiable satisfaction signatures. (2) The assistant interacts under different feedback and access conditions. (3) Matched rollouts are compared on task outcomes and interaction behavior.

Benchmark	P	C	M	N	O
MultiWOZ, SGD ( Budzianowski et al., 2018 ; Rastogi et al., 2020 )	$\times$	$\times$	✓	$\triangle$	✓
SpokenWOZ ( Si et al., 2023 )	✓	$\times$	✓	$\triangle$	✓
StyleTalk, ParaS2S ( Lin et al., 2024 ; Yang et al., 2026 )	✓	✓	$\times$	$\times$	$\times$
MULTI-Bench, HumDial-EIBench ( Deng et al., 2025 ; Wang et al., 2026b )	✓	$\times$	✓	$\times$	$\times$
Hear2Act (ours)	✓	✓	✓	✓	✓

Table 1: Benchmark positioning. P: prosodic input, C: matched control of prosodic access with fixed lexical content, M: multi-turn task decisions, N: task-grounded hidden user need, and O: verifiable trajectory and outcome.

\triangle

marks structured user goals conveyed lexically rather than hidden needs.

Figure 2: Illustrative Hear2Act trajectory under Prosody-mediated feedback. Three representative candidates are shown from the full 11-candidate set. Transcript-only access may confirm prematurely, while ground-truth concern-state access supports further elicitation and selection of the best-fitting option.

Benchmark artifact
SGD-seeded domains	48
Scenarios per domain	10
Benchmark scenarios	480
Candidate options per scenario	11
Hidden concern layers per scenario	3
Feedback realizations	2

Table 2: Hear2Act benchmark and rollout coverage. The 480 model-independent scenarios expand to 54,240 evaluation rollouts across models, access conditions, renderers, and interventions.

	Qwen2.5-Omni			Qwen2-Audio			Average
Input / representation	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$
Direct input
Transcript only ( $T$ )	15.4	22.9	12.3	13.7	20.1	12.0	14.6	21.5	12.2
Audio only ( $A$ )	17.3	22.7	13.3	14.7	22.4	12.2	16.0	22.6	12.8
Audio + transcript ( $A+T$ )	15.9	24.0	14.6	14.7	23.3	13.3	15.3	23.7	14.0
Textualized prosodic representations

Table 3: Spoken-assistant results under Prosody-mediated feedback with Qwen3-TTS.

T

A

S

, and

\hat{S}

denote transcript, audio, ground-truth state, and audio-inferred state; audio-derived representations are textualized and paired with the transcript. Average is computed across the two audio-capable LLMs. Bold/underline indicate the best/second-best value per column. See Table 4 for Explicit-lexical results and Appendix B for VoxCPM2.

	Qwen2.5-Omni			Qwen2-Audio			Average
Input / representation	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$
Direct input
Transcript only ( $T$ )	56.2	45.3	26.9	48.3	42.1	26.3	52.3	43.7	26.6
Audio only ( $A$ )	55.3	45.7	26.4	48.0	41.3	26.0	51.7	43.5	26.2
Audio + transcript ( $A+T$ )	52.2	45.4	26.5	47.6	41.0	25.9	49.9	43.2	26.2
Textualized prosodic representations

Table 4: Spoken-assistant use of concern information under Explicit lexical feedback with Qwen3-TTS (Qwen2.5-Omni

n{=}480

, Qwen2-Audio

n{=}1{,}440

per condition). Notation follows Table 3 . Results are similar because the concern is explicit in the transcript. Bold/underline mark column-wise highest/next-highest values.

	Prosody-mediated feedback						Explicit lexical feedback
	1st% $\uparrow$		OptSat% $\uparrow$		SvcSat% $\uparrow$		1st% $\uparrow$		OptSat% $\uparrow$		SvcSat% $\uparrow$
Model	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$	$T$	$T{+}S$
Claude Opus 4.6	49.7	79.1	37.0	41.1	24.7	26.2	74.9	81.9	44.6	43.0	25.9	26.4
Kimi K2.5	27.9	69.2	27.7	36.1	14.4	19.6	69.4	74.9	37.1	36.2	17.7	17.7
GLM-5	22.4	60.1	26.1	33.4	13.0	15.8	67.3	73.5	36.8	35.7	15.5	16.0
Qwen3-32B	21.1	53.1	31.0	35.5	10.4	13.7	64.3	64.8	43.1	37.7	12.9	12.6

Table 5: Effect of ground-truth concern-state access on text LLMs. Results use 480 matched scenarios with two runs each (

n{=}960

per cell).

T

is transcript only;

T{+}S

adds turn-level ground-truth state tags. Bold marks the larger value in each pair; action composition appears in Figure 3 .

Figure 3: Change in assistant action composition with concern-state access. Points show the

T{+}S{-}T

change, in percentage points, in the share of recommend , ask , and clarify decision turns under Prosody-mediated (red circles) and Explicit lexical (blue triangles) feedback; means are macro-averaged across models.

Figure 4: Outcome tracks label fidelity, not asking frequency. Optimal-solution rate on the 48-scenario intervention subset under Prosody-mediated feedback, pooled over five text LLMs. All label conditions have similar ask shares (

44

–

45\%

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Service family	#	Domains
Travel & Events	5	Budget airline tickets, vacation rental properties, restaurant reservations, concert ticket purchases, and wedding venue selection.
Home & Property	8	Home cleaning services, home renovation contractors, home security systems, landscaping contractors, lawn care contractors, solar panel installation, kitchen appliance upgrades, and mattress replacement.
Finance & Insurance	9	Credit card applications, mortgage lender comparison, investment portfolio allocation, retirement planning advisors, tax preparation services, car insurance policies, health insurance plans, pet insurance policies, and business insurance coverage.
Health & Wellness	7	Dermatologist appointments, pediatrician selection, mental health therapists, meditation apps, gym membership options, fitness tracker devices, and prescription eyeglasses.
Education & Career	5	College major selection, online coding bootcamps, language learning platforms, professional development courses, and laptop purchase for students.
Family & Lifestyle	6	Children’s daycare centers, dog training classes, online dating platforms, wine club memberships, meal delivery subscriptions, and video game purchases.

Appendix

Table 6: Domain inventory. Hear2Act covers 48 consumer-service domains grouped into eight families. Each domain is expanded into ten scenario seeds, yielding 480 benchmark scenarios.

	Subdomain situation (who / pressure)	Opening request
1	Last-minute emergency travel for family medical situation with extremely limited budget	“I need to fly to see my sick grandmother tomorrow but only have $200—what are my cheapest options?”
2	College student planning spring break trip with friends on tight budget	“Can you help me find the cheapest flights for four college students going to Miami for spring break?”
3	Budget-conscious family of five planning annual vacation	“What’s the most affordable way to fly my family of five to Orlando for our Disney World trip?”
4	Digital nomad seeking flexible travel dates for extended European backpacking	“I want to backpack through Europe for 3 months—which budget airlines offer the best multi-city deals?”
5	Job interview candidate needing quick affordable travel for an unexpected opportunity	“I have a job interview in Seattle next week and need the cheapest flight possible from Chicago.”
6	Retiree on fixed income wanting to visit grandchildren regularly	“As a senior on a fixed income, what budget airline options exist for regular visits to see my grandkids?”

Appendix

Table 7: One expanded domain. Ten subdomain seeds for budget airline tickets illustrate variation in user situation and decision pressure before scenario instantiation.

	Prosody-mediated feedback			Explicit lexical feedback
Input / representation	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$	1st % $\uparrow$	OptSat % $\uparrow$	SvcSat % $\uparrow$
Direct input
Transcript only ( $T$ )	12.5	22.5	11.9	53.2	44.4	28.8
Audio only ( $A$ )	16.1	22.6	14.1	53.9	45.4	28.2
Audio + transcript ( $A+T$ )	16.9	25.6	14.8	48.4	44.9	28.3
Textualized prosodic representations

Appendix

Table 8: Full spoken-assistant condition grid for Qwen2.5-Omni with VoxCPM2 (

n{=}480

per condition).

T

A

S

, and

\hat{S}

denote transcript, audio, ground-truth concern state, and audio-inferred concern state. Audio-derived representations are supplied as text alongside the transcript. Bold/underline mark the highest/next-highest value per column.

Figure 5: Concern-state access increases hidden-concern disclosure. Bars show mean revealed concerns, dots show per-model means, and in-bar percentages show rollouts revealing all three (

T

: transcript only;

T{+}S

: transcript plus turn-level ground-truth concern-state tags). The effect is concentrated under Prosody-mediated feedback, where concerns remain lexically implicit.

Contrast	Prosody-mediated	Explicit lexical
Text LLMs, pooled over five models ( $\Delta=\text{+State}-\text{Base}$ )
1st%	+36.7 [+34.3, +38.9]	+6.4 [+4.7, +8.1]
OptSat%	+7.3 [+5.8, +8.7]	$-$ 1.6 [ $-$ 2.6, $-$ 0.8]
SvcSat%	+4.1 [+2.8, +5.4]	+0.3 [ $-$ 0.3, +1.0]
Qwen2.5-Omni-7B, 1st% diagnostic contrasts
Ground-truth state on transcript ( $(T+S)-T$ )	+29.0 [+23.6, +34.4]	$-$ 6.5 [ $-$ 12.9, +0.0]

Appendix

Table 9: Scenario-level paired bootstrap 95% CIs for key diagnostic contrasts (2,000 joint scenario resamples). Point estimates correspond to Tables 5 , 3 , and 4 , and Figure 4 .

T

A

S

, and

\hat{S}

denote transcript, audio, ground-truth concern state, and audio-inferred textual state, respectively. The label-fidelity block uses 48 intervention scenarios.

Concern status	Qwen3-TTS	VoxCPM2
Concern resolved ( $O^{+}$ )	0.93	0.90
Concern unresolved ( $O^{-}$ )	0.91	0.93
Overall	0.92	0.92
$\kappa$ (annotators)	0.96	0.90

Appendix

Table 10: Speech-rendering validation. Human listeners recover resolved versus unresolved concern status from both renderers with

0.92

accuracy (

n{=}100

per renderer; balanced classes), confirming that the intended prosodic contrast remains perceptible after rendering.

\kappa

denotes inter-annotator agreement.

	Qwen3-TTS		VoxCPM2
	Human	Model	Human	Model
Resolved ( $O^{+}$ )	0.93	0.86	0.90	0.88
Unresolved ( $O^{-}$ )	0.91	0.84	0.93	0.64
Overall	0.92	0.85	0.92	0.76
$\kappa$ , annotators	0.96		0.90
Agreement, model	0.87		0.78

Appendix

Table 11: Qwen2.5-Omni-7B as a concern-cue reader : accuracy against the intended concern status on the audited clips (100 per renderer, balanced 50/50; protocol of Section 4.4 ). Human values average the two annotators. Bottom block: inter-annotator

\kappa

; raw model–annotator agreement; model–annotator

\kappa

(all averaged over the two annotators).

Concern state	Delivery labels
Resolved: genuine acceptance	satisfied, warm, enthusiastic, relieved
Unresolved: reluctant acceptance	underwhelmed, lukewarm, hesitant, flat
Unresolved: voiced concern	concerned
Unresolved: rejection	frustrated, disappointed, impatient, firm

Appendix

Table 12: State-to-delivery mapping. Each concern state is mapped to graded delivery labels used for speech realization.

Explore similar work

Sep 1, 2026cs.AI

VoiceLongMemEval: Do Assistants Remember How You Sounded?

With the growing scale of multi-agent architectures and large language models, deployed AI assistants are increasingly tasked with reasoning over long, continuous, multi-session conversation histories. Current benchmarks evaluate this dialogue history as information retrieval over long horizon, temporal reasoning, or knowledge updates, while crucially ignoring the fundamental dynamics of human-agent interaction, i.e. how they said it. To address this gap, we present VoiceLongMemEval (VLME) benchmark, where every answer depends on paralinguistic metadata (emotion labels, prosody descriptors, and voice events) attached to conversational turns, which is otherwise unrecoverable from the words alone. Every item passes a three-stage adversarial gate, ensuring that a strong language model fails when given only the transcript. Evaluating leading frontier and open-weight models reveals a pervasive affect gap; providing text-track paralinguistic metadata yields a 0.09 to 0.38 accuracy boost (0.61 to 0.69 when prompted with evidence hints), while standard ASR pipelines systematically discard this signal. Additionally, audio-native models successfully extract these cues directly from speech (0.354 to 0.412 vs. 0.325 blind). Code and dataset will be made available upon acceptance.

Ramit Pahwa, Parivesh Priye, Apoorva Beedu

Rivian VW Tech

Audio-Language Model Evaluation Long-Context Language Model Evaluation Audio Reasoning Long-Term Conversational Memory

Ask PDF

Oct 8, 2026cs.CL

Prosody-to-Text: Predicting text from low-pass filtered speech

While predicting prosody from text is an established task in the field, the opposite direction, predicting text that fits a given prosodic pattern, remains largely overlooked. We find this unfortunate, because this opposite direction could lead to some very interesting use cases. Therefore, in this paper, we make the first steps in the prosody-to-text direction by inves- tigating how much of the original sentence can be recovered from its prosodic pattern. To this end, we fine-tune the Whis- per model using only the 12 lowest Mel bins (low-pass filter with approximately 450Hz cutoff), and obtain surprisingly accurate results (WER 36%), with 10% of utterances be- ing recovered perfectly, and 40% of utterances having Word Error Rate at or below 25%. We also find that, given the correct prefix, the next token was predicted correctly in 79% of cases. Our results suggest that the relationship between low-frequency speech features and lexical content is much stronger than previously thought, and we believe that direct- ing more attention to this topic might open the door to new applications, such as using prosody to guide text generation of modern LLMs

David Porteš, Aleš Horák

Natural Language Processing Centre Faculty of Informatics, Masaryk University Botanicka 68a, 602 00 Brno, Czech Republic

Speech Prosody Automatic Speech Recognition

Ask PDF

Jun 30, 2026cs.CL

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because

F_0

, speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours of dyadic English conversation from the Seamless Interaction dataset, we construct matched reference regimes for

F_0

mean,

F_0

expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. We then define a percentile-based evaluation protocol: extract the same metrics from an S2S output waveform, compare them to the closest matched human reference stratum, and report percentile deviations or 5th-95th percentile out-of-regime flags. On held-out human rows, pooled references over-flag state-conditioned

F_0

expressivity and rhythm, while matched references return flag rates closer to the nominal 10% and make deviation direction interpretable. These outputs serve as behavioral plausibility checks that complement, rather than replace, perceptual and user-centered evaluation.

Ashish Hallur, Thomas Thebaud, Georgi Tinchev +2

Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · Amazon Inc., Seattle, WA, USA

Voice Agent Evaluation Speech Generation Evaluation Spoken Dialogue Systems Speech Prosody

Ask PDF