A Shaky Voice Is Not Always a Dodge: Benchmarking Textual and Vocal Evasion Detection in Earnings Calls

Mirae Kim; Seonghun Jeong; Youngjun Kwak

cs.CLAug 28, 2026

A Shaky Voice Is Not Always a Dodge: Benchmarking Textual and Vocal Evasion Detection in Earnings Calls

Authors: Mirae Kim, Seonghun Jeong, Youngjun Kwak

Organizations: Financial Tech Lab, KakaoBank Corp. · Yonsei University

Abstract

Existing approaches to evasion detection in earnings calls focus on textual transcripts, treating evasion as a single-dimensional phenomenon. We argue that evasion in spoken communication is inherently multidimensional: beyond what executives say, how they say it carries independent and complementary information. To study these dimensions jointly, we introduce DualEvasion, a benchmark for evasion detection across text and audio in earnings call Q&A. The benchmark contains 505 annotated question-answer pairs from 60 earnings calls, each with two independent labels: textual evasion (direct vs. evasive) and vocal cues operationalized as speaker confidence (confident vs. unconfident). Our experiments show that state-of-the-art multimodal models struggle to detect vocal confidence, particularly on unconfident responses. Our analysis suggests these models interpret acoustic cues in isolation rather than relative to each speaker's baseline. Providing speaker-level references yields modest improvements, but a substantial gap with human performance remains.

Figures & tables

Figure 1: The challenge of DualEvasion : an LLM detects textual evasion but misses vocal unconfidence in the same response.

	SubjECTive-QA	Nuaimi	EvasionBench	DualEvasion
Period	’07–’21	’19–’22	Multi-yr	’23–’25
Audio	✗	✗	✗	✓
Evasion	$\triangle$	✓	✓	✓
Label	Text	Text	Text	Text+Vocal
Annotation	Human	H+LLM	LLM	Domain Expert

Table 1: Comparison with earnings call Q&A datasets Pardawala et al. (2024) ; Nuaimi et al. (2025) ; Ma et al. (2026) . Existing datasets label evasion from text only; ours additionally captures vocal cues annotated by finance professionals.

\triangle

= partial.

Textual Evasion: Direct vs. Evasive
Model	Direct/Confident			Evasive/Unconfident			Overall
Model	Precision	Recall	F1	Precision	Recall	F1	Macro F1	Accuracy
Qwen2-7B-Instruct	79.3	91.6	85.0	60.8	35.3	44.7	64.8	76.4
Llama-3.1-8B-IT	84.9	80.8	82.8	53.9	61.0	57.2	70.0	75.4
Gemini-2.5-Flash	93.2	74.8	83.0	55.5	85.3	67.2	75.1	77.6
Gemini-3-Flash	88.7	97.6	92.9	90.9	66.2	76.6	84.7	89.1

Table 2: Zero-shot performance on textual evasion and vocal confidence detection. Results for commercial audio models are averaged over three runs, while all other results are from a single run. Bold indicates best, underline indicates second best per column within each section.

Text	Vocal	Count	%
Direct	Confident	302	59.8
Direct	Unconfident	67	13.3
Evasive	Confident	95	18.8
Evasive	Unconfident	41	8.1
Cross-modal disagreement		162	32.1

Table 3: Joint distribution of textual evasion and vocal confidence labels. Shaded rows indicate cross-modal disagreement.

	Text	Vocal
LLM–Annotator	0.813	0.225
Annotator–Annotator	0.866	0.774

Table 4: Cohen’s

\kappa

agreement with human annotators, averaged over two annotators, and human–human agreement. GPT-5 is used for text and Gemini-3-Flash for vocal confidence.

Figure 2: Correlation between acoustic features and unconfident predictions for Gemini-2.5-Flash, the model with the largest gap, measured using absolute values and speaker-relative deviations (response value minus the speaker’s mean).

	$\|r_{\mathrm{raw}}\|$	$\|r_{\mathrm{cent}}\|$	Gap ( $p$ )
Models
Gemini-2.5-Flash	.195	.032	.164 (.005)
Gemini-3-Flash	.179	.045	.135 (.003)
GPT-Audio	.142	.030	.113 (.025)
Gemini-2.5-Flash-lite	.093	.013	.080 (.234)
Audio-Flamingo-3	.040	.015	.025 (.522)

Table 5: Mean absolute point-biserial correlation between acoustic features and the unconfident label, before (

|r_{\mathrm{raw}}|

) and after (

|r_{\mathrm{cent}}|

) speaker-centering. Gap

=|r_{\mathrm{raw}}|-|r_{\mathrm{cent}}|

;

p

-values from a cluster bootstrap over the 60 calls. Bold indicates gaps significant at

p<.05

Model	Baseline	+Speaker Norm.	$\Delta$ F1
Gemini-2.5-Flash	38.5	43.2	+4.7
Gemini-3-Flash	38.7	41.6	+2.9
GPT-Audio	38.2	39.2	+1.0

Table 6: Effect of speaker-level normalization on unconfident F1. The model infers the speaker’s vocal baseline before classifying the target response.

Figure 3: $R^{2}$ of text-only, audio-only, and combined regression models for post-earnings stock volatility across Q&A coverage thresholds. Coverage is the fraction of each call’s Q&A pairs included in DualEvasion ; at each threshold, only calls with coverage at or above that level are retained. Annotations indicate the number of retained calls ( $n$ ); $\dagger$ : $p<0.10$ .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Composition
Earnings calls	60
Q&A pairs	505
Unique tickers	49
Call period	2023–2025
Market cap range	$56M–$ 189B
Market Group

Appendix

Table 7: Dataset composition.

	Med.	Min	Max
Response duration (sec)	66.9	0.2	296.9
Response length (tokens)	186	2	918
Q&A pairs per call	10	4	10
Total audio duration			10.9 hrs

Appendix

Table 8: Response duration and length statistics.

	A1	A2	A3	A4	A5
A1	–	.915	.706	.651	.917
A2		–	.623	.651	.834
A3			–	.442	.713
A4				–	.662
A5					–

Appendix

Table 9: Pairwise Cohen’s

\kappa

for vocal confidence labels on the validation subset (

N

= 52).

	Gem-2.5	Gem-3.0	GPT-Aud
Gem-2.5	–	.483	.309
Gem-3.0		–	.407
GPT-Aud			–

Appendix

Table 10: Inter-model agreement on vocal confidence labels measured by pairwise Cohen’s

\kappa

(

N

= 52). Gem-2.5 denotes Gemini-2.5-Flash, Gem-3.0 denotes Gemini-3-Flash, and GPT-Aud denotes GPT-Audio.

Figure 4: Annotation interface for vocal confidence.

Model	Thr.	Conf. F1	Unconf. F1	Macro F1
Audio-Flamingo-3	0.40 M	82.7	30.0	56.4
Audio-Flamingo-3	0.95 U	2.5	35.5	19.0
Qwen2-Audio	0.05 M	65.2	34.3	49.7
Qwen2-Audio	0.45 U	24.4	36.7	30.5

Appendix

Table 13: Threshold sweep for the two open models that expose logits. M : threshold maximizing macro F1; U : threshold maximizing unconfident F1. Bold marks the maximized metric.

Model	Setting	Unconf. F1		Macro F1
Qwen2-Audio	zero-shot	36.2		28.0
Qwen2-Audio	fine-tuned	32.4	(-3.8)	48.6	(+20.6)
Audio-Flamingo-3	zero-shot	30.8		51.7
Audio-Flamingo-3	fine-tuned	34.7	(+3.9)	53.5	(+1.8)
VibeVoice-ASR	fine-tuned	9.9		48.8

Appendix

Table 14: Supervised fine-tuning with five-fold cross-validation. Values in parentheses indicate changes from zero-shot performance. Fine-tuning improves overall performance but does not consistently improve unconfident detection.

Model	Methods	Confident			Unconfident			Macro-F1
Model	Methods	Precision	Recall	F1	Precision	Recall	F1	Macro-F1
Gemini-2.5-Flash	baseline	83.7	72.0	77.4	32.0	48.4	38.5	57.9
	fewshot( $n{=}1$ )	83.6	77.1	80.2	34.5	44.4	38.9	59.5
	fewshot( $n{=}2$ )	81.1	85.6	83.3	34.1	26.9	30.1	56.7
	fewshot( $n{=}3$ )	81.1	91.9	86.2	41.8	21.3	28.2	57.2
Gemini-3-Flash	baseline	84.0	67.8	75.0	30.7	52.5	38.7	56.8

Appendix

Table 15: Effect of few-shot same-speaker reference audio on vocal confidence detection. Bold indicates the best score and shaded indicates the lowest score within each model.

Panel of Table 16 — Table 16: Prompt for textual evasion classification.

Panel of Table 17 — Table 17: Prompt for vocal confidence classification (audio-only).

Panel of Table 18 — Table 18: Prompt for few-shot vocal confidence classification with same-speaker reference audio.

Panel of Table 19 — Table 19: Prompt for speaker-normalized vocal confidence classification with explicit speaker-baseline inference.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 772 papers

Lie to me: Detecting Managerial Evasiveness in Earnings Calls via Conversational Audio Encoders
Sep 12, 2026Huizhong Chen, Huan Zhang
Beijing Normal-Hong Kong Baptist University · Individual researcher
Earnings25: A Comprehensive 500-Hour Speech Benchmark for Finance
Jul 26, 2026Denglin Jiang, Haoran Zhou, Anshul Wadhawan +7ASR Evaluation Speech Recognition Benchmarks
Bloomberg, United States
The Acoustic Camouflage Phenomenon: Re-evaluating Speech Features for Financial Risk Prediction
Apr 16, 2026Dhruvin Dungrani, Disha DungraniSpeech Processing Multimodal Learning
Department of Information Systems, Independent Researchers

Sep 12, 2026J/K move · Enter open · S save

Lie to me: Detecting Managerial Evasiveness in Earnings Calls via Conversational Audio Encoders

Huizhong Chen, Huan Zhang

Beijing Normal-Hong Kong Baptist University · Individual researcher

Open paper AskPDF arXiv

Earnings conference calls are a primary channel through which managers disclose information under analyst scrutiny. Prior work has linked vocal and lexical cues to future adverse outcomes, but often pools features over an entire call and underuses the interactive structure of Q&A. We propose a two-branch late-fusion framework for detecting managerial evasiveness as a predictor of extrinsic SEC events (primarily late filings): (i) an LLM-as-a-judge that maps Q&A text to an interpretable call-level vector X_text via a structured binary rubric, and (ii) a frozen conversational encoder whose temporal hidden states are read by a DeepVoice-style sequential reader to produce an audio representation h. Late fusion of (X_text, h) yields a call-level risk score p. On n=1,039 calls (212 late filings) with firm-grouped 5-fold CV, fusion reaches AUROC approx. 0.89, versus 0.55 for the text judge and 0.71 for duration alone. These results show that conversational audio dynamics encode managerial evasiveness beyond lexical content and call length, yielding a stronger early-warning signal of adverse SEC outcomes.

Abstract

Multimodal Model Evaluation

Explore similar work

Lie to me: Detecting Managerial Evasiveness in Earnings Calls via Conversational Audio Encoders

Earnings25: A Comprehensive 500-Hour Speech Benchmark for Finance

The Acoustic Camouflage Phenomenon: Re-evaluating Speech Features for Financial Risk Prediction