Same Text, Different Numbers: The Divergence of LLM-Based Measures

Hamid Boustanifar; Sasan Mansouri

cs.AISep 25, 2026

Same Text, Different Numbers: The Divergence of LLM-Based Measures

Authors: Hamid Boustanifar, Sasan Mansouri

Abstract

Researchers increasingly use generative large language models (LLMs) to convert corporate text into empirical variables. We examine the extent to which LLM-based textual measures are invariant to model choice using thirteen measures, including sentiment, management clarity, uncertainty, answer specificity, and climate and political risk. Seven LLMs from different providers score earnings call transcripts of S&P 500 companies on these constructs. Cross-model rank correlations average only 0.52, and transcript-level differences common across providers account for only 34% of total score variation. Cross-model disagreement does not predict subsequent analyst or market disagreement, consistent with a substantial model-specific component rather than common ambiguity in the underlying disclosure. Model choice significantly affects downstream inference, with coefficient magnitudes, signs, and statistical significance varying substantially across models. Averaging across providers makes transcript rankings more stable for most constructs, but score levels remain sensitive to the models included in the ensemble. LLM-generated variables should therefore be treated as model-contingent measurements and validated across providers.

Figures & tables

Figure 1: Inter-provider score correlations: highest- and lowest-agreement constructs. Pairwise correlations between the seven LLM providers’ scores (upper triangle: Pearson; lower triangle: Spearman) for the two constructs with the highest mean pairwise correlation—sentiment and innovation—and the two with the lowest—integrity and the specificity ratio. The last row and column report the correlation of each LLM measure with the corresponding traditional, non-LLM benchmark for that construct. The colour scale is identical to the correlation heatmaps for all constructs reported in the Online Appendix (Appendix A ).

LLM-related variables (call $\times$ construct level)
	N	Mean	Std	Min	P25	Median	P75	Max
LLM Dispersion (Z)	25,298	0.618	0.315	0.067	0.415	0.583	0.784	2.787
Justif. Cosine Sim	25,298	0.618	0.102	0.204	0.564	0.624	0.682	0.931
Justif. Jaccard Sim	25,298	0.183	0.061	0.010	0.150	0.169	0.193	0.584
Firm-level variables (call level)
ln(Assets)	1,945	10.348	1.281	7.288	9.498	10.246	11.102	15.253

Table 1: Summary Statistics

Construct	$N$	Min	Mean	Median	Max
Sentiment	1,946	0.77	0.85	0.86	0.90
Innovation	1,946	0.59	0.76	0.75	0.87
Uncertainty	1,946	0.54	0.64	0.62	0.75
Climate Risk	1,946	0.36	0.58	0.53	0.79
Specificity	1,946	0.34	0.54	0.51	0.70
Management Clarity	1,946	0.38	0.53	0.51	0.66

Table 2: Inter-Provider Agreement by Construct: Distribution of Pairwise Spearman Correlations

Construct	PC1 Variance Explained (%)	# PCs for 90%
Sentiment	84.95	3
Climate Risk	75.71	3
Innovation	74.28	4
Uncertainty	65.28	5
Political Risk	64.24	4
Specificity	58.89	5

Table 3: Cross-Model Commonality: PCA Summary

Construct	Transcript	Provider	Residual	$G$	$\Phi$	$N$
	(% of Total)	(% of Total)	(% of Total)
Sentiment	74.0%	9.9%	16.1%	0.970	0.952	13,622
Innovation	52.0%	24.6%	23.4%	0.940	0.884	13,622
Climate Risk	58.5%	10.1%	31.4%	0.929	0.908	13,622
Uncertainty	20.9%	62.3%	16.9%	0.896	0.648	13,622
Specificity	42.4%	15.0%	42.6%	0.875	0.838	13,622

Table 4: Variance Decomposition of LLM Scores: Provider Calibration vs. Transcript Ambiguity

Panel A: Firm Controls
	Textual Measure Dispersion (Std. Dev.)
	(1)	(2)	(3)	(4)
$\ln(\text{Assets})$	0.0067 ∗∗∗	0.0029	0.0030	0.0046 ∗∗
	(2.85)	(1.22)	(1.24)	(2.01)
Book-to-Market	0.0028	-0.0050	-0.0051	-0.0092
	(0.27)	(-0.43)	(-0.44)	(-0.94)

Table 5: Determinants of textual measure dispersion (standard deviation).

	Climate Risk	Corporate Culture	Innovation	Integrity	Management Clarity	Political Risk	Quality	Respect	Sentiment	Specificity	Specificity Ratio	Teamwork	Uncertainty
Mistral	0.2	0.4 ∗∗∗	0.2	0.3 ∗∗	0.2	-0.1	0.3 ∗∗	0.4 ∗∗	1.3 ∗∗∗	0.1	0.4 ∗∗∗	0.3 ∗∗	-0.5 ∗∗∗
	(1.27)	(2.66)	(1.25)	(2.05)	(1.01)	(-0.84)	(2.17)	(2.46)	(7.60)	(0.95)	(2.92)	(2.13)	(-2.92)
Gemini	0.0	0.5 ∗∗∗	0.3 ∗	0.3 ∗	0.2	-0.0	0.3 ∗	0.6 ∗∗∗	1.4 ∗∗∗	0.1	0.2	0.6 ∗∗∗	-0.4 ∗∗∗
	(0.22)	(2.68)	(1.83)	(1.76)	(0.99)	(-0.29)	(1.73)	(3.79)	(8.61)	(0.72)	(1.33)	(3.62)	(-2.85)
OpenAI	-0.1	0.5 ∗∗∗	0.3 ∗	0.4 ∗∗	0.1	0.2	0.4 ∗∗	0.4 ∗∗∗	1.1 ∗∗∗	0.1	0.4 ∗∗∗	0.4 ∗∗∗	-0.5 ∗∗∗
	(-0.46)	(3.11)	(1.80)	(2.55)	(0.82)	(0.96)	(2.46)	(2.67)	(7.07)	(0.87)	(3.12)	(2.71)	(-3.80)

Table 7: Relation Between LLM scores and Market Reaction

Provider	Knowledge cutoff	Calls pre	Calls post	Straddles?
Anthropic	2023-08	0	1,946	No
OpenAI	2023-10	0	1,946	No
Mistral	2023-10	0	1,946	No
Meta (Llama)	2023-12	0	1,946	No
Qwen	2024-06	970	976	Yes
DeepSeek	2024-07	1,242	704	Yes

Table 8: Knowledge-Cutoff Robustness: Model Cutoffs and Calls Before and After Each Cutoff

Construct	DeepSeek $\times$ post	Qwen $\times$ post
Corp. culture	0.0748 ∗∗∗	-0.0402
	(3.35)	(-1.54)
Quality	0.0668 ∗∗∗	-0.0220
	(2.88)	(-0.82)
Mgmt clarity	0.0573 ∗∗∗	0.0847 ∗∗∗
	(2.65)	(3.36)

Table 9: Knowledge-Cutoff Robustness: Per-Construct Difference-in-Differences on Disagreement

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix


Variable	Definition
LLM-based variables
LLM Score	the score assigned by an LLM provider to the Q&A text of a call for a given construct, on the scale requested in the prompt (0–1 for all constructs except sentiment, which is scored from $-1$ to $+1$ ; the specificity ratio is defined as specific words over total words). The full prompts are reported in the Appendix. Scores are winsorized at the 1st and 99th percentiles within construct and provider, and z-standardized within construct and provider where indicated
LLM Confidence	the self-reported confidence (0–1) that each provider returns together with its score
LLM Dispersion (Z)	the standard deviation of the seven providers’ scores for the same call and construct, computed after z-standardizing each provider’s scores within construct
Justif. Cosine Sim	the average pairwise cosine similarity of sentence embeddings of the seven providers’ written justifications for the same call and construct

Appendix

Table A1 : Definition of variables

Panel of Figure 11 — Figure OA.A.1 : Sentiment

Panel of Figure 12 — Figure OA.A.2 : Uncertainty

Panel of Figure 13 — Figure OA.A.3 : Specificity

Panel of Figure 14 — Figure OA.A.4 : Specificity Ratio

Panel of Figure 15 — Figure OA.A.5 : Innovation Score

Panel of Figure 16 — Figure OA.A.6 : Climate Risk

Panel of Figure 17 — Figure OA.A.7 : Political Risk

Panel of Figure 18 — Figure OA.A.8 : Corporate Culture

Panel of Figure 19 — Figure OA.A.9 : Management Clarity

Panel of Figure 20 — Figure OA.A.10 : Teamwork Score

Panel of Figure 21 — Figure OA.A.11 : Quality Score

Panel of Figure 22 — Figure OA.A.12 : Respect Score

Panel of Figure 23 — Figure OA.A.13 : Integrity Score

	N	Mean	Std	Min	P25	Median	P75	Max
LLM-related variables (call $\times$ construct level)
LLM Dispersion (Z)	25,298	0.618	0.315	0.067	0.415	0.583	0.784	2.787
Justif. Cosine Sim	25,298	0.618	0.102	0.204	0.564	0.624	0.682	0.931
Justif. Jaccard Sim	25,298	0.183	0.061	0.010	0.150	0.169	0.193	0.584
Firm-level variables (call level)
ln(Assets)	25,285	10.348	1.281	7.288	9.498	10.246	11.102	15.253

Appendix

Table OA.B.1: Summary Statistics — Full Detail (incl. per-construct LLM scores)

Panel A: Firm Controls
	Textual Measure Dispersion (Range)
	(1)	(2)	(3)	(4)
$\ln(\text{Assets})$	0.0169 ∗∗∗	0.0069	0.0070	0.0112 ∗
	(2.63)	(1.04)	(1.06)	(1.76)
Book-to-Market	0.0108	-0.0090	-0.0092	-0.0191
	(0.38)	(-0.29)	(-0.29)	(-0.70)

Appendix

Table OA.C.1: Determinants of textual measure dispersion (range-based).

Figure OA.D.1 : Self-reported confidence: distribution and conditional agreement. Panel (a): pooled distribution of the confidence field across the seven providers and thirteen constructs (a single invalid response with confidence $>1$ is excluded). Panel (b): mean pairwise Spearman correlation of scores per construct, using all calls (gray) versus only calls where both models of a pair report confidence at or above their own provider–construct median (red); “degenerate” marks constructs where nearly all high-confidence pairs lack score variation. Stricter definitions of high confidence lower the conditional correlations further.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Oct 1, 2026Yingzhu Zhao, Vlad Pandelea, Han Yuan +4LLM Evaluation Financial QA
Global Decision Science, American Express · Singapore Decision Science Center of Excellence, American Express, 1 Marina Boulevard, 018989, Singapore.
Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts
Oct 15, 2026Alina KhaybullinaFinancial Forecasting Numerical Reasoning in Language Models
Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
Oct 15, 2026Wonjeong Jo, Gyuwan Kim, Ming Shan HeeLLM Reliability Language Model Robustness
UC Davis · UC Santa Barbara · MBZUAI

Oct 1, 2026J/K move · Enter open · S save

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu+3

Global Decision Science, American Express · Singapore Decision Science Center of Excellence, American Express, 1 Marina Boulevard, 018989, Singapore.

LLM Evaluation Financial QA Evidence-Grounded Generation LLM Abstention

Open paper AskPDF arXiv

Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations. Empirical results show that LLMs perform well in groundedness but face notable limitations in correctness, with informational insufficiency presenting an additional challenge.

Abstract

LLM Reliability

Explore similar work

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring