From SEC Filings to Financial Signals: Evaluating LLMs on Stock Market Reactions and Analyst Forecasts

Shengkun Wang; Taoran Ji; Linhan Wang; Sattar Mansi; Chang-Tien Lu; Yinlin Chen

cs.AIOct 16, 2026

From SEC Filings to Financial Signals: Evaluating LLMs on Stock Market Reactions and Analyst Forecasts

Authors: Shengkun Wang, Taoran Ji, Linhan Wang, Sattar Mansi, Chang-Tien Lu, Yinlin Chen

Abstract

Form 10-Q MD&A sections combine long-form management narrative with accounting and forward-looking information. Benchmarks for financial filings commonly evaluate question answering or numerical reasoning, leaving unclear whether outputs from large language models (LLMs) align with market and analyst outcomes. We evaluate prompted, zero-shot outputs from 13 general-purpose and finance-specialized LLMs on 45,964 Form 10-Q filing events from 2001 to 2024, linking MD&A text to cross-sectional fundamental ranks, pre-filing analyst forecasts, and realized abnormal returns. The evaluation covers return ranking under five controlled information views and next-quarter EPS forecasting under direct and consensus-anchored interfaces. Relative to disclosure alone, adding cross-sectional fundamental ranks raises mean effective-date rank correlation (RankIC) by 0.0089 for effective-event-date returns and 0.0098 for following-day returns, whereas analyst-summary context provides no comparable average gain. Across the four disclosure-containing views, finance-specialized models show an effective-event-date group advantage but no statistically reliable following-day advantage, while general-purpose Qwen models produce the strongest individual return-ranking results. Under the full-information view, direct LLM forecasts average the 43.4th percentile of the analyst distribution and do not consistently beat individual analysts. Consensus anchoring raises the mean inside-range share from 41.1% to 59.5% but increases reported-EPS error.

Figures & tables

Figure 1. Adding cross-sectional fundamental ranks to the same 10-Q MD&A can reverse the return signal inferred from the filing. A schematic shows the same 10-Q MD\&A producing a negative return score under the disclosure-only view and a positive score after cross-sectional fundamental ranks are added. — Figure 1. Adding cross-sectional fundamental ranks to the same 10-Q MD&A can reverse the return signal inferred from the filing. A schematic shows the same 10-Q MD&A producing a negative return score under the disclosure-only view and a positive score after cross-sectional fundamental ranks are added.

Figure 2. Annual counts of 10-Q filing events by SEC acceptance-time bucket. A stacked bar chart shows annual 10-Q event counts from 2001 through 2024 divided into pre-open, market-hours, after-close, and missing-time filings. After-close filings form the largest category in most years.

Figure 3. Score granularity by response interface: mean distinct parsed scores on RankIC-eligible effective event dates. $V_{k}$ and $N_{k}$ denote $k$ -level verbal and numeric ordinal interfaces; Cont. denotes continuous scoring. A line chart compares Qwen2.5-14B and Fino1-14B across verbal and numeric ordinal interfaces with 3, 5, 7, or 10 levels and a continuous interface. Distinct scores generally increase with interface resolution and are highest under continuous scoring.

General-purpose LLMs
Information View	Disclosure only				Fundamentals only				Disclosure + expectations				Disclosure + fundamentals				Disclosure + fundamentals + expectations
Model	$\mathrm{IC}_{y_{0}}$	$\mathrm{IC}_{y_{1}}$	$\mathrm{ICIR}_{y_{0}}$	$\mathrm{LS}_{0}$	$\mathrm{IC}_{y_{0}}$	$\mathrm{IC}_{y_{1}}$	$\mathrm{ICIR}_{y_{0}}$	$\mathrm{LS}_{0}$	$\mathrm{IC}_{y_{0}}$	$\mathrm{IC}_{y_{1}}$	$\mathrm{ICIR}_{y_{0}}$	$\mathrm{LS}_{0}$	$\mathrm{IC}_{y_{0}}$	$\mathrm{IC}_{y_{1}}$	$\mathrm{ICIR}_{y_{0}}$	$\mathrm{LS}_{0}$	$\mathrm{IC}_{y_{0}}$	$\mathrm{IC}_{y_{1}}$	$\mathrm{ICIR}_{y_{0}}$	$\mathrm{LS}_{0}$
Qwen3-32B	0.07068	0.02640	0.24043	0.00707	0.06019	0.03025	0.19626	0.00613	0.06124	0.03200	0.19077	0.00634	0.07555	0.03906	0.25582	0.00761	0.07447	0.03426	0.23678	0.00564
Qwen3-8B	0.04513	0.04402	0.13885	0.00533	0.05507	0.02764	0.18312	0.00388	0.05198	0.03972	0.16100	0.00601	0.06279	0.05742	0.19573	0.00705	0.06146	0.05179	0.18890	0.00670
Qwen2.5-32B	0.05557	0.02875	0.18114	0.00573	0.05914	0.02995	0.20532	0.00538	0.06204	0.02648	0.19842	0.00665	0.06384	0.03217	0.20596	0.00668	0.06331	0.03507	0.20453	0.00645
Qwen2.5-14B	0.05935	0.02048	0.19028	0.00677	0.06618	0.03652	0.21219	0.00394	0.06604	0.02355	0.21016	0.00696	0.07668	0.03449	0.24074	0.00762	0.06859	0.03144	0.21608	0.00716

Table 1. Main 10-Q return-ranking results. IC is mean effective-date RankIC, and ICIR is its information ratio.

y_{0}

is the effective-event-date abnormal return;

y_{1}

is the following-trading-day abnormal return. Column maxima are blue (general-purpose) or green (finance-specialized); runners-up are underlined, and boldface marks the metric-wide maximum.

Figure 4. Analyst-relative forecast position and reported-EPS accuracy under direct and consensus-anchored interfaces. Left: shares beating the median, top-quartile, and best individual analyst; center: analyst-equivalent percentile; right: shares below, inside, and above the pre-filing analyst range. Two model-by-model panels compare direct forecasts with consensus-anchored adjustments. The left panel reports analyst-relative beat rates, the center reports analyst-equivalent percentiles, and the right reports shares below, inside, and above the analyst range. Anchoring generally places more forecasts inside the analyst range but lowers the rate of beating the best analyst.

	$y_{0}$		$y_{1}$
Input configuration	RankIC	$t$	RankIC	$t$
Original D + F	0.07555	9.319	0.03906	4.963
Masked D	0.06163	7.852	0.02073	2.583
Masked D + matched F	0.07170	8.618	0.02776	3.596
Masked D + shuffled F	0.04935	6.104	0.01147	1.453
Paired contrast	$\Delta$ RankIC	Paired $t$	$\Delta$ RankIC	Paired $t$

Table 2. Qwen3-32B RankIC under original and identity-masked input configurations. D denotes disclosure and F denotes cross-sectional fundamental ranks. Input-configuration

t

-statistics test mean effective-date RankIC against zero; the paired contrast reports matched-minus-shuffled

\Delta

RankIC and its

t

-statistic.

Panel of Table 3 — Table 3. Condensed prompt templates and output schemas for the three prediction interfaces.

Financial Language Model Evaluation

See all 23 papers

Oct 15, 2026

Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts

Which inputs drive a language model's financial forecast? We construct synthetic earnings announcements that independently reverse 20-day momentum, non-informational commentary tone, and whether fixed actual EPS beats a manipulated consensus. Six instruction/chat checkpoint conversions from four families (0.49-7.61B base parameters) receive 2,016 prompts each under a direct-answer Q8_0 protocol. The design includes 24 numerical blocks, six commentary pairs, profits and losses, rising and falling annual EPS, and two surprise sizes. Both A/B answer assignments and U/D cover all 384 primary scenarios per conversion. We estimate matched changes in conditional answer-prefix scores, not calibrated market probabilities. Across these checkpoints, the primary EPS contrast ranges from -0.19 to 86.80 percentage points. The magnitude of the primary signed tone contrast exceeds that of EPS in 5 of six models, and momentum exceeds it in 5. Natural-word numerical diagnostic accuracy ranges from 51.0 to 95.8 percent. Answer-encoding, wording, context, and conversion controls qualify these results. The experiment measures observable input reliance and numerical recognition; it establishes neither an internal reasoning mechanism nor realized return predictability. Public benchmark prompts, recorded scores, and an interactive recorded-results explorer accompany the study.

Alina Khaybullina

Financial Forecasting Numerical Reasoning in Language Models

Ask PDF

Sep 2, 2026

The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis

Large Language Models (LLMs) increasingly use user context such as memory, profiles, and role prompts to personalize their responses. This personalization can affect evidence-based judgment: the same evidence may lead to different conclusions under different user contexts. Finance provides a high-stakes setting to study this problem because decisions often depend on interpreting long and complex documents. We test this using 3,575 SEC filings across twelve LLMs. We compare persona-conditioned retrieval, neutral retrieval, and memory-framed context to separate the effect of evidence selection from the effect of interpretation. We find that most user-context spillover comes from how models interpret the same evidence under different roles, rather than from retrieving different evidence. We then test two simple mitigation strategies: expressing the same investor mindset as a user profile instead of an assistant role, and separating evidence-based and personalized outputs. Both reduce spillover, but neither removes it completely, and their effectiveness varies substantially across models.

Ahmed Asaad, Amr Mohamed, Yang Zhang +1

Durham University Business School · MBZUAI & Ecole Polytechnique · Ecole Polytechnique +1

LLM Evaluation LLM Auditing LLM Context Management Financial Decision-Making

Ask PDF

Jul 14, 2026

Fin-Analyst at FinMMEval 2026 Task 3: A Live Hybrid Trading Agent with LLM Specialists and Rule-Based Signals

Large language model (LLM) trading agents show promising performance in equity markets, yet remain narrowly focused on US equities with little evidence from live deployment. We present Fin-Analyst, a hybrid agent for FinMMEval 2026 Task 3: an eight-specialist LLM pipeline over news, SEC filings, fundamentals, analyst forecasts, technical indicators, and social sentiment, aggregated by a Meta-Agent for Tesla (TSLA), and a lightweight rule based three-signal vote for Bitcoin (BTC). On the final official leaderboard (accessed 2026-07-05), Fin-Analyst ranks first of all agents on TSLA with a +13.51% return, +28.33 points over Buy-and-Hold (Sharpe 4.10, 88% win rate), while the BTC vote ends flat yet well above a sharply falling baseline. Relative to the interim performance, the asset ranking reversed, indicating that short live windows yield volatility-sensitive rankings. Ablation identifies event-driven 8-K disclosures as the most influential TSLA signal. Error analysis shows that the memoryless agents repeat wrong calls for days at a time, and that the fixed-threshold BTC rules lost money by trading on noise in a sideways market while the LLM pipeline gained under similar conditions, motivating a memory-aware, LLM-based successor for both assets.

Mohotarema Rashid, Lingzi Hong, Junhua Ding +1

Department of Information Science, University of North Texas, Denton, TX, United States · Department of Data Science, University of North Texas, Denton, TX, United States

Multi-Agent LLM Systems LLM Agent Evaluation AI Agent Benchmarks LLM Agents Finance Algorithmic Trading

Ask PDF

Abstract

Financial Language Model Evaluation

Explore similar work

Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts

The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis

Fin-Analyst at FinMMEval 2026 Task 3: A Live Hybrid Trading Agent with LLM Specialists and Rule-Based Signals