E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

Xiaoya Wang; Yutong Xu; Junjie Wang

cs.CLSep 13, 2026

E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

Authors: Xiaoya Wang, Yutong Xu, Junjie Wang

Organizations: Tsinghua University · Jinan University

Abstract

Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly claim-centric; they assess whether generated statements are supported, but not whether evidence remains traceable through rationale, confidence, and final action. We introduce E2A-Bench, a 969-query benchmark for financial chart reasoning, constructed from 323 HS300 constituents under three input modalities with deterministic OHLCV-derived evidence anchors. E2A-Bench evaluates grounding, reasoning-action consistency, evidence-confidence calibration, and directional coverage through UCR, RCI, ECI, and NDR, where NDR measures coverage-aware evidence-to-action reliability rather than realized trading performance. Evaluating 20 VLMs reveals three failures hidden by scalar hallucination scores: the lowest-UCR model ranks near the bottom by NDR due to only 6.4% directional coverage; oracle-aided verification reduces unsupported claims but can collapse coverage; and financial fine-tuning amplifies the BUY:SELL ratio by factors of 4.21 to 4.68 across strict base-fine-tuned pairs. These results show that financial VLM evaluation should trace the full evidence-to-action chain rather than rely on a single hallucination score. Code and data: https://github.com/wanng-ide/E2A-Bench

Figures & tables

Figure 1: Overview of coverage-aware evidence-to-action evaluation in E2A-Bench. Existing evaluations are often coverage-blind and chain-blind: they measure whether claims are supported, but not whether models preserve actionable coverage or maintain full-chain reliability. E2A-Bench addresses both by evaluating evidence-to-action reliability through grounding, coherence, calibration, and net decision reliability.

Benchmark	Scale	#Inp.	Main evaluation target	Ev.	Act.	Cov.
CharXiv	$\approx$ 2K charts	1	Realistic chart QA	✗	✗	✗
ChartHal	$\approx$ 2K claims	1	Hallucination detection	✓	✗	✗
VisFinEval	$\approx$ 1K Q&A	1	Financial chart QA	✗	✗	✗
FinChart-Bench	$\approx$ 5K Q&A	1	Financial chart QA	✗	✗	✗
E2A-Bench (ours)	$\approx$ 1K Q&A	3	Evidence-to-action reliability	✓	✓	✓

Table 1: Comparison with representative chart and financial multimodal benchmarks. “#Inp.” denotes the number of supported input types among text-only, chart-only, and multimodal inputs. “Ev.” indicates explicit evidence-grounding evaluation; “Act.” indicates decision-oriented action evaluation; and “Cov.” indicates coverage-aware scoring.

Fact Key	Compact Definition
MA alignment	Ordering of MA5, MA10, and MA20 on the last day
MA cross	Golden/death cross of MA5 and MA10 in the last 5 days
Volume change	Current volume relative to the 20-day mean
Price breakout	Current close relative to the preceding 20-trading-day high–low band, excluding the evaluation day
Trend direction	Linear-regression slope of 20-day close prices
Volatility	Annualized volatility of recent returns

Table 2: Compact summary of the ten deterministic OHLCV-derived evidence anchors used in E2A-Bench .

Figure 2: Evidence-to-action scoring protocol in E2A-Bench . A model response is parsed into claims, rationale polarity, action, and confidence; claims are verified against deterministic OHLCV evidence anchors; and the response is scored along grounding, reasoning–action consistency, evidence–confidence calibration, and directional coverage.

Figure 3: Example of evidence-to-action scoring. Given a chart-only query, the VLM produces a technical analysis, rationale, action, and confidence score, which are parsed and verified against deterministic OHLCV evidence anchors. Here, a high-confidence $\mathrm{SELL}$ recommendation has coherent reasoning but contains two unsupported claims and insufficient evidence for its confidence, yielding $\mathrm{UCR}=0.50$ , $\mathrm{RCI}=0$ , $\mathrm{ECI}=1$ , and $\mathrm{NDR}=0$ .

Statistic	Value
Stocks	$323$ HS300 constituents
Input modalities	text-only, chart-only, multimodal
Benchmark queries	$969$
Evidence anchors	$3{,}230$ total facts
Input token length, mean $\pm$ std.	text-only: $1628.8\pm 61.2$ ; chart-only: $1027.0\pm 0.0$ ; multimodal: $2466.8\pm 61.2$

Table 3: Benchmark-level statistics of E2A-Bench .

Proprietary
Model	Size	Cov. $\uparrow$	$\mathbf{UCR}_{\mathrm{all}}$ $\downarrow$	$\mathbf{UCR}_{\mathrm{dir}}$ $\downarrow$	$\mathbf{RCI}$ $\downarrow$	$\mathbf{ECI}$ $\downarrow$	$\mathbf{NDR}$ $\uparrow$
Gemini-3.1-Pro	–	59.0	70.0	67.5	5.2	7.5	16.8
Gemini-3.1-Flash	–	37.9	58.1	53.8	2.7	6.3	16.0
Claude-Sonnet-4.6	–	55.1	71.5	69.4	3.6	1.9	16.0
GPT-5.5	–	25.9	61.2	47.6	2.0	4.0	12.8
GPT-5.4	–	15.2	65.4	54.4	2.7	4.1	6.5

Table 4: Main trimodal leaderboard on E2A-Bench . Reported metrics aggregate performance over the three input modalities; NDR is computed from the aggregated components rather than by averaging modality-specific NDR values. Cov. denotes directional coverage, and NDR denotes coverage-aware net decision reliability. Within each group, bold and underlined entries mark the best and second-best values. Superscripts indicate the base–fine-tuned pairs: a Qwen2.5-VL-3B, b Qwen2.5-VL-7B, and c Qwen3-VL-4B-Thinking.

Figure 4: Input modality effects on baseline NDR . (a) Model-wise NDR across text-only, chart-only, and multimodal inputs for 12 representative models. (b) Mean NDR by modality across all 20 evaluated models, with $95\%$ bootstrap confidence intervals. Modality effects are model-dependent; chart-only input achieves the highest average baseline NDR across the full model set.

Figure 5: Risk redistribution under financial fine-tuning. (a) Base-to-fine-tuned migration on the coverage–grounding plane, where the dashed curves show iso-

\mathrm{Cov}_{\mathrm{dir}}(1-\mathrm{UCR}_{\mathrm{dir}})

contours. (b) Fine-tuning produces inconsistent NDR changes across pairs. (c) All pairs show strong

\mathrm{BUY}

\mathrm{SELL}

amplification, indicating a consistent shift toward

\mathrm{BUY}

actions despite heterogeneous reliability changes.

Figure 6: Coverage trap in claim-level hallucination evaluation. Gemma-4-E2B-It avoids verifiable technical claims and outputs a non-directional $\mathrm{HOLD}$ decision, yielding zero observed UCR but no actionable coverage. In contrast, Qwen3.5-4B issues a directional $\mathrm{SELL}$ recommendation whose claims are supported by the evidence anchors. This case illustrates why claim-level hallucination scores alone can overestimate conservative non-action, while NDR distinguishes non-action from evidence-backed decision reliability.

Figure 7: Scaling behavior within Qwen3.5 under chart-only inputs. (a) Local diagnostics ( UCR , RCI , ECI ) show non-monotonic and partially conflicting trends across model sizes. (b) NDR also varies non-monotonically, peaking at 2B.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Signal Type	Qualifying Labels
Price breakout	breakout_high , breakout_low
Volume anomaly	heavy_volume , light_volume
Strong trend	uptrend or downtrend with $\|\mathrm{slope}\|>0.30\%$ /day
Vol–price divergence	bullish_divergence , bearish_divergence
Extreme price position	high_zone , low_zone
Momentum acceleration	accelerating_up , accelerating_down

Appendix

Table 5: Evidence-anchor labels counted as strong signals for

S(F)

. A sample has weak evidence when

S(F)=0

; ECI fires when such a sample receives a high-confidence directional action.

Quantity	Value
Stocks	$323$
Input modalities	$3$
Benchmark queries	$969$
Evaluated models	$20$
Recorded response cells	$19{,}380$
Max claim judgments / response	$16$

Appendix

Table 6: Core trimodal benchmark and scoring inventory. The response-level judgments comprise reasoning–action consistency, evidence–confidence consistency, and directional coverage. Chart-only intervention settings are treated as experimental conditions and are not counted as additional core benchmark queries.

Figure 8: Final-action distribution over 55,209 successfully parsed actions. $\mathrm{HOLD}$ and $\mathrm{UNCERTAIN}$ account for 63.6% of parsed actions, motivating coverage-aware evaluation beyond claim-level hallucination.

#	Fact Key	Dimension	Inputs	Computation Rule	Output Labels	ECI Signal
1	MA alignment	MA stacking	Close	Compare MA5, MA10, and MA20 on the last day.	bullish , bearish , mixed	No
2	MA cross	Short-term crossover	Close	Detect golden/death cross of MA5 and MA10 within the last 5 days.	golden_cross , death_cross , no_cross	Yes
3	Volume change	Volume anomaly	Volume	Compare current volume with the 20-day mean; heavy if $>1.5\times$ , light if $<0.7\times$ .	heavy_volume , light_volume , normal_volume	Yes
4	Price breakout	Range breakout	Close, High, Low	Compare the evaluation-day close with the high and low over the preceding 20 trading days, excluding the evaluation day.	breakout_high , breakout_low , within_range	Yes
5	Trend direction	Price trend	Close	Fit a linear trend to 20-day close prices; use $0.15\%$ /day as the trend-label threshold.	uptrend , downtrend , sideways	Yes †
6	Volatility	Return variability	Close	Compute annualized standard deviation of recent returns.	high_volatility , medium_volatility , low_volatility	No

Appendix

Table 7: Full specification of the ten deterministic OHLCV-derived evidence anchors. “Strong” indicates whether the anchor contributes to the strong-signal count

S(F)

used by ECI . † Trend direction uses

0.15\%

/day for label assignment and a stricter

0.30\%

/day threshold for strong-signal counting.

Diagnostic	AC1	Precision	Recall	F1
UCR	0.71	91.3%	88.7%	90.0%
RCI	0.74	87.6%	84.2%	85.9%
ECI	0.82	93.1%	90.5%	91.8%

Appendix

Table 8: Human validation of the automatic scorer. AC1 measures inter-annotator agreement among four anonymous industry annotators; precision, recall, and F1 are computed against human-consensus labels.

Panel of Figure 9 — Figure 9: English translation of the base inference prompt. The original prompt is written in Simplified Chinese and used for model inference; examples in the paper are translated for readability.

Threshold	Diagnostic interpretation
$\theta=0$	Main-paper setting. No evidence filter is applied to directional coverage, so any directional action can contribute.
$\theta=1$	Minimal evidence threshold. A directional action contributes only when at least one strong chart signal is present.
$\theta>1$	Stricter evidence threshold. Coverage progressively decreases as directional actions are required to be associated with more strong chart signals.
$\theta\to\infty$	Conservative endpoint. Coverage approaches zero for all models. The rate of decay reflects how much directional coverage each model retains under increasingly strict evidence requirements.

Appendix

Table 9: Operational interpretation of the evidence threshold

\theta

\mathrm{NDR}_{\theta}

. The threshold changes only the coverage aggregation rule over existing response logs.

Figure 10: NDR sensitivity to evidence threshold

\theta

. The main-paper setting is

\theta=0

, where all directional actions contribute to coverage. Increasing

\theta

progressively restricts coverage to directional actions associated with more strong chart signals, while the directional-conditioned chain-reliability terms remain fixed. The leading models retain similar relative ordering under mild thresholds, whereas all models experience substantial NDR decay under stricter evidence requirements.

Figure 11: Local diagnostics capture non-redundant failure modes. The heatmap reports pairwise Spearman correlations among $\textsc{UCR}_{\mathrm{dir}}$ , RCI , and ECI across 20 models. All off-diagonal correlations are small ( $\rho\leq 0.26$ ), indicating that factual grounding, reasoning–action consistency, and evidence–confidence calibration measure largely distinct aspects of evidence-to-action reliability.

Aggregation	Formula	Key observation
Product / NDR	$\prod_{i=1}^{4}z_{i}$	Current aggregation; Gemma-4-E2B-It ranks $19/20$ .
Arithmetic mean	$\frac{1}{4}\sum_{i=1}^{4}z_{i}$	$\rho=0.747$ with NDR ; Gemma-4-E2B-It rises to rank $6/20$ .
Minimum	$\min_{i}z_{i}$	$\rho=0.889$ with NDR ; improvements outside the weakest component are ignored.
Pareto dominance	–	$16/20$ models are non-dominated, and only about $2\%$ of model pairs can be ordered.

Appendix

Table 10: Sensitivity of model comparison to alternative aggregation rules. The multiplicative form preserves the necessary-link interpretation while retaining sensitivity to all four components.

Deterministic OHLCV evidence
Item	Paper-snapshot evidence
Stock / date	601799.SH, 2023-06-30
Input modality	Chart-only (candlestick PNG with MA5/MA10/MA20 and volume)
MA alignment	Bullish: MA5 $=119.92$ , MA10 $=118.30$ , MA20 $=114.04$
Volume ratio	$0.98\times$ the 20-day mean (normal volume)
Price breakout	Breakout above the prior 20-trading-day high: current close $=123.60$ , prior-window high $=122.48$

Appendix

Table 11: Released-artifact evidence audit for a contrastive chart-only case. Gemma-4-E2B-It has zero response-level UCR but does not issue a directional action. Qwen3.5-9B incurs a

25\%

response-level UCR from one unsupported volume claim, while its

\mathrm{BUY}

action remains directionally covered, rationale-consistent, and calibrated.

Figure 12: Imperfect claims can still yield reliable action. Qwen3.5-4B makes one unsupported claim, resulting in non-zero UCR , but its $\mathrm{SELL}$ action remains directionally covered, coherent with the bearish rationale, and supported by multiple verified evidence signals. This example shows that NDR is not a perfection-based hallucination score; it evaluates whether an actionable recommendation remains sufficiently evidence-backed across the full chain.

Confidence Estimation for VLMs

See all 25 papers

Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding
Aug 6, 2026Reza Khanmohammadi, Simerjot Kaur, Charese H. Smiley +2Visual Question Answering Selective Prediction
Michigan State University East Lansing, Michigan, USA · JPMorgan AI Research New York, New York, USA
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Aug 11, 2026Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhmasari +2Numerical Reasoning in Language Models Synthetic Data Generation
ETH Zürich, Zürich · Aisot Technologies Ltd
FinTradeBench: A Financial Reasoning Benchmark for LLMs
Mar 19, 2026Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan +2Financial Reasoning in Language Models Financial Reasoning Benchmarks
University of Central Florida

Aug 6, 2026J/K move · Enter open · S save

Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding

Reza Khanmohammadi, Simerjot Kaur, Charese H. Smiley, Ivan Brugere+1

Michigan State University East Lansing, Michigan, USA · JPMorgan AI Research New York, New York, USA

Visual Question Answering Selective Prediction VLM Calibration Finance

Open paper AskPDF arXiv

LVLMs are increasingly used to read financial charts, tables, and documents, where a single misread figure can move a decision and the most authoritative-looking answer is sometimes one the model produced without reading the exhibit. The operational question is therefore trust, not accuracy: which answers can be acted on, and which escalated to a reviewer. We evaluate seven confidence estimators, three inference-only and four trained internal probes, across five open-weight LVLMs and four conditions from three financial visual question-answering benchmarks, one bilingual; every probe is trained only on natural images and applied to finance without adaptation, so the results measure out-of-distribution transfer. Three findings hold. First, the scarce property is calibration, not ranking: the inference baselines rank correct above incorrect answers competitively but are badly overconfident, calibration error far above what a threshold can tolerate, and only the trained probes produce a thresholdable score. Second, reliability is structured rather than global, along two axes a practitioner can read directly: the best estimator shifts with both model and task, none leading more than eight of twenty (model, condition) cells, and a controlled bilingual contrast exposes an apparent language robustness as a composition artifact that dissolves once models are read one at a time. Third, cast as deferral under an error budget, how much can be safely automated is set first by the model's competence and only narrowed by its confidence, so deferral clears a real share of the easiest condition and almost none of the hardest, near zero at a strict 5% budget. Two trained probes carry the calibration a deferral policy needs, and among them only the grounding-aware one lowers its confidence on answers a model gives without using the figure, separating detected non-grounding from a fluent guess.

Abstract

Confidence Estimation for VLMs

Explore similar work

Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding

V-FiLLM: Verified Financial LLM Reasoning Benchmark

FinTradeBench: A Financial Reasoning Benchmark for LLMs