FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review

Suyang Zhong; Jingzhe Zhu; Qi Xu; Liyao Sun; Yin Wang; Qingqing Sun; Shuai Chen; Tianyi Zhang

cs.AIAug 26, 2026

FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review

Authors: Suyang Zhong, Jingzhe Zhu, Qi Xu, Liyao Sun, Yin Wang, Qingqing Sun, Shuai Chen, Tianyi Zhang

Organizations: Ant International · Xiamen University · Shanghai University · Tongji University

Abstract

Deploying large language models for professional financial review requires more than measuring general financial competence: models must perform the specific review operation required by a workflow and determine whether available evidence is sufficient for a defensible decision. Existing financial benchmarks cover knowledge, reasoning, compliance, and professional tasks, but their evaluation units are often organized around datasets or task formulations rather than the decisions that deployed systems support. We introduce FinRiskAtlas, a Chinese-language benchmark that evaluates financial LLMs along two complementary dimensions: operation execution under fixed evidence states and evidence-state control under evolving review conditions. The static benchmark contains 9,742 instances across 53 task families, including 42 Domain Knowledge families and eleven downstream review operations defined by explicit evaluation contracts. FinRisk-Ask extends this framework through offline replay of 680 pre-action states from 104 de-identified professional trajectories, withholding future evidence during inference and using it only to construct expert-verified evidence targets. Across 33 model configurations, operation-level evaluation yields non-redundant rankings (mean pairwise Spearman correlation 0.42 across downstream operations), and knowledge-based shortlisting can incur up to 18.01 points of regret on individual operations. FinRisk-Ask further shows that entering the Ask branch more frequently does not necessarily improve request targeting or end-to-end evidence acquisition. These results show that broad financial capability scores do not fully capture where models are reliable in professional workflows, motivating evaluation units aligned with the decisions and evidence states that deployed systems must support.

Figures & tables

Figure 1: Composition of FinRiskAtlas. The static benchmark evaluates whether a model can execute a specified financial review operation under a fixed evidence state. FinRisk-Ask evaluates whether a model should proceed or acquire additional evidence from an intermediate review state and, after choosing to ask, whether the request targets a trajectory-supported unresolved need. Static instances and trajectory states represent different evaluation units and are evaluated separately.

Figure 2: Benchmark design around operation and evidence state. Professional source materials are organized through an operation-aligned taxonomy and fixed evaluation contracts to construct the static benchmark. FinRisk-Ask complements fixed-evidence operation evaluation through offline replay of model-visible pre-action states, measuring both the Ask-or-Proceed transition and the alignment of generated requests with later-observed, expert-verified evidence targets. Later evidence is withheld during inference and used only for target construction.

Evaluation component	Operations	Workflow outputs
Evidence-Grounded Processing	Case classification; information extraction; institution matching; person matching; quantitative reasoning	Procedure routing, structured evidence fields, entity-identity decisions, and normalized quantities
Applied Review	Risk classification; applicable-provision selection; legal-outcome prediction; legal-judgment generation; disputed-issue generation; decision-view generation	Risk grades, governing provisions, dispositions, reasoned analyses, issue lists, and review opinions
State-level extension FinRisk-Ask	Ask-or-Proceed decision, with one focused evidence request after selecting Ask	Advancement of the current state or a targeted evidence request

Table 1: Operation coverage in FinRiskAtlas. The static benchmark contains eleven fixed-evidence review operations. FinRisk-Ask evaluates the complementary state-level decision of whether the current evidence state should be advanced or expanded.

Figure 3: Operation-level profiles across evaluated configurations. (a) Within-column percentile heatmap, with configurations ordered by the Domain Knowledge macro; hatched cells denote evaluator self-scores excluded from evaluator-dependent comparisons. (b) Raw downstream-operation scores for the knowledge-near pair Gemini-3-Flash-preview and Kimi-K3. (c) Spearman correlations between the Domain Knowledge macro and downstream operations, using 33 configurations for structured operations and 32 for judge-scored operations.

Figure 4: Evidence-state behavior and operation-specific screening regret. (a) Relationship between Balanced Recorded-Action Agreement and Evidence-Request Alignment across configurations. (b) Decomposition of request alignment into recorded-Ask recall and Conditional Request Alignment. (c–d) Score forgone under Domain-Knowledge-based shortlisting across different shortlist sizes.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Primary focus	Operation aligned	Decision contract	Evidence acquisition	Expert trajectories
FLUE Shah et al. (2022)	Financial language understanding	✗	✗	✗	✗
FinQA / ConvFinQA Chen et al. (2021) , Chen et al. (2022)	Numerical reasoning over financial reports	✗	✗	✗	✗
BizBench Krumdick et al. (2024)	Business quantitative reasoning	✗	✗	✗	✗
PIXIU / FinBen Xie et al. (2023) , Xie et al. (2024)	Broad financial capability coverage	✗	$\circ$	✗	✗
CFinBench / FinEval Nie et al. (2025) , Guo et al. (2025)	Chinese financial knowledge evaluation	✗	$\circ$	✗	✗
CNFinBench Ding et al. (2026)	Financial capability, compliance, and safety	$\circ$	$\circ$	✗	✗

Appendix

Table 2: Positioning of FinRiskAtlas relative to representative financial and information-acquisition benchmarks. Operation-aligned indicates that downstream units are defined by their role in a professional workflow rather than source format or task type. Decision contract indicates that the visible evidence regime, decision object, reviewer artifact, and scoring protocol are fixed at the family level. Evidence acquisition indicates evaluation of whether and what additional information should be requested. ✓ denotes present as a benchmark-level design unit, ✗ not documented as such, and

\circ

partial.

Layer	Evaluation focus	Families / settings	Instances
Domain Knowledge	Financial concepts, regulations, obligations, and risk mechanisms supporting professional review	42	4,679
Evidence-Grounded Processing	Transforming heterogeneous records into structured and decision-relevant evidence	5	1,502
Applied Review	Evidence-based case analysis, rule grounding, and professional review generation	6	3,561
Static benchmark	Three-layer financial review evaluation	53	9,742
FinRisk-Ask	Evidence-state control over reconstructed professional review states	1 setting	680 states (583 Ask, 97 Proceed)

Appendix

Table 3: Benchmark composition of FinRiskAtlas. Static instances and active-review states represent different evaluation units and are not combined.

Layer	Group	Coverage	Families	Instances
Domain Knowledge	Financial risk	Risk management, markets, products, valuation, credit, operational, liquidity, and fixed-income risk	8	892
	Fraud risk	Card, insurance, telecom, contract, invoice, and AI-enabled fraud	6	624
	Money-laundering risk	Anti-money-laundering obligations and typologies	2	244
	Compliance risk	Regulatory obligations and compliance controls	4	442
	Trade theory and policy	International trade theory and policy	4	457
	FDI and multinationals	Foreign direct investment and multinational operations	2	208

Appendix

Table 4: Group-level composition of FinRiskAtlas. Each row reports the taxonomy group, coverage, number of families, and total number of instances. The complete per-family inventory, including sources and contracts, is released with the benchmark package.

Source category	Capability coverage	Expert involvement
Professional examinations and textbooks	Financial concepts, regulations, and risk mechanisms	Reviewed capability alignment and reference correctness
Regulatory documents and industry guidance	Compliance obligations and rule-grounded reasoning	Reviewed regulatory interpretation and applicability
Business, transaction, entity, and legal records	Evidence extraction, entity reconciliation, quantitative reasoning, and applied review	Reviewed decision targets and evidence support
Completed risk-review trajectories	Ask-or-Proceed transitions and evidence-request evaluation	Verified reconstructed states, action mappings, and evidence targets

Appendix

Table 5: Data sources and expert roles in benchmark construction.

Task type	Output	Scoring rule	Metric
Knowledge and classification	Label, option, or option set	Correctness after family-specific extraction, set handling, and normalization	Accuracy-style score
Information extraction	Requested fields	Normalized field-level comparison with reference fields, preserving required field identity and order	Field-level score
Entity reconciliation	Entity identity	Exact correspondence with the reference identity decision	Accuracy
Quantitative reasoning	Numerical answer	Family-specific numerical normalization and correctness checking	Accuracy-style score
Risk and legal decisions	Categorical decision	Exact decision matching after label normalization	Accuracy
Open-ended review generation	Professional analysis	Fixed semantic rubric evaluating correctness, coverage, and evidence use	Rubric score

Appendix

Table 6: Task-native evaluation contracts in FinRiskAtlas.

Operation	Model-visible information	Decision object	Reviewer artifact	Scoring
Evidence-Grounded Processing
Case classification	Full case record and a fixed candidate category set	Case category	Procedure-routing label	Accuracy
Information extraction	Source record and an ordered list of requested fields	Values of the requested fields	Structured evidence-field sequence	Normalized field-level score
Institution matching	Two institution descriptions, possibly from different records or languages	Whether the descriptions denote the same institution	Institution-identity decision	Accuracy
Person matching	Two person records with partially overlapping attributes	Whether the records denote the same natural person	Person-identity decision	Accuracy
Quantitative reasoning	Case figures together with the applicable definition or formula	Requested numerical quantity	One normalized quantity	Numerical correctness

Appendix

Table 7: Evaluation contracts for the eleven fixed-evidence downstream operations. The operation name specifies the target capability

c_{f}

; the remaining columns summarize the model-visible information regime

\mathcal{I}_{f}

, decision object

d_{f}

, reviewer-artifact schema

\mathcal{Y}_{f}

, and scoring protocol

s_{f}

Figure 5: Additional analyses of operation-specific capability profiles. (a) Relationship between Domain Knowledge difference and downstream profile gap over 496 configuration pairs. Knowledge-near pairs (

|\Delta K|\leq 0.5

) still exhibit substantial downstream divergence. (b) Eigenspectrum of the eleven-operation Spearman correlation matrix, showing shared but non-identical ranking structure across operations. (c) Number of observed operation leaders recovered by Domain Knowledge-based shortlists of different sizes. (d) Domain Knowledge rank of each configuration attaining an observed operation maximum. All maxima correspond to point estimates from the evaluated configuration pool.

Figure 6: Additional analyses of static capability and evidence-state behavior. (a) Spearman associations between downstream operation scores and FinRisk-Ask metrics. (b) Ask recall and Proceed recall across non-evaluator configurations, showing asymmetric reproduction of recorded transitions. (c) Relationship between Recorded-Action Agreement (RAA) and Balanced Recorded-Action Agreement (BAcc). (d) Rank displacement between BAcc and Evidence-Request Alignment (ERA) for representative configurations.

Model	Risk & Compliance	Trade	FinTech & Security	Commerce & Payment	Finance & Law	Overall
Dianjin-R1-7B	64.00	63.43	70.25	72.50	68.00	65.00
FinR1-7B	66.35	68.00	78.25	75.50	76.00	68.93
Ling-2.6-Flash	71.80	74.71	80.00	79.00	77.50	74.17
Ling-2.6-1T	75.22	78.96	83.22	78.82	80.49	77.65
Seed-2.0-Mini	73.22	76.64	80.43	78.32	80.04	75.62
Seed-2.0-Lite	73.70	80.36	78.78	81.51	78.55	77.01

Appendix

Table 8: Complete Domain Knowledge macro results over all 33 archived configurations (%).

Model	Case	Extract.	Inst.	Person	Quant.	Risk	Legal	Provision
Dianjin-R1-7B	77.00	59.00	90.00	70.00	63.00	31.00	38.00	33.00
FinR1-7B	76.00	10.00	82.00	82.00	73.00	29.00	64.00	35.00
Ling-2.6-Flash	80.00	72.00	78.00	80.00	62.00	75.00	60.00	40.00
Ling-2.6-1T	82.92	74.53	78.80	80.40	86.21	75.68	64.76	52.10
Seed-2.0-Mini	83.15	75.42	86.80	73.80	79.31	66.61	62.22	36.95
Seed-2.0-Lite	85.62	84.48	84.00	81.80	79.31	73.22	63.81	56.40

Appendix

Table 9: Complete Evidence-Grounded Processing and structured Applied Review results over all 33 archived configurations (%). Column abbreviations correspond to the operations summarized in Table 1 ; their full evaluation contracts are reported in Table 7 .

Model	Legal Judgment	Decision View	Disputed Issue
Dianjin-R1-7B	29.00	20.00	36.00
FinR1-7B	18.00	15.00	52.00
Ling-2.6-Flash	30.00	37.00	63.00
Ling-2.6-1T	40.27	55.69	63.30
Seed-2.0-Mini	28.95	47.11	72.73
Seed-2.0-Lite	29.66	58.73	73.35

Appendix

Table 10: Complete open-ended Applied Review results over all 33 archived configurations (%). DeepSeek-V4-Flash is the fixed semantic evaluator. Its self-scored row is marked with

\dagger

and excluded from best-value annotation and evaluator-dependent comparisons.

	End-to-end	Action behavior				Request alignment
Model	ERA	BAcc	AskR	ProceedR	RAA	CRA	DEH	REH
Reference policies
Always-Ask	–	50.00	100.00	0.00	85.74	–	–	–
Always-Proceed	–	50.00	0.00	100.00	14.26	–	–	–
Model configurations
Dianjin-R1-7B	57.72	45.97	91.94	0.00	78.82	62.78	51.97	63.46

Appendix

Table 11: Complete FinRisk-Ask results for two reference policies and 33 archived model configurations (%). ERA is the end-to-end evidence-acquisition metric. DeepSeek-V4-Flash is the fixed request-alignment evaluator; its self-scored row is marked with

\dagger

and excluded from evaluator-dependent comparisons.

Figure 7: Domain Knowledge example: AML customer-identification compliance. The task evaluates whether the model can retrieve the applicable customer-identification requirement and determine the stated upper bound of the corresponding administrative fine. Solving the item requires distinguishing the directly applicable penalty range from more severe sanctions associated with different consequences or aggravating conditions.

Figure 8: Evidence-Grounded Processing example: long-context maritime contract classification. The input contains contracts, waybills, delivery records, freight and insurance amounts, performance evidence, and procedural events. The model must distinguish the operative legal relationship from contextual and procedural distractors and select the most specific case category.

Figure 9: Evidence-Grounded Processing example: quantitative financial reasoning. The effective-protection-rate task requires the model to identify the relevant variables, apply the correct formula, compute the tariff burden on imported inputs, and normalize by domestic value added.

Figure 10: Evidence-Grounded Processing example: multi-field extraction from a long legal record. The model must extract five evidence spans in a predefined order while ignoring unrelated monetary values, dates, and procedural events. The task contract evaluates field correctness, field ordering, delimiter validity, and agreement with the reference output.

Figure 11: Evidence-Grounded Processing example: cross-lingual company entity resolution. The task requires the model to determine whether Chinese and English company names refer to the same entity. The decision depends on the joint alignment of geographic information, transliterated name components, translated industry terms, and legal-entity suffixes.

Figure 12: Applied Review example: open-ended disputed-issue generation. The source case contains overlapping agency, letter-of-credit, guarantee, insurance, and payment relationships. The model must transform these facts into a structured set of issues covering the responsible parties, contractual instruments, disputed conduct, claimed amounts, interest and fee calculations, and alternative liability relationships.

Figure 13: Applied Review example: legal-judgment generation. The model must identify the appropriate legal route, establish the claimant’s standing, synthesize transaction and valuation evidence, determine whether the challenged asset transfer impaired creditors, address the disputed execution settlement, and produce an appropriate dispositive analysis.

Figure 14: Applied Review example: structured decision-view generation. The model must express the supported review position in a constrained dispositive schema that preserves the affected parties, monetary obligations, and procedural treatment. This example illustrates a structured output form within the decision-view generation contract rather than an additional downstream operation.

Figure 15: Direct request alignment ( $z_{t}=1$ ). The visible record leaves a proxy-registration or batch-onboarding concern unresolved. The model follows the recorded Ask branch and requests verification of that concern. The request matches a retained need supported by the later trajectory, so the state is both an Ask-action hit and a direct alignment hit. The observed-future panel was unavailable at inference.

Figure 16: Partial request alignment ( $z_{t}=0.5$ ). The model follows the recorded Ask branch, but the question captures only part of the retained storefront-authenticity need and remains underspecified relative to the closest verified target. Selecting Ask is an action hit; full request-alignment credit is not automatic. Credit is assigned by the best-matching retained target, so the model need not enumerate every item that later appeared.

Figure 17: Missed information acquisition ( $z_{t}=0$ ). The recorded state is Ask, but the model selects Proceed and issues no request. Because the Ask branch is not entered, no candidate request is compared with the retained target set and the state contributes zero to ERA. The state is an Ask-recall error even if proceeding from the visible record appears locally plausible.

Explore similar work

May 14, 2026

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

Large language models (LLMs) are increasingly being applied to financial analysis, reporting, investment decision support, risk management, compliance, and professional training. However, robust evaluation of their domain competence in finance remains incomplete. Widely used open benchmarks such as FinQA, ConvFinQA, and TAT-QA have played an important role in advancing financial question answering and numerical reasoning, but they focus primarily on question answering over financial reports and do not provide an explicit hierarchy of professional difficulty. Broader resources, including FinanceBench, PIXIU, FinBen, and FLaME, expand the coverage of financial tasks, yet the problem of evaluating the transition from foundational knowledge to expert-level financial reasoning remains open. In this work, we present FINESSE-Bench, a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in LLMs. FINESSE-Bench combines exam-oriented datasets inspired by professional certifications (CFA-like Levels 1-3, CMT-like Level 2, and CFTe-like Level 1), applied trading task collections, and a Russian-language olympiad benchmark. This design enables evaluation of domain breadth, performance degradation as difficulty increases, the ability to solve computational tasks, and model behavior in specialized financial domains. We also describe a unified evaluation protocol covering multiple-choice questions, numerical answers, and short open-ended responses, together with an automated scoring scheme for freeform answers based on the LLM-as-judge paradigm. FINESSE-Bench is intended both as a complement to existing open financial benchmarks and as a tool for more substantive evaluation of professionally relevant financial competencies in large language models.

Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov +4

Lime FinTech

LLM Evaluation Quantitative Finance Benchmark Design Finance

Ask PDF

Jul 30, 2026

Benchmarks Are Not Validation: A System-Level View of Financial LLM Applications

Large language models are increasingly deployed in financial applications that combine retrieval, proprietary data, tool use, orchestration logic, monitoring, and human escalation. Yet evaluation often remains model-centric: benchmark scores, task accuracy, or one-off qualitative reviews are treated as evidence of readiness. In financial settings, this is insufficient. We take the position that financial LLM systems should not be approved for production based on benchmark performance alone. They require system-level validation evidence across the application stack: data, model design, retrieval and generation performance, agent behavior, governance, and implementation. Drawing on industry experience validating GenAI applications in financial institutions, we outline a multi-layer validation view and explain why hybrid evaluation is necessary. We discuss where LLM-as-a-judge methods are useful and why they require controls such as multiple judges, rubrics, agreement, and auditability checks. We also highlight failure modes poorly captured by static benchmarks, including retrieval failures, unfaithful generation, tool misuse, escalation errors, and operational instability. Our position is that financial LLM validation should be an ongoing system discipline rather than a one-time model scoring exercise. Validation should produce decision-ready evidence, not only scores. We conclude with a research agenda for system-aware benchmarks, agent trace validation, judge alignment protocols, and lifecycle validation standards.

Burak Payzun, İrem Demirtaş, Simona Scala +2

Prometeia S.p.A.

Financial Services LLM Evaluation Finance

Ask PDF

Feb 25, 2026

FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation

We introduce FIRE, a comprehensive benchmark designed to evaluate both the theoretical financial knowledge of LLMs and their ability to handle practical business scenarios. For theoretical assessment, we curate a diverse set of examination questions drawn from widely recognized financial qualification exams, enabling evaluation of LLMs deep understanding and application of financial knowledge. In addition, to assess the practical value of LLMs in real-world financial tasks, we propose a systematic evaluation matrix that categorizes complex financial domains and ensures coverage of essential subdomains and business activities. Based on this evaluation matrix, we collect 3,000 financial scenario questions, consisting of closed-form decision questions with reference answers and open-ended questions evaluated by predefined rubrics. We conduct comprehensive evaluations of state-of-the-art LLMs on the FIRE benchmark, including XuanYuan 4.0, our latest financial-domain model, as a strong in-domain baseline. These results enable a systematic analysis of the capability boundaries of current LLMs in financial applications. We publicly release the benchmark questions and evaluation code to facilitate future research.

Xiyuan Zhang, Huihang Wu, Jiayu Guo +8

Du Xiaoman Technology · The PBC School of Finance, Tsinghua University · The School of Finance, Renmin University of China

LLM Evaluation Financial QA

Ask PDF