FinDialogLens: Event Extraction over Multi-Party Dialogue for Missed-Trade Identification in Financial Chatrooms

Chin-Lun Fu; Hong Ni; Behrouz Madahian

cs.CLOct 1, 2026

FinDialogLens: Event Extraction over Multi-Party Dialogue for Missed-Trade Identification in Financial Chatrooms

Authors: Chin-Lun Fu, Hong Ni, Behrouz Madahian

Organizations: Machine Learning Center of Excellence, JPMorgan Chase & Co.

Abstract

Multi-party financial chatrooms are vital for sales-and-trading professionals, but their complexity makes manual recovery of missed trades infeasible: each Request for Quote (RFQ) is an event whose final price and trade outcome appear many messages after the RFQ-trigger message (the inquiry message), interleaved with concurrent RFQs from other participants. We cast this as event extraction (EE) over multi-party dialogue and present FinDialogLens, a hybrid LLM pipeline in which compact fine-tuned classifiers act as inference-time scaffolds: they detect RFQ-triggers and price/trade outcome metadata, an RFQ-Level Module segments per-event RFQ windows, and a Trade Engine fills argument roles. With GPT-4o, FinDialogLens reaches 92.1% and 94.3% accuracy on final price and trade outcome, respectively, outperforming full-chatroom CoT prompting methods; fine-tuned open-source LLMs with as few as 3B parameters achieve comparable performance with modest in-domain data. To make the LLM-based solution practical at scale, a difficulty-aware router balances cost and accuracy by allocating RFQs between a low-cost rule-based engine and the higher-performing LLM-powered Trade Engine, cutting LLM calls by 85% on final price while recovering half of the accuracy gap to FinDialogLens (GPT-4o), saving over $300/day at our 70,000-RFQ/day scale.

Figures & tables

Figure 1: Overview of the FinDialogLens pipeline for event extraction over multi-party dialogue. The Message-Level Module produces message-level metadata through RFQ-trigger detection, trade outcome classification, and price NER. The RFQ-Level Module uses these predictions to segment per-event RFQ windows from the chatroom. The Trade Engine performs LLM-based argument role filling over each RFQ window.

Method family	Multi-	Cross-	Trigger-	Event
	party	turn	anchored	record
Sentence EE	$\times$	$\times$	$\checkmark$	$\checkmark$
Doc-level EE	$\times$	$\checkmark$	$\times$	$\checkmark$
Dialogue disent.	$\checkmark$	$\checkmark$	–	$\times$
Ours	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Structural axes along which existing EE method families differ from our application setting.

Figure 2: Difficulty-Aware Router. For each RFQ event, the router assigns easier cases to rule-based engine and escalates harder cases to the LLM-powered engine.

Method	w/ Metadata	Input Context		Trade Engine	Final Price	Trade Outcome
Method	w/ Metadata	Whole Day	RFQ Windows	Trade Engine	Acc	Acc	F1
Rule-based Engine	$\checkmark$	$\checkmark$		Rule-based	77.1	90.3	90.7
CoT Prompting		$\checkmark$		GPT-4o (Zero-shot)	60.9	65.5	77.1
		$\checkmark$		GPT-4o (Few-shot)	64.4	70.5	80.5
	$\checkmark$	$\checkmark$		GPT-4o (Zero-shot)	73.9	85.9	91.5
	$\checkmark$	$\checkmark$		GPT-4o (Few-shot)	79.2	87.0	92.3

Table 2: Comparison of final price and trade outcome extraction performance across the Rule-based Engine, full-chatroom CoT prompting baselines, and FinDialogLens. “w/ Metadata” indicates whether the input context is augmented with predicted RFQ-trigger, price, and trade-outcome tags from the Message-Level Module. “Input Context” specifies whether the model consumes the full daily chatroom or per-event RFQ windows produced by the RFQ-Level Module.

Figure 3: Accuracy of fine-tuned open-source LLMs on final price and trade outcome extraction as training-set size varies. Dashed lines mark FinDialogLens (GPT-4o) and zero-shot CoT Prompting (GPT-4o) baseline.

Model	Precision	Recall	F1
RFQ Identifier	0.97	0.97	0.97
Trade Outcome Classifier	0.96	0.93	0.95
Price NER Model	0.91	0.95	0.93

Table 3: Performance of the Message-Level Module, including the RFQ Identifier, Trade Outcome Classifier, and Price NER Model.

Router Type	Final Price		Trade Outcome
Router Type	CPT (50%)	CPT (80%)	CPT (50%)	CPT (80%)
Random	49.2%	81.3%	52.1%	75.8%
RoBERTa-base	14.9%	32.4%	29.5%	62.6%
Mistral-7B-Instruct	16.4%	31.8%	26.7%	36.3%
Flan-T5-XL	16.1%	29.9%	25.1%	32.8%

Table 4: Call-Performance Threshold (CPT) for different router types. CPT is the fraction of RFQ windows routed to the LLM-powered Trade Engine required to recover a target percentage of the gap between the rule-based weak engine and FinDialogLens (GPT-4o). Bold indicates the best (lowest) value per column.

Figure 4: Performance of routers under various budget constraints ( $\alpha$ ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Difficulty	Count	Percentage
Easy: 1 RFQ in window	598	61.9%
Hard: $\geq$ 2 RFQs in window	368	38.1%
Total	966	100.0%

Appendix

Table 5: Distribution of test RFQs by the number of gold RFQ-trigger messages in the RFQ window. Hard cases correspond to interleaved RFQ windows containing two or more RFQs.

Figure 5: Example of an edge case in financial dialogue with three RFQs (messages 1, 2, and 8) occurring within a short time frame. Messages are highlighted in different colors to clarify which replies correspond to each RFQ event.

Figure 6: Accuracy of the rule-based engine on per-event RFQ windows grouped into deciles by router logit ranking.

Router Type	Final Price		Trade Outcome
Router Type	CPT (50%)	CPT (80%)	CPT (50%)	CPT (80%)
Random	$2.66	$4.39	$2.81	$4.09
RoBERTa-base	$0.80	$1.75	$1.59	$3.38
Mistral-7B-Instruct	$0.89	$1.72	$1.44	$1.96
Flan-T5-XL	$0.87	$1.61	$1.36	$1.77

Appendix

Table 6: Cost analysis of different router types for final price and trade outcome under CPT (50%) and CPT (80%). Reported values are USD per 1,000 RFQs; processing 1,000 RFQs with GPT-4o costs approximately $5.40.

Router Type	Final Price		Trade Outcome
Router Type	CPT (50%)	CPT (80%)	CPT (50%)	CPT (80%)
Random	717.8	1186.2	760.1	1105.9
RoBERTa-base	217.4	427.7	430.4	913.3
Mistral-7B-Instruct	239.3	464.0	389.6	529.6
Flan-T5-XL	234.9	436.2	366.2	478.6

Appendix

Table 7: Latency analysis of different router types for final price and trade outcome under CPT (50%) and CPT (80%). Reported values are total LLM API latency (seconds) for the full test set; processing the full test set with GPT-4o alone takes approximately 1,459 seconds.

Method	w/ Metadata	Input Context		Trade Engine	Final Price	Trade Outcome
Method	w/ Metadata	Whole Day	RFQ Windows	Trade Engine	Acc	Acc	F1
CoT Prompting		✓		GPT-4o (Zero-shot)	$62.2_{\pm 0.95}$	$66.7_{\pm 0.94}$	$76.7_{\pm 0.37}$
		✓		GPT-4o (Few-shot)	$66.1_{\pm 2.17}$	$67.5_{\pm 2.47}$	$77.2_{\pm 2.09}$
	✓	✓		GPT-4o (Zero-shot)	$76.4_{\pm 1.73}$	$84.4_{\pm 1.32}$	$89.7_{\pm 1.40}$
	✓	✓		GPT-4o (Few-shot)	$79.2_{\pm 0.59}$	$85.2_{\pm 0.81}$	$90.3_{\pm 0.93}$
FinDialogLens	✓		✓	GPT-4o	$\mathbf{91.8}_{\pm 0.42}$	$\mathbf{93.4}_{\pm 0.27}$	$\mathbf{93.5}_{\pm 0.38}$

Appendix

Table 8: Multi-run results for GPT-4o-based methods. We report mean accuracy/F1 over five runs, with standard deviation in subscript. All GPT-4o runs use temperature-0 decoding.

Method	Final Price	Trade Outcome
Method	Acc	Acc	F1
FinDialogLens (GPT-4o)	92.1	94.3	94.5
FinDialogLens (GPT-4o) + oracle message-level metadata	91.9	95.9	95.7

Appendix

Table 9: Ablation study on metadata error propagation. We compare the original FinDialogLens pipeline using predicted message-level metadata with an oracle setting that replaces predicted metadata with gold message-level labels.

Figure 7: Examples of the two types of Input Context : full daily chatroom and per-event RFQ window, supplied to the Trade Engine. Message-level metadata from the Message-Level Module is interleaved with the dialogue text using special tokens ( $\ll$ rfq $\gg$ , $\ll$ hit $\gg$ , $\ll$ miss $\gg$ , $\ll$ price $\gg$ ). CoT prompting baselines consume the full daily chatroom; FinDialogLens consumes per-event RFQ windows produced by the RFQ-Level Module.

Figure 8: Prompt template for CoT prompting methods.

Figure 9: Prompt template for the Trade Engine within FinDialogLens.

Figure 10: Prompt template for the difficulty-aware router.

Document Information Extraction

Document Information Extraction turns text, tables, forms, and images into structured, searchable data.

See all 290 papers

PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval
Nov 18, 2025Chun Chet Ng, Jia Yu Lim, Zhen Hao Chu +2Information Retrieval LLM Reranking
AI Lens, Kuala Lumpur, Malaysia
FinTradeBench: A Financial Reasoning Benchmark for LLMs
Mar 19, 2026Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan +2Financial Reasoning in Language Models Financial Reasoning Benchmarks
University of Central Florida
Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models
Apr 18, 2026Yuchen Pan, Soung Chang LiewLLM Evaluation Financial Reasoning in Language Models
Department of Information Engineering, The Chinese University of Hong Kong

Nov 18, 2025J/K move · Enter open · S save

PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

Chun Chet Ng, Jia Yu Lim, Zhen Hao Chu, Yixi Zhou+1

AI Lens, Kuala Lumpur, Malaysia

Information Retrieval LLM Reranking Financial Language Model Evaluation In-Context Learning Financial Reasoning Benchmarks

Open paper AskPDF arXiv

Deploying LLMs for financial information retrieval demands accuracy, cost efficiency, and reproducibility, yet most approaches require expensive fine-tuning. PRISM is a training-free framework that composes three deployable modules (prompt engineering, in-context learning, and a candidate refinement module) with a boundary-condition study of multi-agent coordination for financial document retrieval. On FiQA-2018, stacked reranking and

k

-reciprocal encoding raise Recall@100 from 0.7783 to 0.8326. On FinAgentBench, a Cohere-refined top-50 pool yields a higher observed end-to-end NDCG@5, below this evaluation's minimum detectable effect, while cutting per-query cost by about 35%. Added complexity does not always pay off:

k

-reciprocal encoding stops helping once the upstream stage is already high-recall, and multi-agent coordination is not a default for fine-grained chunk ranking. Matched pairs that hold prompt, model, and exemplars fixed lose 0.0702 and 0.0681 combined NDCG@5 to their single-path controls, and the whole gap sits at the chunk stage. We release an expert-annotated FinAgentBench evaluation set, together with a label-robustness sweep over 16 alternative ground-truth constructions, restoring a reproducible evaluation path after the original evaluator went offline. Without fine-tuning, PRISM placed third on the official FinAgentBench private leaderboard (leaderboard basis, NDCG@5 0.71181), reaches NDCG@10 0.6197 on FiQA-2018, and 98% accuracy on FinanceBench. We report per-query latency, token, and dollar accounting across roughly 40 configurations, present the quality-cost-latency Pareto frontier, and introduce FACET, a transparent SLA-specific scalarization for choosing among non-dominated configurations. Code and dataset are released at https://bit.ly/prism-ailens.

Abstract

Document Information Extraction

Explore similar work

PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

FinTradeBench: A Financial Reasoning Benchmark for LLMs

Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models