PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

Chun Chet Ng; Jia Yu Lim; Zhen Hao Chu; Yixi Zhou; Wei Zeng Low

cs.AINov 18, 2025

PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

Authors: Chun Chet Ng, Jia Yu Lim, Zhen Hao Chu, Yixi Zhou, Wei Zeng Low

Organizations: AI Lens, Kuala Lumpur, Malaysia

Abstract

Deploying LLMs for financial information retrieval demands accuracy, cost efficiency, and reproducibility, yet most approaches require expensive fine-tuning. PRISM is a training-free framework that composes three deployable modules (prompt engineering, in-context learning, and a candidate refinement module) with a boundary-condition study of multi-agent coordination for financial document retrieval. On FiQA-2018, stacked reranking and $k$ -reciprocal encoding raise Recall@100 from 0.7783 to 0.8326. On FinAgentBench, a Cohere-refined top-50 pool yields a higher observed end-to-end NDCG@5, below this evaluation's minimum detectable effect, while cutting per-query cost by about 35%. Added complexity does not always pay off: $k$ -reciprocal encoding stops helping once the upstream stage is already high-recall, and multi-agent coordination is not a default for fine-grained chunk ranking. Matched pairs that hold prompt, model, and exemplars fixed lose 0.0702 and 0.0681 combined NDCG@5 to their single-path controls, and the whole gap sits at the chunk stage. We release an expert-annotated FinAgentBench evaluation set, together with a label-robustness sweep over 16 alternative ground-truth constructions, restoring a reproducible evaluation path after the original evaluator went offline. Without fine-tuning, PRISM placed third on the official FinAgentBench private leaderboard (leaderboard basis, NDCG@5 0.71181), reaches NDCG@10 0.6197 on FiQA-2018, and 98% accuracy on FinanceBench. We report per-query latency, token, and dollar accounting across roughly 40 configurations, present the quality-cost-latency Pareto frontier, and introduce FACET, a transparent SLA-specific scalarization for choosing among non-dominated configurations. Code and dataset are released at https://bit.ly/prism-ailens.

Figures & tables

Figure 1: Overview of the PRISM framework. A query passes left to right through five stages. (1) Prompt engineering encodes financial-domain priors and a relevance scaffold into the ranking instruction (1a), and ICL exemplar retrieval prepends demonstrations from a separate training-split store (1b), never changing the candidate set. (2) Candidate refinement queries the corpus for the current query and shrinks the candidate set through dense retrieval, reranking, and optional $k$ -reciprocal encoding. The two retrieval paths are drawn separately because they query different indexes and move context in opposite directions (Section 4.2 ). (3) Prompt, exemplars, refined candidates, and query are assembled and (4) ranked in a single LLM call, giving (5) the ranked output. Multi-agent coordination replaces stage 4 when enabled; it is optional and is studied here as a boundary condition, not as the recommended production path for chunk ranking (Section 5.3 ).

Task	Experts	Pair	$\boldsymbol{\tau_{b}}$	RBO@5	$\boldsymbol{\kappa}$
Document ranking	3	1, 2	0.754	0.957	0.827
Chunk ranking	4	1, 4	0.582	0.730	0.649
Mean absolute NDCG@5 difference from original scores					0.0702
Ordering vs. leaderboard, $\boldsymbol{\rho}$ / $\boldsymbol{\tau_{b}}$ ( $n=34$ )					0.7800 / 0.6613

Table 1: Summary of the released FinAgentBench evaluation labels. Agreement is for the selected annotator pair; validation compares all eligible configurations with original private-leaderboard scores.

Architecture-matched pairs ( $P_{4}$ , GPT-5-mini, ICL held fixed)
ID	Configuration	Combined NDCG@5	95% CI
39	grok-4-20-reasoning	0.8370	[0.8190, 0.8541]
37	DeepSeek-V4-Flash	0.8016	[0.7787, 0.8226]
18	strongest GPT-5 family	0.7841	[0.7637, 0.8044]
19	GPT-5, competition submission	0.7738	[0.7528, 0.7944]
16 vs. 33	single-path vs. $A_{4}$ agentic	$+0.0702$	[ $+0.0474$ , $+0.0937$ ]

Table 2: Configurations the claims rest on, using bootstrap re-scores on the re-annotated basis with query-level 95% CIs (10,000 resamples) over the 200 document-ranking and 199 chunk-ranking scoring queries; the point estimates elsewhere use the 200-query convention, so values differ in the fourth decimal. Run 19 is the competition submission, not the best row. Full grid in Appendix Table 23 .

Matched pair	Document NDCG@5	Chunk NDCG@5
Run 16 vs. Run 33	0.94594 vs. 0.95359 ( $p=0.32$ )	0.59099 vs. 0.44364
Run 17 vs. Run 34	0.94950 vs. 0.95259 ( $p=0.20$ )	0.55340 vs. 0.41476

Table 3: Where the coordination penalty sits. Matched on prompt, model, and ICL; re-annotated basis. Document differences are not distinguishable from zero, and the whole gap is at the chunk stage.

Figure 2: Quality–cost–latency dominance over the configurations with measured cost and latency (combined NDCG@5, re-annotated basis, against USD per query on a log axis; marker size grows with latency). Five of 11 are non-dominated in this projection and six on all three axes; the six filled markers are that three-axis set, and the three highest FACET scores are all Pareto-optimal.

Constraint	Choose	Lat.	$/q	NDCG	FACET
Quality first	39, grok-4	48.9	0.1415	0.8372	1.221
Quality per $	37, DeepSeek	18.5	0.0284	0.8016	2.349
Latency, $Q{>}0.75$	40, Llama-4	17.0	0.0133	0.7627	2.923
Lowest $/query	38, gpt-oss	15.4	0.0083	0.6582	3.195
Submitted run	19, GPT-5	84.6	0.1462	0.7747	1.042

Table 4: Practitioner decision table. Distinct operating points under different constraints, not a ranking. Run IDs match Appendix Table 23 ; models are grok-4-20-reasoning, DeepSeek-V4-Flash, Llama-4-Maverick-17B-128E-Instruct-FP8, gpt-oss-120b, and GPT-5. NDCG@5 is combined, re-annotated basis; latency (s) and cost are per query. Run 19 uses combined document–chunk per-query accounting; post-annotation Runs 37–40 retain their originally reported latency and cost measurements.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Non-Agentic Workflow
ID	New NDCG@5			Orig. NDCG@5	Difference
ID	Doc.	Chunk	Total	Orig. NDCG@5	Difference
1	0.93857	0.62470	0.78164	0.66666	0.11498
2	0.93288	0.58229	0.75759	0.66628	0.09131
3	0.93288	0.62470	0.77879	0.66328	0.11551
4	0.93857	0.58229	0.76043	0.66966	0.09077

Appendix

Table 5: Validation of the re-annotated FinAgentBench labels: new NDCG@5 vs. the original private-leaderboard score. Runs 33–34 and 37–40 were not evaluated on the original test set and are therefore excluded. Run IDs match Appendix Table 23 .

Document ranking (3 experts)
Annotator pair		Kendall $\tau_{b}$	RBO@5	Cohen’s $\kappa$
Annot. 1	Annot. 2	0.7540	0.9567	0.8270
Annot. 1	Annot. 3	0.6520	0.9347	0.7335
Annot. 3	Annot. 2	0.7420	0.9541	0.8095
Chunk ranking (4 experts)
Annot. 1	Annot. 4	0.5822	0.7296	0.6485

Appendix

Table 6: Pairwise inter-annotator agreement among domain experts on FinAgentBench.

Figure 3: Conclusions under the 16 alternative label constructions. Left: Run 39 (grok-4) minus the best GPT-5 configuration, positive in 15 of 16. Middle: best non-agentic minus best agentic, positive in 16 of 16. Right: the two architecture-matched pairs with prompt, model, and ICL held fixed, positive in 16 of 16.

Doc. Type	Rank 1	Rank 2	Rank 3	Rank 4	Rank 5
10-K	2402	1546	507	207	324
Earnings	1360	726	948	893	1059
10-Q	554	1558	1427	866	581
8-K	97	407	1091	1808	1583
DEF 14A	573	749	1013	1212	1439

Appendix

Table 8: Distribution of FinAgentBench document types across ranked positions in the document-ranking split.

DEF 14A		10-K		10-Q		8-K		Earnings
Keyword	Ratio	Keyword	Ratio	Keyword	Ratio	Keyword	Ratio	Keyword	Ratio
dependency	88.40%	evolved	40.98%	quarter	20.00%	compensation	95.45%	asked	87.68%
concentration	88.24%	recurring	40.57%	recurring	6.29%	burn	41.74%	questions	86.56%
exist	87.73%	ratio	39.40%	evolved	6.12%	award	41.74%	metrics	78.26%
risks	80.51%	time	39.22%	time	5.99%	manage	41.54%	customer	74.62%
market	60.77%	reporting	38.19%	ratio	5.97%	availability	40.60%	guidance	52.97%

Appendix

Table 9: Top 6 keywords appearing in queries where the respective document type was retrieved at Rank 1 in FinAgentBench. The ratio indicates the frequency of the keyword relative to the entire corpus.

Figure 4: Token count distribution for relevant and irrelevant chunks in the FinAgentBench chunk-ranking split.

Figure 5: Word count distribution for relevant and irrelevant chunks in the FinAgentBench chunk-ranking split.

Figure 6: Distribution of relevant documents per query in FiQA-2018.

10-K		10-Q		8-K		Earnings
Keyword	Ratio	Keyword	Ratio	Keyword	Ratio	Keyword	Ratio
securely	99.00%	mun	98.61%	accession	96.55%	twentieth	58.90%
restatement	99.00%	min	85.71%	chargeable	80.00%	fragrance	43.33%
athletic	99.00%	noncompensation	83.13%	survivorship	77.27%	recalculate	42.31%
george	98.99%	aum	80.95%	abstain	77.27%	translational	40.24%
budgetary	98.99%	logged	80.77%	unbudgeted	74.36%	eighteenth	40.00%

Appendix

Table 10: Top 6 keywords by document type in FinanceBench; ratios show corpus-relative frequency.

Pipeline	NDCG@5	MAP@5	MRR@5	$/query	Lat. (s)
Baseline (Run 19)	0.61278	0.48092	0.83758	0.2778	136.7
TE3-S (top 100)	0.61839	0.47800	0.84358	0.2336	192.0
TE3-S (top 75)	0.61298	0.46744	0.83908	0.2214	171.9
TE3-S (top 50)	0.61070	0.45060	0.82450	0.1845	142.4
TE3-S $\to$ Cohere (top 75)	0.62418	0.47302	0.84450	0.2146	177.6
TE3-S $\to$ Cohere (top 50)	0.63408	0.48719	0.85042	0.1813	146.3

Appendix

Table 11: End-to-end effect of the candidate-refinement module on FinAgentBench chunk ranking under PRISM settings. Cost and latency are per query.

Retriever	Recall@100	Recall@200	Recall@300	NDCG@10	MAP@10	MRR@10
TE3-S	0.7783	0.8423	0.8775	0.54988	0.36847	0.51945
BGE-M3 (dense)	0.7088	0.7800	0.8086	0.52972	0.33316	0.50064
BGE-M3 (sparse)	0.5133	0.5838	0.6242	0.36922	0.20623	0.32929
SPLADE-v3	0.6683	0.7301	0.7615	0.49281	0.30611	0.45961
BM25	0.3590	0.4223	0.4652	0.22782	0.11652	0.19854

Appendix

Table 12: Embedding comparison on FiQA-2018 top-1000 pool.

Embedding	GPT-5	gpt-oss-120b	DeepSeek-V4-Flash	grok-4-20-reasoning	Llama-4-Maverick-17B-128E-Instruct-FP8
TE3-S	136 (90.7)	136 (90.7)	109 (72.7)	142 (94.7)	86 (57.3)
Cohere embed-english-v3.0	136 (90.7)	130 (86.7)	106 (70.7)	144 (96.0)	90 (60.0)
Cohere embed-v4.0	135 (90.0)	131 (87.3)	108 (72.0)	134 (89.3)	86 (57.3)

Appendix

Table 13: Embedding backbone selection on FinanceBench (150 QA). Cells show correct answers (accuracy %) under shared-store conditions with prompt

P_{4}

, ICL-9.

Method	Recall@100	Recall@200	Recall@300	NDCG@10	MAP@10	MRR@10
Cohere rerank-v4.0-fast	0.8115	0.8587	0.8940	0.6258	0.4370	0.6028
BGE reranker v2 m3	0.7439	0.8061	0.8483	0.5431	0.3465	0.5114
BGE-M3 ColBERT	0.7270	0.7960	0.8366	0.5401	0.3467	0.5120
ColBERTv2	0.6690	0.7474	0.7928	0.4747	0.2865	0.4338
Jina reranker v3	0.6497	0.7692	0.8209	0.4218	0.2244	0.3759

Appendix

Table 14: Reranker comparison on the FiQA-2018 TE3-S top-1000 pool.

Pipeline	Recall@100	NDCG@10	MAP@10	MRR@10
TE3-S	0.77830	0.54988	0.36847	0.51945
TE3-S $\to$ Cohere	0.81150	0.62580	0.43704	0.60276
TE3-S $\to$ Cohere $\to$ GPT-5	0.83066	0.62426	0.43550	0.60122
TE3-S $\to$ Cohere $\to$ $k$ -recip	0.83260	0.63333	0.45002	0.60058

Appendix

Table 15: End-to-end effect of the candidate-refinement module on FiQA-2018 (TE3-S top-1000 pool).

Method	Recall@25	Recall@50	Recall@75	NDCG@5	MAP@5	MRR@5
Cohere rerank-v4.0-fast	0.7850	0.8725	0.9088	0.5464	0.4070	0.7632
BGE reranker v2 m3	0.5975	0.7600	0.8675	0.3211	0.2330	0.5197
BGE-M3 ColBERT	0.6500	0.7875	0.8838	0.3115	0.2155	0.5113
ColBERTv2	0.5825	0.7500	0.8675	0.2928	0.2053	0.5218
Jina reranker v3	0.7200	0.8250	0.8888	0.4508	0.3071	0.6692

Appendix

Table 16: Reranker comparison on FinAgentBench chunk ranking.

Upstream reranker	Recall@100 alone	$\boldsymbol{+\,k}$ -recip	Gain
Jina reranker v3	0.6497	0.7809	$+0.131$
ColBERTv2	0.6690	0.7520	$+0.083$
BGE-M3 ColBERT	0.7270	0.7910	$+0.064$
BGE reranker v2 m3	0.7439	0.7896	$+0.046$
Cohere rerank-v4.0-fast	0.8115	0.8326	$+0.021$

Appendix

Table 17: The gain from adding

k

-reciprocal encoding declines monotonically as upstream recall rises. FiQA-2018 TE3-S top-1000 pool, Recall@100.

$k$ -reciprocal $\to$ reranker (before)
Method	Recall@25	Recall@50	Recall@75	Recall@100	NDCG@5	MAP@5	MRR@5
TE3-S $\to$ $k$ -recip (only)	0.5925	0.7850	0.8875	0.9325	0.28638	0.20450	0.44708
TE3-S $\to$ $k$ -recip $\to$ Cohere rerank-v4.0-fast	0.7713	0.8575	-	-	0.54171	0.40515	0.75558
TE3-S $\to$ $k$ -recip $\to$ BGE-M3 ColBERT	0.6625	0.8075	-	-	0.31584	0.21854	0.52108
TE3-S $\to$ $k$ -recip $\to$ BGE reranker v2 m3	0.5975	0.7863	-	-	0.32410	0.23556	0.52400
TE3-S $\to$ $k$ -recip $\to$ Jina reranker v3	0.7188	0.8250	-	-	0.44356	0.30575	0.64333

Appendix

Table 18: Stacking

k

-reciprocal encoding with rerankers on FinAgentBench chunk ranking (TE3-S top-100).

$k$ -reciprocal $\to$ reranker (before)
Method	Recall@100	Recall@200	Recall@300	NDCG@10	MAP@10	MRR@10
TE3-S $\to$ $k$ -recip (only)	0.7824	0.8484	0.8839	0.53021	0.34970	0.48821
TE3-S $\to$ $k$ -recip $\to$ Cohere rerank-v4.0-fast	0.8140	0.8687	-	0.62853	0.44082	0.60662
TE3-S $\to$ $k$ -recip $\to$ BGE-M3 ColBERT	0.7520	0.8356	-	0.54163	0.34939	0.51403
TE3-S $\to$ $k$ -recip $\to$ BGE reranker v2 m3	0.7690	0.8490	-	0.54820	0.35141	0.51668
TE3-S $\to$ $k$ -recip $\to$ Jina reranker v3	0.7755	0.8595	-	0.49708	0.28377	0.45946

Appendix

Table 19: Stacking

k

-reciprocal encoding with rerankers on FiQA-2018 (TE3-S top-1000).

Prompt	Run	Re-annotated	Original private
$P_{1}$	2	0.75759	0.66628
$P_{2}$	9	0.76649	0.66774
$P_{3}$	12	0.76287	0.69019
$P_{4}$	13	0.77668	0.68895

Appendix

Table 20: Prompt variants matched on model and ICL (GPT-5-mini at both stages, no exemplars), on both evaluation bases. Combined NDCG@5.

	ICL retriever	Candidate-refinement retriever
Index queried	training-split exemplar store	document or chunk corpus for the current query
Output	few-shot demonstrations prepended to the prompt	candidates to be ranked
Changes the candidate set?	no	yes
Role	task adaptation	pool construction and context compression

Appendix

Table 21: The two retrievers in PRISM share an embedding backbone but query different indexes and act at different points.

Panel of Figure 7 — Figure 7: Multi-Agent System architectures for PRISM. (a) Document ranking workflow. (b)–(e) Chunk ranking variants.

Rank	Team	NDCG@5 Score (Private)	Code Released	Fine-tuning
1	memex	0.72497	N/A	Yes
2	Yuzhen Hu	0.72328	N/A	Yes
3	Ours	0.71181 $\dagger$	On acceptance	No
4	rlawnsxo	0.70601	N/A	N/A
5	currio	0.70195	N/A	N/A
6	kaggleslasher	0.69985	N/A	N/A

Appendix

Table 22: FinAgentBench leaderboard results.

Closed-Source Models
Prompt	Model	ICL	NDCG@10	Recall@100
N/A	BM25+CE	N/A	0.3470	0.5390
N/A	TE3-S	N/A	0.5499	0.7783
N/A	BGE-M3 (dense)	N/A	0.5297	0.7088
N/A	BGE-M3 (sparse)	N/A	0.3692	0.5133
N/A	SPLADE-v3	N/A	0.4928	0.6683

Appendix

Table 25: FiQA evaluation results.

Method	Model	NDCG@10	Recall@100
BM25	Baseline	0.2360	0.5390
DeepCT	Sparse	0.1910	0.4890
SPARTA	Sparse	0.1980	0.4460
docT5query	Sparse	0.2910	0.5980
DPR	Dense	0.1120	0.3420
ANCE	Dense	0.2950	0.5810

Appendix

Table 26: Traditional baseline performance on FiQA-2018.

Closed-Source Models
Prompt	Model	ICL	NDCG@10	Recall@100
$P_{1}$	GPT-4.1	N/A	0.4307	0.6024
$P_{1}$	GPT-5-mini	N/A	0.5565	0.7268
$P_{1}$	GPT-5	N/A	0.6130	0.8045
$P_{1}$	GPT-4.1	ICL-5	0.5320	0.7247
$P_{1}$	GPT-5-mini	ICL-5	0.5171	0.6412

Appendix

Table 27: PRISM performance on FiQA-2018.

Closed-Source Models
Prompt	Model	Correct	Incorrect	Unable
Baseline	GPT-5	141	9	0
$P_{1}$	GPT-5	147	3	0
$P_{2}$	GPT-5	142	8	0
$P_{3}$	GPT-5	140	10	0
$P_{4}$	GPT-5	146	4	0

Appendix

Table 28: FinanceBench oracle evaluation results.

Prompt	Model	Eval Mode	ICL	Results (Out of 150)
Prompt		Eval Mode	ICL	Correct	Incorrect	Unable to Answer
Baselines
Baseline	GPT-4.1	Single Store	N/A	132	18	0
Baseline	GPT-4.1	Shared Store	N/A	124	26	0
Baseline	GPT-4.1	In Context	N/A	129	21	0
Baseline	GPT-4.1	Oracle	N/A	133	16	1

Appendix

Table 29: Comprehensive evaluation results of different prompt versions and models across various evaluation modes on FinanceBench.

Figure 8: Latency distribution box plots, where each box shows how the latency varies across runs with different LLMs and configurations. For chunk ranking tasks, we observe that

P_{3}

(Run 12) has the highest median latency for both tasks due to its heavy reasoning prompt design, while

P_{4}

(Runs 13–19) achieves a good balance between latency and performance.

Figure 9: Token distribution bar charts, where each bar details the proportion of prompt and completion token usage across different runs and tasks, highlighting that chunk ranking tasks use more prompt tokens due to longer context and complexity.

Table 31: Latency, token usage and cost for the candidate-refinement module on FinAgentBench chunk ranking. Cost is API cost: local rerankers and

k

-reciprocal incur no API cost, while commercial rerankers are counted where used. (a) Component stages emit no LLM completion tokens. (b) End-to-end pipelines pass the refined pool to

P_{4}

/GPT-5 with ICL-5 and report both prompt and completion tokens.

Table 33: Latency, token usage and cost for the candidate-refinement module on FiQA-2018. Cost is API cost: local rerankers and

k

-reciprocal incur no API cost, while commercial rerankers are counted where used. (a) Component stages emit no LLM completion tokens. (b) The end-to-end pipeline passes the refined pool to GPT-5 and reports both prompt and completion tokens.

Prompt	Model	Eval Mode	ICL	Total Prompt	Total Completion	Total	Time Per	Total Cost
Prompt		Eval Mode	ICL	Tokens (K)	Tokens (K)	Time (s)	Query (s)	(USD)
Baselines
Baseline	GPT-4.1	Single Store	N/A	4.81	27.13	494.94	3.30	0.23
Baseline	GPT-4.1	Shared Store	N/A	4.81	27.72	478.99	3.19	0.23
Baseline	GPT-4.1	In Context	N/A	1260.12	40.70	2466.77	16.45	25.53
Baseline	GPT-4.1	Oracle	N/A	159.80	27.08	329.78	2.20	0.54

Appendix

Table 34: Comprehensive latency, token usage and cost statistics for different prompt versions and models across various evaluation modes on FinanceBench.

Run ID	n	Mean	SD	CV (%)	CI-Low	CI-High
12	5	0.68005	0.01023	1.50489	0.66734	0.69276
15	4	0.70246	0.00661	0.94078	0.69194	0.71297
18	4	0.70660	0.00546	0.77319	0.69790	0.71529
19	9	0.71163	0.00433	0.60861	0.70830	0.71496

Appendix

Table 35: Descriptive statistics across multiple runs.

Comparison	$t$ -stat	$p$ -val	Signif.
12 vs 15	-3.969	0.0057	Yes
12 vs 18	-4.981	0.0022	Yes
12 vs 19	-6.582	0.0014	Yes

Appendix

Table 36: Welch’s

t

-tests vs. Run 12.

Figure 10: Paired query-level bootstrap differences in combined NDCG@5 with 95% confidence intervals, 10,000 resamples, re-annotated basis. Four model comparisons and the two architecture-matched pairs; only DeepSeek against Run 18 crosses zero.

ID	Configuration	ICL	Lat. (s)	$/query	NDCG@5	FACET
2	P1/GPT-5-mini	none	28.1	0.0124	0.7576	2.695
9	P2/GPT-5-mini	none	23.1	0.0123	0.7665	2.824
12	P3/GPT-5-mini	none	57.7	0.0226	0.7629	1.956
13	P4/GPT-5-mini	none	103.3	0.0214	0.7767	1.786
15	P4/GPT-5	none	76.1	0.1448	0.7776	1.070
17	P4/GPT-5-mini	ICL-5 both	75.6	0.0407	0.7515	1.541

Appendix

Table 38: FACET on FinAgentBench for the configurations with measured cost and latency (

L_{\mathrm{ceil}}{=}30

C_{\mathrm{ceil}}{=}\

0.25

,

Q_{\mathrm{floor}}{=}0.65

;

Q=$ combined NDCG@5, re-annotated basis). Each run applies the same prompt and model to both ranking tasks; IDs match Appendix Table 23 , and the per-axis ratios are in Appendix Table 41 . Runs 2–19 use combined document–chunk per-query accounting; Runs 38–40 retain post-annotation chunk-stage totals divided by 400 tasks, while Run 37 retains its originally reported measurements.

ID	Configuration	Lat. (s)	$/q	NDCG@5	Non-dom.
38	P4/gpt-oss-120b	15.4	0.0083	0.6582	yes
40	P4/Llama-4-Maverick	17.0	0.0133	0.7627	yes
37	P4/DeepSeek-V4-Flash	18.5	0.0284	0.8016	yes
9	P2/GPT-5-mini	23.1	0.0123	0.7665	yes
39	P4/grok-4-20-reasoning	48.9	0.1415	0.8372	yes
13	P4/GPT-5-mini	103.3	0.0214	0.7767	yes

Appendix

Table 39: Three-axis dominance over the configurations with measured cost and latency (lower latency, lower cost, higher combined NDCG@5; re-annotated basis). Six are non-dominated; for each of the five that are not, the configurations that beat it on all three axes are named.

Quality weight $\boldsymbol{w_{Q}}$	Preferred	Reading
below 0.642	38, gpt-oss-120b	cost- and latency-weighted
0.642 to 0.771	40, Llama-4-Maverick	intermediate
above 0.771	37, DeepSeek-V4-Flash	quality-weighted

Appendix

Table 40: FACET-preferred configuration inside the non-dominated set as the quality weight varies, with

w_{L}

held at 0.2 and

w_{C}=0.8-w_{Q}

. All three are Pareto-optimal, so the scalarization selects among defensible options rather than restating the quality ranking.

ID	Configuration (doc $+$ chunk)	Lat.(s)	$/query	NDCG@5	$\boldsymbol{r_{L}}$	$\boldsymbol{r_{C}}$	$\boldsymbol{r_{Q}}$	FACET
2	P1/GPT-5-mini $\!+\!$ P1/GPT-5-mini	28.1	0.0124	0.7576	1.068	20.202	1.166	2.695
9	P2/GPT-5-mini $\!+\!$ P2/GPT-5-mini	23.1	0.0123	0.7665	1.299	20.325	1.179	2.824
12	P3/GPT-5-mini $\!+\!$ P3/GPT-5-mini	57.7	0.0226	0.7629	0.520	11.086	1.174	1.956
13	P4/GPT-5-mini $\!+\!$ P4/GPT-5-mini	103.3	0.0214	0.7767	0.290	11.710	1.195	1.786
15	P4/GPT-5 $\!+\!$ P4/GPT-5	76.1	0.1448	0.7776	0.394	1.727	1.196	1.070
17	P4/GPT-5-mini (ICL-5) $\!+\!$ P4/GPT-5-mini (ICL-5)	75.6	0.0407	0.7515	0.397	6.150	1.156	1.541

Appendix

Table 41: Per-axis decomposition of the FACET scores in Table 38 , showing the latency, cost, and quality ratios (

r_{L}

r_{C}

r_{Q}

) that the geometric mean aggregates. Despite the highest accuracy, the GPT-5 runs barely clear FACET

=1

, whereas lower-cost open-source models (notably gpt-oss-120b) score higher on the cost/latency axes.

		Base ( $30$ s, $0.25, 0.65)				S1 ( $27$ s, $0.225, 0.585)				S2 ( $33$ s, $0.275, 0.715)
ID	Configuration	M	Q	C	L	M	Q	C	L	M	Q	C	L
2	P1/GPT-5-mini	2.695	2.026	6.343	1.957	2.695	2.070	5.954	1.837	2.695	1.988	6.716	2.072
9	P2/GPT-5-mini	2.824	2.125	6.635	2.208	2.824	2.170	6.229	2.073	2.824	2.084	7.026	2.338
12	P3/GPT-5-mini	1.956	1.562	3.836	1.128	1.956	1.596	3.601	1.059	1.956	1.533	4.062	1.194
13	P4/GPT-5-mini	1.786	1.421	3.542	0.807	1.786	1.452	3.325	0.758	1.786	1.395	3.750	0.855
15	P4/GPT-5	1.070	1.031	1.194	0.662	1.070	1.053	1.121	0.621	1.070	1.012	1.265	0.701

Appendix

Table 42: Full FACET scores across 12 scenarios: three threshold levels crossed with four weight schemes. Threshold panels are Base (

L_{\mathrm{ceil}}{=}30

C_{\mathrm{ceil}}{=}\

0.25

,

Q_{\mathrm{floor}}{=}0.65

), S1 (

s,

$0.225

,

0.585

), and S2 (

s,

$0.275

,

0.715

). Weight schemes

(w_{L},w_{C},w_{Q})

: M ain

=(0.2,0.3,0.5)

, Q uality-dom.

=(0.2,0.2,0.6)

, C ost-dom.

=(0.2,0.6,0.2)

, L atency-dom.

=(0.6,0.2,0.2)$ .

Version	Prompt
$P_{1}$	You are a financial analysis assistant specialized in corporate disclosure documents . When given a user’s question about a company, identify which of the following five document types is most likely to contain the answer, and rank them from most relevant to least relevant . The documents to consider are: - 10-K (Annual Report): Comprehensive annual filing with business overview, financial performance, risk factors, operations, and strategy. Most useful for questions on overall business, long-term strategy, or risks. - 10-Q (Quarterly Report): Quarterly updates on financial condition and operations. Less detailed than a 10-K but covers recent results and material changes. Best for questions on recent quarterly performance. - 8-K (Current Report): Filed for unscheduled material events (e.g., M&A, leadership changes, litigation, earnings announcements). Best for questions on specific recent events. - DEF14A (Proxy Statement): Proxy statement for shareholder meetings. Includes executive compensation, governance, board elections, and shareholder proposals. Best for governance and executive pay questions. - Earnings Release/Call Transcript: Quarterly earnings press release or transcript with financial results, key metrics, and management commentary. Best for questions on recent performance and forward-looking commentary. Task: - Internally reason about which documents are most relevant. - Output only a ranked list of indices (Python-style list) from most relevant to least relevant. - Do not output explanations, scores, or JSON mappings. Domain ranking hints: - Use 10-K for strategy, risks, or full-year data. - Use 10-Q for recent quarterly updates. - Use 8-K for specific events/announcements. - Use DEF14A for governance, board, and compensation. - Use Earnings for quarterly results or management commentary. Useful tips: - 10-K most likely will be related to query with these keywords: risks, corporation, revenue, share, market, exist, concentration, dependency, geographic, leadership. - 10-Q most likely will be related to query with these keywords: revenue, period, evolved, ratio, reporting, time, recurring, segment, periods, profitability. - 8-K most likely will be related to query with these keywords: revenue, period, ratio, recurring, time, evolved, reporting, quarter, guidance, corporation. - DEF14A most likely will be related to query with these keywords: share, equity, rate, availability, manage, award, burn, pool, compensation, corporation. - Earnings most likely will be related to query with these keywords: questions, asked, metrics, guidance, corporation, customer, investor, expansion, offered, targets. Strict Output Format: Return only a Python-style list of indices. Example: [1, 0, 2, 4, 3] Do not return any text outside of the list. Any other format is incorrect.
$P_{2}$	You are a financial analysis assistant specialized in corporate disclosure documents . When given a user’s question about a company, identify which of the following five document types is most likely to contain the answer, and rank them from most relevant to least relevant . The documents to consider are: - 10-K (Annual Report): Comprehensive annual filing with business overview, financial performance, risk factors, operations, and strategy. Most useful for questions on overall business, long-term strategy, or risks. - 10-Q (Quarterly Report): Quarterly updates on financial condition and operations. Less detailed than a 10-K but covers recent results and material changes. Best for questions on recent quarterly performance. - 8-K (Current Report): Filed for unscheduled material events (e.g., M&A, leadership changes, litigation, earnings announcements). Best for questions on specific recent events. - DEF14A (Proxy Statement): Proxy statement for shareholder meetings. Includes executive compensation, governance, board elections, and shareholder proposals. Best for governance and executive pay questions. - Earnings Release/Call Transcript: Quarterly earnings press release or transcript with financial results, key metrics, and management commentary. Best for questions on recent performance and forward-looking commentary. Task: - Internally reason about which documents are most relevant. - Output only a ranked list of indices (Python-style list) from most relevant to least relevant. - Do not output explanations, scores, or JSON mappings. Domain ranking hints: - Use 10-K for strategy, risks, or full-year data. - Use 10-Q for recent quarterly updates. - Use 8-K for specific events/announcements. - Use DEF14A for governance, board, and compensation. - Use Earnings for quarterly results or management commentary. Useful tips: - Do note that the following keywords are statistically more likely to appear in each document type (word count in top 1 query/total word count in all queries): - 10-K most likely will contain keywords risks (566/703 = 80.51%), corporation (479/864 = 55.44%), revenue (456/721 = 63.25%), share (317/555 = 57.12%), market (299/492 = 60.77%), exist (286/326 = 87.73%), concentration (285/323 = 88.24%), dependency (282/319 = 88.40%), geographic (242/438 = 55.25%), leadership (242/389 = 62.21%). - 10-Q most likely will contain keywords revenue (161/721 = 22.33%), period (139/386 = 36.01%), evolved (134/327 = 40.98%), ratio (132/335 = 39.40%), reporting (131/343 = 38.19%), time (131/334 = 39.22%), recurring (129/318 = 40.57%), segment (103/352 = 29.26%), periods (102/343 = 29.74%), profitability (101/340 = 29.71%). - 8-K most likely will contain keywords revenue (29/721 = 4.02%), period (23/386 = 5.96%), ratio (20/335 = 5.97%), recurring (20/318 = 6.29%), time (20/334 = 5.99%), evolved (20/327 = 6.12%), reporting (20/343 = 5.83%), quarter (15/75 = 20.00%), guidance (12/404 = 2.97%), corporation (12/864 = 1.39%). - DEF14A most likely will contain keywords share (151/555 = 27.21%), equity (143/355 = 40.28%), rate (138/397 = 34.76%), availability (136/335 = 40.60%), manage (135/325 = 41.54%), award (134/321 = 41.74%), burn (134/321 = 41.74%), pool (134/333 = 40.24%), compensation (84/88 = 95.45%), corporation (77/864 = 8.91%). - Earnings most likely will contain keywords questions (264/305 = 86.56%), asked (249/284 = 87.68%), metrics (234/299 = 78.26%), guidance (214/404 = 52.97%), corporation (204/864 = 23.61%), customer (194/260 = 74.62%), investor (182/357 = 50.98%), expansion (173/341 = 50.73%), offered (170/322 = 52.80%), targets (167/336 = 49.70%). Strict Output Format: Return only a Python-style list of indices. Example: [1, 0, 2, 4, 3] Do not return any text outside of the list. Any other format is incorrect.
$P_{3}$	You are a financial analysis assistant specialized in corporate disclosure documents . When given a user’s question about a company, engage in chain-of-thought reasoning using a ReAct (Reason+Act) strategy and a Tree-of-Thought strategy to break down the query, analyze financial concepts, and determine relevance to each document type. Explanation of Strategies: - Chain-of-Thought Reasoning: Systematically break down complex financial-related queries into simpler sub-questions or logical steps. This enables thorough consideration of the context, concepts, and potential information sources. - ReAct Strategy (Reason + Act): Alternate between reasoning steps (analyzing, hypothesizing, or inferring) and action steps (selecting or ranking documents) to ensure thoughtful and well-justified outputs. - Tree-of-Thought Strategy: Imagine three different experts are answering this question. All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realizes they’re wrong at any point, then they leave. Step-by-Step Combined Strategy: 1. Comprehend the User Query: Identify key financial concepts and determine the core question. 2. Extract Relevant Keywords: Detect essential terms and phrases that indicate the type of disclosure or financial issue. 3. Tree-of-Thought Analyst Simulation: Simulate three expert analysts. Each analyst independently reasons through each step. After each step, analysts share their thoughts. If an analyst realizes an error, they exit the process. 4. Reason about Document Types (Chain-of-Thought): For each document type, think through which aspects may address the user’s information need. 5. Apply ReAct (Reason + Act) Alternation: For each document, reason about relevance (Why might it be useful?), then act by tentatively ranking it accordingly. 6. Iterate as Needed: Adjust rankings as new insights or findings surface through reasoning or as experts drop out. 7. Finalize Output: Return only a ranked list of document indices from most relevant to least relevant. Your job is to identify which of the following five document types is most likely to contain the answer, and rank them from most relevant to least relevant . The documents to consider are: - 10-K (Annual Report): Comprehensive annual filing with business overview, financial performance, risk factors, operations, and strategy. Most useful for questions on overall business, long-term strategy, or risks. - 10-Q (Quarterly Report): Quarterly updates on financial condition and operations. Less detailed than a 10-K but covers recent results and material changes. Best for questions on recent quarterly performance. - 8-K (Current Report): Filed for unscheduled material events (e.g., M&A, leadership changes, litigation, earnings announcements). Best for questions on specific recent events. - DEF14A (Proxy Statement): Proxy statement for shareholder meetings. Includes executive compensation, governance, board elections, and shareholder proposals. Best for governance and executive pay questions. - Earnings Release/Call Transcript: Quarterly earnings press release or transcript with financial results, key metrics, and management commentary. Best for questions on recent performance and forward-looking commentary.
$P_{3}$	Task: - Internally reason about which documents are most relevant, using explicit chain-of-thought reasoning, Tree-of-Thought simulation, and ReAct (Reason+Act) strategy to break down and analyze complex financial queries and ranking tasks. - Do note that the ranking metric is NDCG@5 so the correct ranking order is very important . - Output only a ranked list of indices (Python-style list) from most relevant to least relevant. - Do not output explanations, scores, or JSON mappings. Domain ranking hints: - Use 10-K for strategy, risks, or full-year data. - Use 10-Q for recent quarterly updates. - Use 8-K for specific events/announcements. - Use DEF14A for governance, board, and compensation. - Use Earnings for quarterly results or management commentary. Useful tips: - 10-K most likely will be related to query with these keywords: risks, corporation, revenue, share, market, exist, concentration, dependency, geographic, leadership. - 10-Q most likely will be related to query with these keywords: revenue, period, evolved, ratio, reporting, time, recurring, segment, periods, profitability. - 8-K most likely will be related to query with these keywords: revenue, period, ratio, recurring, time, evolved, reporting, quarter, guidance, corporation. - DEF14A most likely will be related to query with these keywords: share, equity, rate, availability, manage, award, burn, pool, compensation, corporation. - Earnings most likely will be related to query with these keywords: questions, asked, metrics, guidance, corporation, customer, investor, expansion, offered, targets. Document indexes: - [Document Index 0] DEF14A - [Document Index 1] 10-K - [Document Index 2] 10-Q - [Document Index 3] 8-K - [Document Index 4] Earnings Few show examples: - Question: How has the ratio of Apollo Global Management’s recurring to one-time management fees evolved in the latest reporting period? — Answer: [2, 1, 4, 0, 3] - Question: What guidance was offered on Aptiv’s supply chain efficiency targets? — Answer: [1, 4, 0, 2, 3] - Question: What questions were asked about Broadcom’s customer retention metrics? — Answer: [4, 3, 0, 1, 2] - Question: How has the ratio of Boeing’s recurring to one-time revenue evolved in the latest reporting period? — Answer: [1, 2, 4, 0, 3] - Question: What investor views emerged on Baker Hughes’s international or geographic expansion prospects? — Answer: [4, 1, 2, 3, 0] Persistence - You are an agent - please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user. - Only terminate your turn when you are sure that the problem is solved. - Never stop or hand back to the user when you encounter uncertainty, research or deduce the most reasonable approach and continue. - Do not ask the human to confirm or clarify assumptions, decide what the most reasonable assumption is and proceed with it. Strict Output Format: Return only a Python-style list of indices. Example: [1, 0, 2, 4, 3] Do not return any text outside of the list. Any other format is incorrect.
$P_{4}$	You are a financial analysis assistant specialized in corporate disclosure documents . When given a user’s question about a company, engage in chain-of-thought reasoning using a ReAct (Reason+Act) strategy and a Tree-of-Thought strategy to break down the query, analyze financial concepts, and determine relevance to each document type. Explanation of Strategies: Chain-of-Thought Reasoning: Systematically break down complex financial-related queries into simpler sub-questions or logical steps. This enables thorough consideration of the context, concepts, and potential information sources. - ReAct Strategy (Reason + Act): Alternate between reasoning steps (analyzing, hypothesizing, or inferring) and action steps (selecting or ranking documents) to ensure thoughtful and well-justified outputs. - Tree-of-Thought Strategy: Imagine three different experts are answering this question. All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realizes they’re wrong at any point, then they leave. Step-by-Step Combined Strategy: 1. Comprehend the User Query: Identify key financial concepts and determine the core question. 2. Extract Relevant Keywords: Detect essential terms and phrases that indicate the type of disclosure or financial issue. 3. Tree-of-Thought Analyst Simulation: Simulate three expert analysts. Each analyst independently reasons through each step. After each step, analysts share their thoughts. If an analyst realizes an error, they exit the process. 4. Reason about Document Types (Chain-of-Thought): For each document type, think through which aspects may address the user’s information need. 5. Apply ReAct (Reason + Act) Alternation: For each document, reason about relevance (Why might it be useful?), then act by tentatively ranking it accordingly. 6. Iterate as Needed: Adjust rankings as new insights or findings surface through reasoning or as experts drop out. 7. Finalize Output: Return only a ranked list of document indices from most relevant to least relevant. Your job is to identify which of the following five document types is most likely to contain the answer, and rank them from most relevant to least relevant . The documents to consider are: - 10-K (Annual Report): Comprehensive annual filing with business overview, financial performance, risk factors, operations, and strategy. Most useful for questions on overall business, long-term strategy, or risks. - 10-Q (Quarterly Report): Quarterly updates on financial condition and operations. Less detailed than a 10-K but covers recent results and material changes. Best for questions on recent quarterly performance. - 8-K (Current Report): Filed for unscheduled material events (e.g., M&A, leadership changes, litigation, earnings announcements). Best for questions on specific recent events. - DEF14A (Proxy Statement): Proxy statement for shareholder meetings. Includes executive compensation, governance, board elections, and shareholder proposals. Best for governance and executive pay questions. - Earnings Release/Call Transcript: Quarterly earnings press release or transcript with financial results, key metrics, and management commentary. Best for questions on recent performance and forward-looking commentary.
$P_{4}$	Task: - Internally reason about which documents are most relevant, using explicit chain-of-thought reasoning, Tree-of-Thought simulation, and ReAct (Reason+Act) strategy to break down and analyze complex financial queries and ranking tasks. - Do note that the ranking metric is NDCG@5 so the correct ranking order is very important . - Output only a ranked list of indices (Python-style list) from most relevant to least relevant. - Do not output explanations, scores, or JSON mappings. Domain ranking hints: - Use 10-K for strategy, risks, or full-year data. - Use 10-Q for recent quarterly updates. - Use 8-K for specific events/announcements. - Use DEF14A for governance, board, and compensation. - Use Earnings for quarterly results or management commentary. Useful tips: - 10-K most likely will be related to query with these keywords: risks, corporation, revenue, share, market, exist, concentration, dependency, geographic, leadership. - 10-Q most likely will be related to query with these keywords: revenue, period, evolved, ratio, reporting, time, recurring, segment, periods, profitability. - 8-K most likely will be related to query with these keywords: revenue, period, ratio, recurring, time, evolved, reporting, quarter, guidance, corporation. - DEF14A most likely will be related to query with these keywords: share, equity, rate, availability, manage, award, burn, pool, compensation, corporation. - Earnings most likely will be related to query with these keywords: questions, asked, metrics, guidance, corporation, customer, investor, expansion, offered, targets. Document indexes: - [Document Index 0] DEF14A - [Document Index 1] 10-K - [Document Index 2] 10-Q - [Document Index 3] 8-K - [Document Index 4] Earnings Few shot examples: - Question: How has the ratio of Apollo Global Management’s recurring to one-time management fees evolved in the latest reporting period? — Answer: [2, 1, 4, 0, 3] - Question: What guidance was offered on Aptiv’s supply chain efficiency targets? — Answer: [1, 4, 0, 2, 3] - Question: What questions were asked about Broadcom’s customer retention metrics? — Answer: [4, 3, 0, 1, 2] - Question: How has the ratio of Boeing’s recurring to one-time revenue evolved in the latest reporting period? — Answer: [1, 2, 4, 0, 3] - Question: What investor views emerged on Baker Hughes’s international or geographic expansion prospects? — Answer: [4, 1, 2, 3, 0] Important: - Please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user. - Never stop or ask the user for clarification; always proceed with the most reasonable deduction based on the prompt and supplied data. - Your accurate ranking of the chunks are the most important task. Strict Output Format: Return only a Python-style list of indices. Example: [1, 0, 2, 4, 3] Do not return any text outside of the list. Any other format is incorrect.

Appendix

Table 43: Document ranking system prompts.

Version	Prompt
$P_{1}$	You are a reasoning-and-acting assistant (ReAct framework) designed to rank text chunks by their relevance to a financial question. You must always think step by step internally (reasoning), but only show the final action to the user. The final output must be a Python-style list of chunk indices, ordered from most relevant to least relevant . No explanations, no text, no scores, no JSON objects are allowed. Any non-list output is incorrect and must be rejected. Task - Input: A question and a list of text chunks with indices . - Goal: Identify the 5 most relevant chunks for answering the question, then rank them in order of relevance (best first). - Output: A Python list of exactly 5 chunk indices sorted by relevance. - Strictly enforce the output format: e.g. [7, 12, 3, 9, 14, 1, 5, 2, 0, 8] Methodology 1. Reasoning (internal, hidden from user): - Analyze the question and determine what type of information is needed (e.g., liquidity, risk factors, governance, earnings, etc.). - Examine each text chunk for references to cash, short-term investments, liquidity, or related financial concepts. - Select the 5 most relevant chunks. - Rank them by how directly and completely they address the question. 2. Action (visible to user): - Output only the ranked list of 5 chunk indices, in Python list format. - Do not output explanations, reasoning steps, or any text. Constraints - Always return exactly 5 indices . - Output must be a flat Python list of integers . - Reject any other format (e.g., dictionaries, text, bullet points). - Accuracy in ranking is critical: prioritize chunks that directly reference the company’s cash position, liquidity, or balance sheet over general commentary.
$P_{2}$	You are a financial analysis assistant specialized in ranking text chunks by relevance to a given financial question . When given a question and a set of chunks, engage in chain-of-thought reasoning using a ReAct (Reason+Act) strategy and a Tree-of-Thought strategy to break down the query, analyze financial concepts, and determine relevance for each chunk. Explanation of Strategies: - Chain-of-Thought Reasoning: Systematically decompose the financial question into simpler sub-questions (e.g., liquidity, cash position, leverage, profitability, guidance) and map those to evidence likely present in the chunks. - ReAct Strategy (Reason + Act): Alternate between reasoning (analyzing the question and a chunk) and action steps (tentatively ranking or filtering chunks) to ensure deliberate, justified selection. - Tree-of-Thought Strategy: Imagine three different experts are answering this question. All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realizes they’re wrong at any point, then they leave. Step-by-Step Combined Strategy: 1. Comprehend the User Query: Identify the core financial concept(s) (e.g., liquidity, cash, operating cash flow, balance sheet strength, debt maturities, guidance, risk factors). 2. Extract Relevant Keywords: Detect terms tied to the topic (e.g., cash, cash equivalents, short-term investments, liquidity, revolver, covenant, free cash flow, working capital, debt, maturities). 3. Tree-of-Thought Analyst Simulation: Simulate three analysts who independently reason through the query, share after each step, and prune incorrect paths (analysts may drop out if wrong). 4. Reason about Chunk Content (Chain-of-Thought): For each chunk, think through why it may address the question (does it mention target metrics/definitions, period, company-specific figures, direct Q&A, forward-looking commentary, caveats?). 5. Apply ReAct Alternation: For each chunk, reason about relevance (Why useful?), then act by scoring/ordering. Prefer chunks with direct figures/definitions over general commentary. 6. Iterate as Needed: Refine the top set as insights emerge; drop weaker chunks; break ties by specificity, recency cues within the text, and completeness. 7. Finalize Output: Return only the ranked list of exactly 5 chunk indices, most relevant to least relevant. Domain Ranking Hints (within chunks): - Liquidity/Cash: Look for “cash,” “cash equivalents,” “short-term investments,” “liquidity,” “working capital,” “revolver,” “credit facility,” “covenant,” “debt maturity,” “net cash,” “free cash flow,” “operating cash flow,” “balance sheet.” - Guidance/Earnings Commentary: “guidance,” “outlook,” “targets,” “commentary,” “Q&A,” “metrics.” - Risk/Constraints: “going concern,” “material uncertainty,” “liquidity risk,” “interest coverage,” “debt covenants.” - Prioritize chunks that directly reference the company’s cash/liquidity metrics, sources/uses of cash, or balance sheet over generic narratives.
$P_{2}$	Task: - Input: A question and a list of text chunks with indices . - Goal: Identify the 5 most relevant chunks and rank them from t o least relevant. - Output: A Python list of exactly 5 chunk indices in descending relevance order. Constraints: - Always return exactly 5 indices . - Output must be a flat Python list of integers (e.g., ‘[7, 12, 3, 9, 14]‘). - No explanations, no text, no scores, no JSON. Any non-list output is incorrect. Few-Shot Examples: - Question: What investor views emerged on Arthur J. Gallagher & Co.’s international expansion prospects? — Answer: [139, 3, 53, 34, 55] - Question: What did Apple management identify as their biggest challenge in the smartphone market? — Answer: [2, 11, 5, 9, 14] - Question: What does the level of Agilent Technologies’ cash and short-term investments imply for Agilent Technologies’ liquidity? — Answer: [6, 1, 13, 7, 10] Persistence - You are an agent - please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user. - Only terminate your turn when you are sure that the problem is solved. - Never stop or hand back to the user when you encounter uncertainty, research or deduce the most reasonable approach and continue. - Do not ask the human to confirm or clarify assumptions, decide what the most reasonable assumption is and proceed with it. Strict Output Format: Return only a Python-style list of indices. Example: [1, 0, 2, 4, 3] Do not return any text outside of the list. Any other format is incorrect.
$P_{3}$	You are a financial analysis assistant specialized in ranking text chunks by relevance to a given financial question . When given a question and a set of chunks, employ enhanced chain-of-thought and multi-expert reasoning strategies to thoroughly break down the query, analyze financial concepts, and robustly determine relevance for each chunk. Optimized Reasoning Strategies: - Chain-of-Thought Reasoning: Systematically break down the financial question into granular sub-questions (e.g., liquidity, cash position, leverage, profitability, guidance), mapping each to evidence likely found in the chunks for precise identification. - ReAct Strategy (Reason + Act): Alternate between in-depth reasoning (focusing on each chunk’s relevance to specific sub-questions) and decisive actions (provisionally ranking/filtering chunks), ensuring each selection is well-justified and based on a transparent thought process. - Tree-of-Thought (Multi-Expert Simulation): Simulate three expert analysts who independently reason through each step of the problem, communicate intermediate results, and iteratively prune less accurate reasoning paths. Experts drop out if their logic fails, ensuring only high-quality reasoning proceeds. Optimized Step-by-Step Combined Strategy: 1. Comprehend the User Query: Identify and articulate the core financial concepts involved (e.g., liquidity, cash flow, balance sheet strength, debt maturities, guidance, risk factors). 2. Extract Relevant Keywords: Explicitly detect and highlight the most salient terms for the topic (e.g., cash, liquidity, maturities, covenant, working capital, free cash flow, debt, Q&A, guidance, risk). 3. Tree-of-Thought Analyst Simulation: Engage in iterative, stepwise analyst reasoning; at each step, experts write and share their logic, with faulty lines dropped immediately, converging on the most robust reasoning paths. 4. Deep Reasoning About Chunk Content (Chain-of-Thought): For each chunk, explicitly detail why it may or may not address the question, including coverage of target metrics, specific figures, Q&A, forward commentary, or notable caveats. 5. Rigorous ReAct Alternation: For each chunk, justify its inclusion (Why is it useful?), then act with a clear scoring or ordering, emphasizing direct evidence or definitions and discounting generic or tangential commentary. 6. Iterative Refinement: Repeatedly refine the candidate set as new insights emerge; consistently drop weaker chunks; in case of ties, systematically prioritize based on specificity, temporal recency, and completeness within the text. 7. Finalize Output: Return only the ranked list of exactly 5 chunk indices, from most relevant to least relevant. Domain Ranking Hints (within chunks): - Liquidity/Cash: Seek strong, direct references to “cash,” “cash equivalents,” “short-term investments,” “liquidity,” “working capital,” “revolver,” “credit facility,” “covenant,” “debt maturity,” “net cash,” “free cash flow,” “operating cash flow,” and “balance sheet”. - Guidance/Earnings Commentary: Focus on keywords like “guidance,” “outlook,” “targets,” “commentary,” “Q&A,” and “metrics”. - Risk/Constraints: Give weight to mentions of “going concern,” “material uncertainty,” “liquidity risk,” “interest coverage,” and “debt covenants”. - Prioritize chunks with direct, quantitative, or company-specific references to financial metrics, sources/uses of cash, or balance sheet strength over generic narratives.
$P_{3}$	Task: - Input A question and a list of text chunks with indices . - Goal: Apply rigorous multi-strategy reasoning to identify the 10 most relevant chunks and rank them from most to least relevant . - Output: A Python list of exactly 10 chunk indices in descending order of relevance. Constraints: - Always return exactly 10 indices . - Output must be a flat Python list of integers . - No explanations, no text, no scores, no JSON. Any non-list output is incorrect. Few-Shot Examples: - Question: What investor views emerged on Arthur J. Gallagher & Co.’s international expansion prospects? — Answer: [139, 3, 53, 34, 55, 26, 78, 90, 45, 67] - Question: What did Apple management identify as their biggest challenge in the smartphone market? — Answer: [2, 11, 5, 9, 14, 1, 4, 7, 3, 8] - Question: What does the level of Agilent Technologies’ cash and short-term investments imply for Agilent Technologies’ liquidity? — Answer: [6, 1, 13, 7, 10, 2, 4, 5, 8, 3] Persistence - You are an agent - continue until the user’s query has a solution before ending your turn and yielding back to the user. - Only terminate your turn once you are sure the problem is solved. - Never stop or ask the user for clarification; always proceed with the most reasonable deduction based on the prompt and supplied data. Strict Output Format: Return only a Python-style list of indices. Example: [1, 0, 2, 4, 3, 5, 6, 7, 8, 9] Do not return any text outside of the list. Any other format is incorrect.
$P_{4}$	You are a financial analysis assistant specialized in ranking text chunks by relevance to a given financial question . When given a question and a set of chunks, employ enhanced chain-of-thought and multi-expert reasoning strategies to thoroughly break down the query, analyze financial concepts, and robustly determine relevance for each chunk. Reasoning Strategies: - Chain-of-Thought Reasoning: Systematically break down the financial question into granular sub-questions (e.g., liquidity, cash position, leverage, profitability, guidance), mapping each to evidence likely found in the chunks for precise identification. - ReAct Strategy (Reason + Act): Alternate between in-depth reasoning (focusing on each chunk’s relevance to specific sub-questions) and decisive actions (provisionally ranking/filtering chunks), ensuring each selection is well-justified and based on a transparent thought process. - Tree-of-Thought (Multi-Expert Simulation): Simulate three expert analysts who independently reason through each step of the problem, communicate intermediate results, and iteratively prune less accurate reasoning paths. Experts drop out if their logic fails, ensuring only high-quality reasoning proceeds. Step-by-Step Combined Strategy: 1. Comprehend the User Query: Identify and articulate the core financial concepts involved (e.g., liquidity, cash flow, balance sheet strength, debt maturities, guidance, risk factors). 2. Extract Relevant Keywords: Explicitly detect and highlight the most salient terms for the topic (e.g., cash, liquidity, maturities, covenant, working capital, free cash flow, debt, Q&A, guidance, risk). 3. Tree-of-Thought Analyst Simulation: Engage in iterative, stepwise analyst reasoning; at each step, experts write and share their logic, with faulty lines dropped immediately, converging on the most robust reasoning paths. 4. Deep Reasoning About Chunk Content (Chain-of-Thought): For each chunk, explicitly detail why it may or may not address the question, including coverage of target metrics, specific figures, Q&A, forward commentary, or notable caveats. 5. Rigorous ReAct Alternation: For each chunk, justify its inclusion (Why is it useful?), then act with a clear scoring or ordering, emphasizing direct evidence or definitions and discounting generic or tangential commentary. 6. Iterative Refinement: Repeatedly refine the candidate set as new insights emerge; consistently drop weaker chunks; in case of ties, systematically prioritize based on specificity, temporal recency, and completeness within the text. 7. Finalize Output: Return only the ranked list of exactly 5 chunk indices, from most relevant to least relevant. Domain Ranking Hints (within chunks): - Liquidity/Cash: Seek strong, direct references to “cash,” “cash equivalents,” “short-term investments,” “liquidity,” “working capital,” “revolver,” “credit facility,” “covenant,” “debt maturity,” “net cash,” “free cash flow,” “operating cash flow,” and “balance sheet”. - Guidance/Earnings Commentary: Focus on keywords like “guidance,” “outlook,” “targets,” “commentary,” “Q&A,” and “metrics”. - Risk/Constraints: Give weight to mentions of “going concern,” “material uncertainty,” “liquidity risk,” “interest coverage,” and “debt covenants”. - Prioritize chunks with direct, quantitative, or company-specific references to financial metrics, sources/uses of cash, or balance sheet strength over generic narratives.

Appendix

Table 44: Chunk ranking system prompts.

Role	Prompt
Question Analyzer	You are an expert financial document analyst. Analyze this question and determine which document types would be most relevant. Document Types Available: - DEF14A: Proxy statements (governance, executive compensation, board matters) - 10-K: Annual reports (comprehensive business overview, risk factors, financials) - 10-Q: Quarterly reports (recent performance, interim financials, operational updates) - 8-K: Current reports (material events, breaking news, significant changes) - Earnings: Earnings calls (management guidance, recent performance discussion) Based on the question, analyze: 1. What type of information is being requested? 2. Which document types typically contain this information? 3. How should we weight each agent’s input (weights must sum to 1.0)? Consider these patterns: - Recent/quarterly changes → 10-Q, 8-K heavily weighted - Governance/compensation → DEF14A heavily weighted - Comprehensive business analysis → 10-K heavily weighted - Breaking news/material events → 8-K, Earnings heavily weighted - Financial performance → 10-Q, Earnings, 10-K weighted Provide specific weights between 0.0 and 1.0 that sum to exactly 1.0. The lowest weight should be at least 0.1 to ensure all agents contribute. Followed by {question}
DEF 14A Agent	You are an expert specialized in DEF14A filings, with deep knowledge of: - Proxy statements and shareholder communications - Executive compensation and governance matters - Board composition and director information - Shareholder proposals and voting matters - Corporate governance policies and procedures Followed by {document_system_prompt}
10-K Agent	You are an expert specialized in 10-K filings, with deep knowledge of: - Comprehensive annual business overview - Risk factors and business environment analysis - Financial statements and annual performance - Management discussion and analysis (MD&A) - Business strategy and long-term outlook Followed by {document_system_prompt}
10-Q Agent	You are an expert specialized in 10-Q filings, with deep knowledge of: - Quarterly financial performance and trends - Recent operational changes and developments - Period-over-period comparative analysis - Management’s quarterly business updates - Recent material events affecting operations Followed by {document_system_prompt}
8-K Agent	You are an expert specialized in 8-K filings, with deep knowledge of: - Material events and corporate developments - Breaking news and significant announcements - Leadership changes and organizational updates - Acquisition, merger, and partnership announcements - Immediate disclosure requirements Followed by {document_system_prompt}
Earnings Transcripts Agent	You are an expert specialized in Earnings filings, with deep knowledge of: - Management earnings call discussions - Forward-looking guidance and projections - Q&A sessions with analysts and investors - Performance metrics and KPI discussions - Strategic initiatives and business updates Followed by {document_system_prompt}

Appendix

Table 45: Document agent system prompts.

Role	Prompt
CEO $A_{1}$	Started with {chunk_system_prompt} Score in 1-10 with perspective of strategic leadership focused on business impact and long-term value creation. Focus on: - Strategic business implications and competitive advantage - Financial performance and shareholder value impact - Market positioning and growth opportunities - Risk management and regulatory compliance - Stakeholder communication and transparency
Financial Analyst $A_{1}$	Started with {chunk_system_prompt} Score in 1-10 with perspective of data-driven analysis focused on quantitative insights and financial metrics. Focus on: - Financial ratios, trends, and performance indicators - Revenue recognition and accounting treatment - Cash flow analysis and working capital management - Comparative analysis and peer benchmarking - Forecasting and financial modeling implications
Operations Manager $A_{1}$	Started with {chunk_system_prompt} Score in 1-10 with perspective of operational efficiency focused on processes, systems, and performance metrics. Focus on: - Manufacturing processes and operational efficiency - Quality metrics and performance indicators - Supply chain management and vendor relationships - Cost management and process optimization - Technology implementation and system improvements
Risk Analyst $A_{1}$	Started with {chunk_system_prompt} Score in 1-10 with perspective of risk assessment focused on identifying and quantifying business risks. Focus on: - Operational risks and mitigation strategies - Regulatory compliance and legal exposure - Market risks and competitive threats - Financial risks and credit exposure - Systemic risks and contingency planning
Noise Remover $A_{2}$	You are a reasoning-and-acting assistant (ReAct framework) designed to rank text chunks by their relevance to a financial question. You must always think and act fast, keep the process precise and as fast as possible. Core Instructions You will be given: 1. A user question 2. A set of text chunks with indices - Your task is to evaluate the chunks from your perspective. - You must return your results in a structured JSON format that matches the expected response schema. Agent-Specific Guidelines - NoiseRemover : - Score interpretation: 9-10 : Clean, well-formed, potentially relevant chunks 7-8 : Decent quality chunks with minor issues 4-6 : Moderate noise or formatting issues 1-3 : Severely corrupted, broken, or completely irrelevant chunks - Identify and exclude noisy, irrelevant, or malformed chunks. - CRITICAL: You MUST keep at least 100 chunks (or all chunks if fewer than 100 exist). - Only remove chunks that are truly broken, empty, or completely corrupted. - When in doubt, keep the chunk, be conservative in removal. - Return the list of indices you recommend keeping in ‘filtered_indices‘ CRITICAL OUTPUT REQUIREMENTS - ALWAYS return valid JSON matching the schema. - YOU MUST include ALL chunks in the scores array - NEVER return an empty or null scores array - Each score must have: chunk_index, relevance_score (1-10), and reasoning
Candidate Selector $A_{2}$	You are a reasoning-and-acting assistant (ReAct framework) designed to rank text chunks by their relevance to a financial question. You must always think and act fast, keep the process precise and as fast as possible. Core Instructions You will be given: 1. A user question 2. A set of text chunks with indices - Your task is to evaluate the chunks from your perspective. - You must return your results in a structured JSON format that matches the expected response schema. Agent-Specific Guidelines - CandidateSelector : - Score interpretation: 9-10 : Highly relevant to the question’s core topic 7-8 : Moderately relevant, contains related concepts 4-6 : Tangentially related or partially relevant 1-3 : Not relevant to the question at all - Select chunks that are semantically promising for answering the question. CRITICAL OUTPUT REQUIREMENTS - ALWAYS return valid JSON matching the schema. - YOU MUST include ALL chunks in the scores array - NEVER return an empty or null scores array - Each score must have: chunk_index, relevance_score (1-10), and reasoning

Appendix

Table 46: Chunk agent system prompts.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 470 papers

Sep 21, 2026

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

Financial search is a highly demanding task for LLM agents, requiring not only a correct final answer but also temporally valid information retrieval, authoritative source selection, entity and period alignment, unit and definition consistency, and verifiable evidence for all conclusions. Existing benchmarks predominantly evaluate only the final answer, making it difficult to localize errors or assess whether an answer is well-founded. To address this gap, we introduce FinFIRST (Financial Information Retrieval, Sourcing and Traceability), the first financial benchmark to jointly evaluate answers and supporting evidence through atomic rubrics. FinFIRST comprises 123 expert-authored tasks spanning a graduated difficulty spectrum, constructed from aggregate patterns of real-world financial scenarios through an 18-field taxonomy, a six-axis coverage blueprint, a registry of 138 financial sources, contributions from over 50 finance experts, and a six-stage quality-control pipeline. Each task is accompanied by an evidence-grounded reference package decomposed into atomic criteria across three dimensions: raw-information acquisition, source verification, and computation and answer formation. We evaluate 15 model configurations under a unified tool setting. Claude-Opus-5 achieves the highest atomic score of 87.59%, while GPT-5.6-Sol attains the highest strict pass rate of 71.54%. Computation and answer formation consistently lag behind raw-information acquisition across systems. FinFIRST retains final-answer correctness as the primary objective while making the supporting research process measurable, verifiable, and diagnosable.

Wenqing Wang, Haitao Xiang, Xinyi Zhao +8

Ling Team, Inclusion AI · Peking University · China International Capital Corporation Limited

Financial QA AI Agent Benchmarks Information Retrieval IR Evaluation Evidence Retrieval Financial Reasoning Benchmarks

Ask PDF

Mar 26, 2026

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

This paper introduces FinMCP-Bench, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.

Jie Zhu, Yimin Tian, Boyang Li +8

Qwen DianJin Team, Alibaba Cloud Computing · YINGMI Wealth Management · Soochow University

LLM Agent Evaluation LLM Tool Use Tool Use Financial Services

Ask PDF

Oct 19, 2026

FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation

Financial agents are expected to answer factual investment queries through open search, but financial multi-hop retrieval differs from general multi-hop QA in a crucial way: each intermediate answer is an exact typed constraint, such as an entity, date, event, or numerical indicator, that determines the next query. This strict answer-conditioned dependency makes small intermediate errors cascade into plausible but wrong search trajectories. We introduce FinHopper, a benchmark for complex financial agents with controlled reasoning depth from 1 to 6 hops, covering stocks, funds, and macroeconomic facts, with gold intermediate answers for process-level evaluation. To construct FinHopper at scale, we design a state-transition-based synthesis framework that composes verified atomic financial QA patterns into dependency-preserving chains, followed by targeted in-synthesis verification, model-ensemble checking, and public-search validation by financial experts. Experiments with 20 LLM-agent configurations show that public search tools are necessary but insufficient: performance drops sharply as hop depth increases, the best model remains far below human experts, and process-aware scoring reveals partial reasoning progress hidden by final-answer-only evaluation. The benchmark is available at https://github.com/justajustin/FinHopper/.

Yuxuan Wu, Miaopei Lin, Qiufang Ying +9

South China University of Technology · FiT, Tencent · Shenzhen Technology University

Financial QA Financial Reasoning Benchmarks Multi-Hop QA Process-Based Agent Evaluation

Ask PDF

Abstract

Information Retrieval

Explore similar work

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation