Bank-World: A Benchmark for Evaluating Agents on Banking Workflows

Daniel Rupawalla

cs.AIOct 29, 2026

Bank-World: A Benchmark for Evaluating Agents on Banking Workflows

Authors: Daniel Rupawalla

Organizations: Arcanum Labs

Abstract

Bank-World is a benchmark of 50 banking tasks across two simulated institutions. Completing banking work requires more than producing the right calculation: agents must reconcile records, obtain permission, execute transactions, and keep instructions and status reports consistent as circumstances change. Bank-World tests these requirements through single-turn assignments and multi-turn interactions with simulated employees who supply information, approve actions, and revise instructions. Grading combines executable checks of recorded actions and state with model judgments of task requirements, including whether valid alternative solutions satisfy them. Across 1,600 attempts by four models, we find that correct decisions often fail to become completed work. Agents charge fees twice, distribute the correct total in incorrect amounts among the recipients, and execute payments while approval is pending. Their strengths also differ: Sol leads single-turn completion, Opus has the highest observed multi-turn completion rate, and Grok succeeds across more multi-turn tasks but less consistently. Bank-World makes these differences inspectable through task-level results and worked examples connecting source evidence, agent decisions, and simulated consequences. We retain the task environments, executable graders, and full trajectories privately to reduce contamination and preserve the benchmark for future evaluations.

Figures & tables

Workflow	SQ	MQ	Associated skills
C: Credit analysis and funding	8	5	Eligibility, commitments, covenants, approval scope
P: Payments and cash reconciliation	6	4	Net cash effects, collected funds, reserves, payment status
L: Loan payoff and servicing	0	11	Interest, allocations, changed instructions, handoffs
R: Collateral and legal rights	6	2	Entitlement, consent, documentary proof, release conditions
K: Compliance and information controls	1	3	Source verification, escalation, disclosure, pending reviews
T: Trust and asset allocation	4	0	Economic ownership, principal/income, custody, residuals

Table 1: Primary workflows and associated skills.

Figure 1: Assignments connect distributed evidence to consequential actions. Multi-turn interaction supplies information and can change instructions during execution.

Setting	Model	Mean reward [95% interval]	Full passes	Rate
SQ	Gemini	0.513 [0.363, 0.662]	77/200	38.5%
SQ	Opus	0.438 [0.282, 0.596]	84/200	42.0%
SQ	Grok	0.390 [0.284, 0.495]	68/200	34.0%
SQ	Sol	0.638 [0.484, 0.784]	110/200	55.0%
MQ	Gemini	0.510 [0.387, 0.631]	73/200	36.5%
MQ	Opus	0.538 [0.391, 0.682]	88/200	44.0%

Table 2: Results on 25 tasks per setting, with 200 attempts per model in each setting. Brackets give 95% task-cluster bootstrap intervals for mean reward.

Figure 2: Full-pass rates across 200 attempts per model in each setting. Error bars show individual 95% task-cluster bootstrap intervals, retaining eight attempts per task. Intervals in percent: SQ: Gemini [25.5, 52.0], Opus [26.5, 58.0], Grok [23.5, 44.5], Sol [39.5, 70.0]; MQ: Gemini [25.0, 48.5], Opus [30.5, 57.5], Grok [27.5, 45.5], Sol [25.5, 59.5].

Setting	Model	No full pass	1–7 full passes	All 8 pass	$\mathrm{pass}^{2}$	$\mathrm{pass}^{4}$	$\mathrm{pass}^{8}$
SQ	Gemini	6	19	0	24.43	12.91	0.00
SQ	Opus	8	11	6	32.71	27.20	24.00
SQ	Grok	5	19	1	16.43	6.00	4.00
SQ	Sol	5	12	8	44.14	35.83	32.00
MQ	Gemini	6	18	1	20.29	10.17	4.00
MQ	Opus	7	16	2	29.57	15.54	8.00

Table 3: Full-pass counts across eight attempts on each of 25 tasks, and all-success statistics (percent).

Model	Single-turn	Multi-turn
Gemini	95 / 28 / 77	96 / 31 / 73
Opus	112 / 4 / 84	91 / 21 / 88
Grok	121 / 11 / 68	114 / 13 / 73
Sol	71 / 19 / 110	112 / 3 / 85

Table 4: Reward profiles. Each cell gives zero / partial / full-reward attempt counts; each setting contains 200 attempts per model.

	Accepted main decision		Main decision not accepted
Model	Complete	Incomplete	Complete	Incomplete
Gemini	105	40	1	54
Opus	88	39	2	71
Grok	79	48	7	66
Sol	129	46	4	21
Total	401	173	14	212

Table 5: Joint main-decision and required-action outcomes for all 800 single-turn attempts. “Accepted” means the main-answer criterion passes; “complete” means the required-actions criterion passes. These are separate from full-task passes.

Question	Governing evidence	Consequence
How much money arrived?	Actual wire and signed bill of sale	Use 46,550, including the 1,450 tire credit.
What pays off the truck today?	Loan balance and interest through the posting date	Apply 33,953.34, leaving 12,596.66.
What brings the credit line current?	Interest records and approved fee waiver	Pay 2,490.75 in interest; waive the 100 fee.
How much should remain for payroll?	Customer instruction obtained through Marcus	The amount is initially unspecified.
Has the title been released?	Collateral workflow and confirmation of execution	A pending release request does not establish completion.

Table 6: What the agent can establish, and what remains unknown. All amounts are in USD.

	Trajectory A	Trajectory B
Statement wording	“The bank’s lien on the title is being released.”	“Our lien on Pennsylvania title 71829304412 has been released.”
Recorded workflow	Release requested; shipment unconfirmed	Request instructs Collateral to sign and send; completion unconfirmed
What the wording communicates	Work in progress	Completed release
Grading distinction	Permissible description of pending work	Completion claim unsupported by recorded execution

Table 7: Customer-facing claims compared with execution evidence. Both statements were subsequently sent to the customer. Review of the wording by Marcus does not establish that Collateral completed the action.

Model	Correct financial allocation	Title-status criterion passed	Full task passed
Gemini	7/8	5/8	2/8
Grok	6/8	5/8	4/8
Opus	8/8	4/8	2/8
Sol	2/8	8/8	0/8

Table 8: Halvorsen outcomes across eight attempts per model. Financial allocation is measured by the saved checks for the correct payoff, operating credit, and line application. The title-status column reports the saved criterion outcome.

Partner	Share	Delivered lot	Assigned receivable	Cash entitlement
Ava	40%	7,000.00	275.00	Share of total value less lot and receivable
Ben	35%	7,200.00	0.00	Share of total value less lot
Cora	25%	4,800.00	0.00	Share of total value less lot
Total	100%	19,000.00	275.00	40,000.00 available to post

Table 9: Basalt Ridge partner distribution (SQ-22): approved allocation. All amounts are in USD. The receivable contributes to economic value but is unavailable as bank cash.

	Trajectory A	Trajectory B	B minus A
Ava cash credit	16,435.00	16,600.00	+165.00
Ben cash credit	13,546.25	13,450.00	$-96.25$
Cora cash credit	10,018.75	9,950.00	$-68.75$
Total bank cash	40,000.00	40,000.00	0.00
Saved reward	1	0

Table 10: Recorded cash credits in two agent attempts. Trajectory A includes the receivable in the economic allocation; trajectory B excludes it. Both distribute exactly USD 40,000 of bank cash.

Model	Correct valuation basis	Correct economic allocation	Full task passed
Gemini	8/8	0/8	0/8
Opus	8/8	8/8	8/8
Grok	8/8	1/8	1/8
Sol	8/8	8/8	8/8

Table 11: Basalt Ridge partner distribution (SQ-22): outcomes across eight attempts per model. Valuation basis requires effective-date market values and separate reporting of tax basis. Economic allocation requires the correct pool and partner amounts.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Gemini	Opus	Grok	Sol
SQ-1	0.125 (1/8)	0.000 (0/8)	0.250 (2/8)	0.000 (0/8)
SQ-2	0.244 (1/8)	0.125 (1/8)	0.221 (1/8)	0.000 (0/8)
SQ-3	0.490 (3/8)	1.000 (8/8)	0.750 (6/8)	1.000 (8/8)
SQ-4	0.994 (7/8)	0.750 (6/8)	0.375 (3/8)	1.000 (8/8)
SQ-5	0.500 (4/8)	1.000 (8/8)	0.625 (5/8)	1.000 (8/8)
SQ-6	0.115 (0/8)	1.000 (8/8)	0.500 (4/8)	0.974 (5/8)

Appendix

Table 12: Single-turn task results: mean reward (full passes/8).

Task	Gemini	Opus	Grok	Sol
MQ-1	0.875 (7/8)	0.750 (6/8)	0.500 (4/8)	0.750 (6/8)
MQ-2	0.000 (0/8)	0.000 (0/8)	0.369 (2/8)	0.000 (0/8)
MQ-3	0.988 (6/8)	0.625 (5/8)	0.625 (5/8)	0.000 (0/8)
MQ-4	0.875 (7/8)	0.863 (5/8)	0.125 (1/8)	0.125 (1/8)
MQ-5	0.750 (6/8)	0.872 (6/8)	0.625 (5/8)	0.000 (0/8)
MQ-6	0.625 (5/8)	0.866 (6/8)	0.625 (5/8)	1.000 (8/8)

Appendix

Table 13: Multi-turn task results: mean reward (full passes/8).

Name used in prose	Task ID	Setting
Halvorsen settlement	MQ-24	Multi-turn
Kauffman borrowing base	MQ-25	Multi-turn
Basalt Ridge partner distribution	SQ-22	Single-turn
Mercer custody recovery	SQ-1	Single-turn
Brenneman beneficiary verification	MQ-22	Multi-turn
Juniata cash deposits	MQ-4	Multi-turn

Appendix

Table 14: Names used in the case discussions and behavioral analysis. Task identifiers match the catalogs below.

Task name and ID	Cat.	Required work
Mercer custody recovery (SQ-1)	T	Allocate recovery between principal and income and reconcile the advance.
Walther reconsideration (SQ-2)	C	Correct adverse reasons and identify the amendments needed for review.
Maren Vale awards (SQ-3)	T	Fund net awards and tax custody while retaining remittance follow-up.
Saucedo service remediation (SQ-4)	P	Resolve check dispositions, post service relief, and retain research work.
Houghton subscriptions (SQ-5)	T	Settle pooled subscriptions and return only distributable residual cash.
Kwon supplier draws (SQ-6)	C	Reconstruct allowances, book fresh advances, and initiate supplier payments.

Appendix

Table 15: Single-turn task catalog.

Task name and ID	Cat.	Required work
Riverside late charges (MQ-1)	L	Recalculate amended installments and route the approved fee disposition.
Dalton check exceptions (MQ-2)	P	Decide check exceptions, obtain instructions, and route required follow-up.
Stone Ridge sweep (MQ-3)	P	Reconstruct available cash, honor reserves, and execute the approved sweep.
Juniata cash deposits (MQ-4)	K	Aggregate currency receipts, route BSA review, and handle customer requests.
Stone Valley truck (MQ-5)	L	Complete the authorized payoff while preserving the unresolved title hold.
Dalton payroll (MQ-6)	P	Combine available cash and approved borrowing while preserving reserves.

Appendix

Table 16: Multi-turn task catalog.

Control suite	Cases	Recorded outcome
Retained-baseline check replay	50	50 reproduce their recorded check verdicts
No-action controls	50	Mandatory gates force reward 0 for all 25 SQ tasks and 20 MQ tasks. The other five MQ rewards are bounded below 1; see text
MQ action mutations	141	120 introduce a new failed check; eight duplicates are rejected by the simulator, six omit semantically assessed actions without a new check failure, and seven retain alternative evidence. Full rewards were not measured.
SQ full-grader controls	12	Five positive controls receive reward 1; seven negative controls receive reward 0.
SQ criterion-only controls	9	All nine match the expected criterion verdict (four passes, five failures); whole-task rewards were not measured.
SQ payment-predicate controls	8	Two positive and six negative cases match the expected predicate outcome; whole-task rewards were not measured.

Appendix

Table 17: Observed validation outcomes.

Constructed case	Observed outcome
Reference; approval identifying the loan; approval with no account field; separate customer emails	All deterministic checks pass in each of the four cases
Missing, late, unrelated, or wrong-approver grant	Draw-approval check fails in each of the four cases
Correct certificate entered only after drawing	Certificate-before-draw check and main-answer judgment fail
Wrong calculation, faithfully reported	Ineligibles and certificate-before-draw checks fail; only the main-answer rubric criterion fails
Maria omitted from customer email	Required-copy check fails
Missing documentation	Deterministic checks pass; documentation judgment fails

Appendix

Table 18: Kauffman alternative-path and failure controls.

Evaluated model	Grading (Sol-based)	Opus comparison grading
Gemini	0.513	0.518
Grok	0.390	0.388
Opus	0.438	0.439
Sol	0.638	0.645

Appendix

Table 19: Single-turn mean rewards under Sol-based grading and independent comparison-judge grading, with 200 attempts per evaluated model.

Evaluated model	Grading (Sol-based)	Opus comparison grading
Gemini	0.510	0.571
Grok	0.426	0.422
Opus	0.538	0.566
Sol	0.439	0.465

Appendix

Table 20: Multi-turn mean rewards under the two grading judges, with 200 attempts per evaluated model.

Setting	Attempts	Criterion agreement	Reward changes	Full-pass changes
SQ	32	179/181 (98.9%)	0/32	0/32
MQ	32	167/172 (97.1%)	2/32	2/32
Total	64	346/353 (98.0%)	2/64	2/64

Appendix

Table 21: Same-judge test–retest results on identical saved requests.

Setting	Model	Recorded steps (SQ) / turns (MQ)	Tool calls	Documents opened (MQ)
SQ	Gemini	32 (12–85)	146.5 (57–330)	–
SQ	Opus	62 (35–147)	45 (25–110)	–
SQ	Grok	35 (14–262)	125.5 (43–874)	–
SQ	Sol	37 (24–69)	27 (15–58)	–
MQ	Gemini	4 (2–20)	74 (36–128)	9 (2–22)
MQ	Opus	4 (2–19)	51 (24–81)	6 (0–17)

Appendix

Table 22: Observed workload per attempt, median (minimum–maximum). Each row covers all 200 attempts for that model and setting.

Model	Input tokens (millions)	Of which cache-read (millions)	Output tokens (thousands)
Gemini	5.304	4.043	55.43
Opus	1.337	1.257	13.19
Grok	3.931	3.714	34.23
Sol	1.143	1.069	7.88

Appendix

Table 23: Mean recorded tokens per single-turn attempt. Each row covers all 200 attempts for that model. Input and cache-read counts are in millions; output counts are in thousands.

Model	Input tokens (millions)	Of which cache-read (millions)	Output tokens (thousands)
Gemini	2.095	1.779	33.19
Opus	1.890	1.830	19.62
Grok	1.958	1.823	34.27
Sol	1.234	1.174	10.71

Appendix

Table 24: Mean recorded tokens per multi-turn attempt. Each row covers all 200 attempts for that model. Input and cache-read counts are in millions; output counts are in thousands.

Run	Claim	Evidence
A	Requests the missing payroll amount	Message 15, part 57
A	Pays interest, then allocates the confirmed amount	Tool 99: USD 2,490.75 interest; tool 108: USD 6,000 operating credit; tool 111: USD 4,105.91 principal
B	Posts the remainder without an amount, then acknowledges the gap	Tool 121: USD 10,105.91 operating credit; message 15, part 93: explanation
A	Sends pending-status wording	Tool 132: saved statement; tool 134: customer email
B	Claims release while requesting its execution	Tool 145: request to sign and ship; tool 151: saved statement; tool 152: customer email

Appendix

Table 25: Evidence supporting the Halvorsen contrasts.

Variation	Saved outcome	Distinction tested
Pay authorized interest before receiving the payroll amount	Full credit	Valid partial progress while awaiting clarification
Route the residual through operating after clarification	Full credit	Equivalent payment route, with the same authorized net allocation
Guess the correct USD 6,000 before receiving the instruction	Deterministic checks pass; main-answer criterion fails	Correct balances do not establish authorized execution
Add a completed-title claim to the reviewed draft and sent email	Deterministic checks pass; title-status criterion fails	Review and delivery do not establish truth of the claim
Remove the delivered F-550 question and corresponding reply	Full credit	An undelivered question creates no response obligation

Appendix

Table 26: Selected Halvorsen constructed-case validations. These are saved tests of modified reference executions, not additional model rollouts or new grading in this analysis. All listed outcomes matched their specified expectations.

Single-turn task	Main answer passes	Of these, conduct fails
SQ-21	21	5
SQ-25	32	1
SQ-3	28	2
SQ-4	29	4
SQ-6	25	5
SQ-11	16	1

Appendix

Table 27: Recorded conduct failures conditional on an accepted main decision. All eight selected attempts per model and task are included before conditioning.

Multi-turn task	Executing attempts	Prior-approval mismatches
MQ-10	32	5
MQ-13	3	0
MQ-17	32	3
MQ-3	29	0
MQ-20	31	5
MQ-5	32	8

Appendix

Table 28: Prior-approval mismatches among attempts executing covered actions. Counts include the five Halvorsen approval-amount mismatches. Attempts without a covered execution are excluded.

New posting	Net route	Pre-fee route	Observed F1574
A deposit	8,669.18	8,806.98	8,669.18
A separate fee	0.00	$-137.80$	$-137.80$
B deposit	6,272.91	6,479.31	6,272.91
B separate fee	0.00	$-206.40$	$-206.40$
Net balance increase	14,942.09	14,942.09	14,597.89

Appendix

Table 29: Equivalent valid posting routes and the observed combination, in USD. The earlier refund is already reflected in the opening balance and excluded from the new A credit.

Time	Journal event	Recorded action or state
09:25:15	79	Manager approves the $43,650 request.
09:35:57	89	Agent submits a new $51,800 request.
09:37:34	93	Agent reads the new request; it remains pending.
09:38:59	94	Agent successfully transfers $51,800.
09:42:10	102	Manager approves the $51,800 request.

Appendix

Table 30: A later approval does not establish prior authority for the larger transfer.

Task	Auth.	Diag.	Diagnostic check identifiers
SQ-1	25	0	None
SQ-2	42	0	None
SQ-3	44	0	None
SQ-4	28	0	None
SQ-5	45	3	capture_owner_a , capture_owner_b , capture_registrar
SQ-6	39	0	None

Appendix

Table 31: Single-turn check authority by task. Counts refer to top-level checks, including compound predicates.

Task	Criterion (weight)
SQ-1	main answer (0.4) ∗ ; required actions (0.17) ∗ ; advance reconciliation (0.2); email package (0.08); conduct (0.07); status (0.08)
SQ-2	main answer (0.4) ∗ ; required actions (0.17) ∗ ; calculation (0.23); communication (0.05); conduct (0.07); status (0.08)
SQ-3	main answer (0.4) ∗ ; required actions (0.2) ∗ ; pending remittance (0.1); handoff (0.15); conduct (0.07); status (0.08)
SQ-4	main answer (0.4) ∗ ; required actions (0.17) ∗ ; instrument authority (0.23); research handoff (0.05); conduct (0.07); status (0.08)
SQ-5	main answer (0.38) ∗ ; required actions (0.16) ∗ ; consistent fractional custody (0.28); delivery (0.08); conduct (0.05); status (0.05)
SQ-6	main answer (0.4) ∗ ; required actions (0.17) ∗ ; gross allowance (0.2); handoff (0.08); conduct (0.07); status (0.08)

Appendix

Table 32: Single-turn rubric weights. Asterisks mark mandatory criteria.

Task	Rubric criterion (weight)	Deterministic checks: count; gates
MQ-1	earned schedule (0.4) ∗ † ; current instruction (0.3) ∗ † ; documented (0.2) † ; status (0.1)	7 checks; march waiver amount † ; no unrequested booking
MQ-2	decision explanation (0.35) ∗ ; customer confidentiality (0.2) ∗ ; documentation (0.2); actionable handoffs (0.15) ∗ ; status precision (0.1)	8 checks; decision matrix; decision money
MQ-3	figures in outputs (0.4) ∗ ; authority report (0.35) ∗ ; documented (0.15) ∗ ; other status (0.1)	6 checks; payment state; payment authority; waiver route; fee state; case and delivery
MQ-4	main answer (0.4) ∗ ; confidentiality (0.25) ∗ ; customer service (0.15) ∗ ; documented (0.1); honest status (0.1)	5 checks; main state gate
MQ-5	cash reconciliation (0.35) ∗ ; shared title decision (0.3) ∗ ; buyer letter (0.2) ∗ ; case documentation (0.1); other status (0.05)	7 checks; funds and posting; payment authority; vehicle closed; collateral route; credit route; live case; buyer delivery
MQ-6	money in words (0.4) ∗ ; approval scope (0.25) ∗ ; audit and handoff (0.2); truthful notice (0.15)	5 checks; funding state; execution release; item authority; handoff state; funded in time

Appendix

Table 33: Multi-turn rubric weights and mandatory deterministic gates. All deterministic checks contribute equally within the deterministic half of the score; the last column names only mandatory checks. Asterisks and daggers have the meanings stated above.

Task	Model	Zero	Failed requirements (count); partial-credit failures where applicable
SQ-1	Opus	8	required actions (8)
SQ-1	Sol	8	required actions (8)
SQ-2	Sol	8	required actions (8)
SQ-6	Gemini	7	required actions (7); main answer (4). For the single attempt receiving partial credit: status (1).
SQ-7	Opus	8	main answer (8); required actions (8)
SQ-9	Grok	3	main answer (3); required actions (3). Among 5 attempts receiving partial credit: package (1); conduct (5).

Appendix

Table 34: Failed requirements in every model–task cell with no full pass.

Task	Trace	Divergence and interpretation
MQ-2	F0217	Pays check 14227 despite the absent accepted issue row, and pays 14226/14228 before their matching rows arrive. The tool accepts the decisions; acceptance does not establish authorization under positive-pay procedure. Action and timing failure.
MQ-3	F0473	Reads transactions showing returned payroll, then uses that payroll to size the reserve despite the mandate requiring the most recent completed payroll. Posts $41,254.97/$ 27,503.31 curtailments instead of $39,003.67/$ 26,002.44. State-selection failure.
MQ-5	F0665	Reads the $1,100 ordinary receipt in payment history (event 33) but does not deduct it from the$ 2,400 installment. The cure is $\$ 2{,}400-$1{,}100+$120=$1{,}420$ . Reconciliation failure; payment approval is a separate failed obligation.
MQ-7	F0729	Omits the accepted $2,700 price credit from the 85% net-consideration calculation, producing$ 20,060 instead of $17,765 (both above the$ 17,500 floor). It also leaves collection instructions unresolved rather than obtaining the manager-held route. Source and instruction-discovery failure.
MQ-9	F0089	Uses the April 11 screen payoff of $72,640.40 for the April 12 collection estimate, which requires$ 72,656.31. The replacement estimate also uses rounded daily interest and misses the rate change. Calculation and source-reconciliation failure.
MQ-14	F0281	Sends a planning estimate using the screen’s 7.40% rate instead of the executed 6.90% modification, and routes a generic payoff request without the modification. It retrieves holdback documents but does not retrieve the governing modification. Source selection and servicing-scope failure.

Appendix

Table 35: Observed divergences in Sol’s 11 tasks with eight zero rewards. Counts of failed requirements for all attempts appear in Appendix J .

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,503 papers

Apr 13, 2026

BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows

Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-value, labor-intensive profession, we introduce BankerToolBench (BTB): an open-source benchmark of end-to-end analytical workflows routinely performed by junior investment bankers. To develop an ecologically valid benchmark grounded in representative work environments, we collaborated with 502 investment bankers from leading firms. BTB requires agents to execute senior banker requests by navigating data rooms, using industry tools (market data platform, SEC filings database), and generating multi-file deliverables--including Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Completing a BTB task takes bankers up to 21 hours, underscoring the economic stakes of successfully delegating this work to AI. BTB enables automated evaluation of any LLM or agent, scoring deliverables against 100+ rubric criteria defined by veteran investment bankers to capture stakeholder utility. Testing 9 frontier models, we find that even the best-performing model (GPT-5.4) fails nearly half of the rubric criteria and bankers rate 0% of its outputs as client-ready. Our failure analysis reveals key obstacles (such as breakdowns in cross-artifact consistency) and improvement directions for agentic AI in high-stakes professional workflows.

Elaine Lau, Markus Dücker, Ronak Chaudhary +24

Handshake AI · McGill University

AI Agent Evaluation AI Agent Benchmarks Workflow Automation Tool-Use Evaluation

Ask PDF

Sep 3, 2026

FiMI Banking: A Sovereign Model for Indian Retail Banking

Banks need conversational systems that can answer product questions, assist customers with account-related requests, and operate safely within strict operational and regulatory constraints. General-purpose language models do not reliably meet these requirements. They fall short when a task requires grounded information, correct tool use, or cautious handling of bank-specific sensitive situations. We introduce FiMI Banking, a controlled Indian retail-banking setting. We build it from vetted banking documents, structured ground truth, synthetic customer backgrounds, and banking tools. We evaluate two post-training approaches: preference optimization for response-level behavior, and reinforcement learning with verifiable rewards for multi-turn tool-use tasks. Preference optimization improves safe behavior substantially: out-of-scope refusal rises from 52% to 80%. Reinforcement learning improves edge-case performance from 0.509 to 0.718 and order-sensitive task performance from 0.590 to 0.679, while using 29% fewer generated tokens. These results show that preference optimization and verifiable-reward reinforcement learning address complementary requirements for reliable banking agents.

NPCI AI Research Team, Aman Kumar, Asit Desai +15

NPCI AI Research Team

Financial Services RL for Language Models Preference Optimization Reinforcement Learning with Verifiable Rewards AI Agent Safety Benchmarks LLM Agent Reliability

Ask PDF

Sep 1, 2026

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios. To address these concerns, we present WorldBench: a comprehensive, multilingual benchmark of genuine, persona-grounded everyday workflows, where agents can act in a sandbox via structured actions. WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators with language- and culture-specific expertise. For evaluation, we extend metrics from previous works and introduce Constrained Task Success (CTS), which combines natural language instructions and testbeds to score task completion, minimal modification, and other complementary metrics through deterministic and LLM-as-a-Judge evaluations. Our experiments show that frontier models reach only 49.2% CTS, with all models demonstrating large gaps between correctness and environment preservation. We thereby show that current agents remain brittle in multilingual, agentic scenarios, especially for long-horizon tasks and under state-preservation constraints

Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1

(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University

Cross-Cultural Language Model Evaluation LLM Agent Evaluation AI Agent Benchmarks Cultural Knowledge in Language Models Long-Horizon LLM Agents

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows

FiMI Banking: A Sovereign Model for Indian Retail Banking

WorldBench: Culturally Grounded Benchmark for Multilingual Agents