As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary. Such measurement, however, is constrained by limited access to proprietary or privacy-sensitive data. Existing benchmarks therefore often rely on publicly available data, human- and/or LLM-authored tasks, or simplified settings. We introduce FinancialAuditBench, a benchmark for evaluating agents on financial statement audit tasks, along with a framework for systematically generating synthetic engagements. Our task generation framework leverages differentially private aggregate statistics from historical audits along with audit expertise contributed through over 1,100 hours of benchmark development and review. FinancialAuditBench consists of 90 tasks spanning workpaper completion and review across six synthetic audit engagements, each containing an average of 179 files. Evaluation on eleven frontier models shows that while agents complete substantial portions of staff-level audit tasks well, they sometimes perform inappropriate procedures or produce incorrect documentation. Beyond financial auditing, our framework offers an approach for systematically generating synthetic tasks for model evaluation and training in privacy-sensitive domains.
Figures & tables
Figure 1: Task structure in FinancialAuditBench. The agent uses the provided supporting documents to perform the appropriate audit procedures in a workpaper template. The completed workpaper is then evaluated against a rubric.
Figure 2: A toy example illustrating the three node types used in the task generation graph. The source node specifies the industry, which determines the distributions from which annual revenue and the payroll-to-revenue ratio are sampled. The deterministic node calculates annual payroll by multiplying the two sampled values.
Model
Weights
Strict pass (%)
Partial credit (%)
Mean cost (USD)
Mean time (min)
Claude Opus 5.5
Closed
69.17
95.84
3.30
11.21
Claude Fable 5.1
Closed
65.83
94.81
6.49
13.03
Grok 4.6
Closed
47.22
91.12
1.62
11.99
GPT-5.6 Sol
Closed
44.17
89.41
0.91
3.78
GPT-6 Astra
Closed
43.61
90.95
2.96
8.00
GPT-5.6 Terra
Closed
40.28
88.66
0.40
2.06
Table 1: Workpaper-completion performance across frontier models. Strict pass rates and partial-credit scores are averaged across tasks and runs. Cost and runtime are means per trajectory.
Figure 3: Reliability across eight independent runs. Solid lines show pass^k and dotted lines show pass@k. Both metrics use strict task success and are averaged across tasks. Exact values are provided in Appendix D.3 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Query family
DG Δ2≤1
DG Δ2≤60
JointExp
ρ
Financial ratios
6
0
6
0.04944
Record features
16
3
0
0.14903
AJE and AP summaries
2
3
0
0.05203
Total
24
6
6
0.25051
Appendix
Table 2: Privacy accounting by query family. DG denotes the discrete Gaussian mechanism.
Workpaper
Estimated manual workpaper time
Median agent time (min)
Average agent strict pass (%)
Accounts payable
4–6 hr
9
68.18
Accounts receivable
6–10 hr
12
4.17
Cash
3–5 hr
8
56.06
Debt
3–5 hr
8
52.08
Fixed assets
2–4 hr
8
54.17
Journal-entry completeness
2–4 hr
7
94.51
Appendix
Table 3: Estimated manual completion times (without the use of any AI or automation tools), measured agent runtimes, and agent pass rates for the benchmark workpapers. Agent runtimes are medians rounded to the nearest minute.
Model
Before review (%)
After review (%)
Change (pp)
Claude Opus 5.5
28.89 / 87.34
62.22 / 94.57
+33.33 / +7.23
Grok 4.6
28.89 / 87.34
46.67 / 91.07
+17.78 / +3.73
GPT-5.6 Terra
28.89 / 87.34
42.22 / 89.90
+13.33 / +2.56
Muse Spark 1.3
28.89 / 87.34
42.22 / 90.48
+13.33 / +3.14
DeepSeek V4.1 Flash
28.89 / 87.34
37.78 / 89.59
+8.89 / +2.25
GLM-5.2
28.89 / 87.34
37.78 / 89.74
+8.89 / +2.39
Appendix
Table 4: Review-task results. Strict pass / partial-credit scores before and after workpaper review; changes are in percentage points. Scores before review are identical because all models receive the same starting workpapers.
Model
k=1
k=2
k=3
k=4
k=5
k=6
k=7
k=8
Claude Opus 5.5
0.69
0.60
0.54
0.50
0.48
0.46
0.45
0.44
Claude Fable 5.1
0.66
0.56
0.49
0.44
0.40
0.37
0.34
0.31
Grok 4.6
0.47
0.38
0.31
0.27
0.24
0.21
0.19
0.18
GPT-5.6 Sol
0.44
0.33
0.27
0.23
0.20
0.19
0.18
0.18
GPT-6 Astra
0.44
0.29
0.22
0.18
0.16
0.15
0.14
0.13
GPT-5.6 Terra
0.40
0.28
0.21
0.17
0.13
0.11
0.09
0.07
Appendix
Table 5: Pass^k across eight independent runs: the probability that all k attempts strictly pass. Values are fractions.
Model
k=1
k=2
k=3
k=4
k=5
k=6
k=7
k=8
Claude Opus 5.5
0.69
0.79
0.82
0.84
0.85
0.85
0.86
0.87
Claude Fable 5.1
0.66
0.76
0.79
0.81
0.82
0.82
0.82
0.82
Grok 4.6
0.47
0.57
0.60
0.62
0.63
0.64
0.64
0.64
GPT-5.6 Sol
0.44
0.55
0.60
0.62
0.63
0.63
0.64
0.64
GPT-6 Astra
0.44
0.58
0.65
0.69
0.72
0.73
0.75
0.76
GPT-5.6 Terra
0.40
0.52
0.58
0.60
0.62
0.63
0.64
0.64
Appendix
Table 6: Pass@k across eight independent runs: the probability of at least one strict pass within k attempts.
Reasoning effort
Strict pass (%)
Partial credit (%)
Mean runtime (min)
Mean cost ($)
Low
23.61
85.71
3.08
1.32
Medium
43.61
90.95
8.00
2.96
High
55.00
92.71
9.59
3.97
Xhigh
59.44
94.16
10.95
4.49
Max
71.11
95.83
24.94
8.27
Appendix
Table 7: Reasoning effort ablation with GPT-6 Astra. Runtime and model cost are means per trajectory, with eight independent runs per task.
Model
API
Reasoning effort
Temperature
Claude Opus 5.5
claude-opus-5-5
medium
–
Claude Fable 5.1
claude-fable-5-1
medium
–
Grok 4.6
x-ai/grok-4.6
medium
–
GPT-5.6 Sol
gpt-5.6-sol
medium
–
GPT-6 Astra
gpt-6-astra
medium
–
GPT-5.6 Terra
gpt-5.6-terra
medium
–
Appendix
Table 8: Models and inference settings for the main and review experiments. Note that Claude Fable 5.1 is evaluated only in the main experiments.
Workpaper area
Claude Opus 5.5
Claude Fable 5.1
Grok 4.6
GPT-5.6 Sol
GPT-6 Astra
GPT-5.6 Terra
Muse Spark 1.3
DeepSeek V4.1 Flash
GLM-5.2
Kimi K3
Gemini 3.8 Flash
Accounts payable
100.00
98.96
99.48
98.03
94.44
96.06
98.44
90.57
91.44
94.04
92.77
Accounts receivable
89.43
87.01
81.30
83.78
80.63
79.47
84.58
80.23
80.53
84.08
81.20
Cash
99.00
99.43
97.66
96.16
93.67
96.80
94.09
95.21
91.42
88.06
89.45
Debt
99.85
99.08
96.38
94.35
92.15
95.50
94.50
94.60
91.15
89.68
95.11
Fixed assets
95.82
94.82
95.49
89.42
88.21
93.39
95.87
93.70
94.94
92.58
87.64
Journal-entry completeness
100.00
99.58
99.65
100.00
100.00
96.04
99.24
96.46
99.58
93.19
100.00
Appendix
Table 9: Partial-credit scores by workpaper area (%).
Figure 5: System prompt provided to the agent in the preparation and review settings.
Figure 6: Example task prompt for a workpaper completion task.
Figure 7: Example task prompt for a workpaper review task.
Figure 8: User prompt for completing all workpapers in a binder with computer-use tools. The local binder-folder path is replaced with a placeholder.
Figure 9: Final desktop overview during the computer-use experiment with ChatGPT Computer-Use.