cs.AIOct 20, 2026

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

Authors: Pei Liu

Organizations: School of Engineering and Applied Science, George Washington University, Washington, DC, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Practitioner guidance holds that a large language model reviews a document better when the work is divided among agents. We test a narrow form of it in a controlled comparison of single-agent and multi-agent setups on a financial task, reviewing credit assessment memos against requirements. One agent checks six requirements in one call. A team of three on the same model checks the same memo, two requirements each, in the same words, with no roles or later stage. Labels are known for every requirement on 300 synthetic memos, so misses and false alarms are measured separately. Two primary predictions are tested on four models at a Bonferroni-corrected level. Under an instruction to decide each requirement once, the team catches 0.08 to 0.10 more of the defects that carry an explanation on the three models whose single agent misses some, at the corrected level on two. It raises more false alarms on all four, at the corrected level on three. Balanced accuracy, an exploratory measure, changes by 0.02 or less, and the team uses 2.2 to 2.4 times the tokens. Showing each agent the requirements it was not given reduces the added false alarms on the two Claude models, at the corrected level on one. In exploratory comparisons, sampling the single agent three times and flagging on any draw gives higher balanced accuracy than the team on those models, and the team's results depend on the instruction.

Figures & tables

PairRequirement
AR1. Every statement of the income used in the affordability or servicing calculation must give the monthly income stated in Employment.
AR2. Every statement of the approved facility limit must give the same amount, and any stated change to the limit must arrive at that amount.
BR3. Every sum the memo writes out about monthly income must be arithmetically correct, whether it adds an amount, deducts one, or compares the income with an earlier figure.
BR4. Every income multiple quoted for the facility limit must follow from the approved limit and the monthly income stated in Employment.
CR5. Every statement about the repayment record must agree with the six periods shown in the repayment history.
CR6. Every stated period of employment or sector experience must be possible given the applicant’s age and the age at which education was completed.
Table 1: The six requirements and the three pairs in which the team receives them.
ModelArmHbareH^{\mathrm{bare}}HmaskedH^{\mathrm{masked}}HarithH^{\mathrm{arith}}HHFFBBKa/KsingleK_{a}/K_{\mathrm{single}}
Qwen3.5 9BSingle agent0.7170.5590.5740.6130.1600.7261.00
 Team0.8490.6420.7220.7330.2540.7402.36
 Scoped team0.8980.6460.7300.7530.2730.7402.55
 Three draws, majority0.7560.4020.6260.5890.1400.7243.11
 Three draws, any0.9020.7950.8700.8540.3510.7513.11
Qwen3.5 27BSingle agent0.9710.7340.6910.7920.0470.8731.00
Table 2: Share of defects caught by class and over all defects, HH , false alarms on clean cells, FF , balanced accuracy, BB , and tokens relative to the single agent. The majority rule flags when at least two of three draws flag, and the any rule when at least one does. All 300 memos, decide-once instruction.
Figure 1: Each arm as an operating point, false alarms on clean cells against the share of all defects caught, with the Claude models enlarged on the right. A thick segment joins a model’s single agent to its team, and a star marks the scoped team. A thin line joins three rules applied to three draws of the single agent’s prompt, flagging when all, at least two or any of the draws flag. Bars on the single agent and the team are 95 percent intervals. Grey lines join points of equal balanced accuracy.
Figure 2: The team against the single agent, on one scale. The contrasts in masked defects caught, ΔHmasked\Delta H^{\mathrm{masked}} , and in false alarms, ΔF\Delta F , are primary, with intervals at the corrected level. The contrast in balanced accuracy, ΔB\Delta B , is exploratory, with 95 percent intervals. A filled marker has an interval that excludes zero, and the number beside each row is the estimate. Table A1 gives the bounds.
Figure 3: The scoped team, on the scale of Figure 2 . Its contrast with the single agent in masked defects caught and its contrast with the team in false alarms are the two pre-specified contrasts, with intervals at the corrected level. Its contrast with the team in all defects caught is exploratory, with 95 percent intervals. Markers and numbers are as in Figure 2 . Table A2 gives the bounds.
ModelΔH\Delta H , withΔH\Delta H , withoutΔF\Delta F , withΔF\Delta F , withoutΔB\Delta B , withΔB\Delta B , without
Qwen3.5 9B+0.133 [0.050, 0.214]+0.084 [0.000, 0.164]+0.110 [0.065, 0.157]+0.126 [0.072, 0.179]+0.012 [ −- 0.037, 0.060]−- 0.021 [ −- 0.068, 0.026]
Qwen3.5 27B+0.071 [0.017, 0.126]+0.004 [ −- 0.046, 0.052]+0.035 [0.006, 0.064]+0.123 [0.050, 0.192]+0.018 [ −- 0.013, 0.049]−- 0.059 [ −- 0.105, −- 0.013]
Haiku 5.5+0.022 [ −- 0.022, 0.066]+0.071 [0.026, 0.118]+0.008 [ −- 0.008, 0.026]+0.019 [0.000, 0.038]+0.007 [ −- 0.016, 0.030]+0.026 [0.001, 0.053]
Sonnet 5.5−- 0.004 [ −- 0.019, 0.009]−- 0.013 [ −- 0.030, 0.000]+0.008 [0.000, 0.018]+0.021 [0.008, 0.037]−- 0.006 [ −- 0.015, 0.002]−- 0.017 [ −- 0.028, −- 0.008]
Table 3: The contrasts ΔH\Delta H over all defects, ΔF\Delta F and ΔB\Delta B on the 100 paired memos, with the decide-once instruction and without it. 95 percent intervals.
ModelΔF/ΔH\Delta F/\Delta Hp=0.37p=0.37p=0.05p=0.05p=0.02p=0.02p=0.01p=0.01
Qwen3.5 9B0.77 [0.48, 1.33]1.3153877
Qwen3.5 27B0.69 [0.21, 2.73]1.2133468
Haiku 5.50.22 [0.01, 0.65]0.441122
Table 4: Team against single agent under the decide-once instruction. The second column is ΔF/ΔH\Delta F/\Delta H , with ΔH\Delta H over all defects and a 95 percent interval. The others are the break-even value of wM/wFw_{M}/w_{F} at four values of the defect share pp , assuming the rates carry over.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

ModelΔHmasked\Delta H^{\mathrm{masked}}ΔF\Delta FΔB\Delta B
Qwen3.5 9B+0.083 [ −- 0.028, 0.198]+0.093 [0.056, 0.128]+0.014 [ −- 0.013, 0.039]
Qwen3.5 27B+0.083 [0.004, 0.164]+0.029 [0.006, 0.051]+0.007 [ −- 0.012, 0.025]
Haiku 5.5+0.100 [0.028, 0.175]+0.011 [ −- 0.002, 0.026]+0.020 [0.005, 0.036]
Sonnet 5.5−- 0.004 [ −- 0.019, 0.000]+0.013 [0.006, 0.022]−- 0.008 [ −- 0.013, −- 0.003]
Appendix
Table A1: The contrasts of Figure 2 , computed from unrounded rates. ΔHmasked\Delta H^{\mathrm{masked}} and ΔF\Delta F are the two primary contrasts, with intervals at the corrected level. ΔB\Delta B is exploratory, with 95 percent intervals.
ModelHscopedmasked−HsinglemaskedH^{\mathrm{masked}}_{\mathrm{scoped}}-H^{\mathrm{masked}}_{\mathrm{single}}Fscoped−FteamF_{\mathrm{scoped}}-F_{\mathrm{team}}Hscoped−HteamH_{\mathrm{scoped}}-H_{\mathrm{team}}
Qwen3.5 9B+0.087 [ −- 0.010, 0.196]+0.019 [ −- 0.018, 0.055]+0.020 [ −- 0.020, 0.058]
Qwen3.5 27B+0.074 [0.000, 0.144]−- 0.013 [ −- 0.034, 0.007]−- 0.020 [ −- 0.045, 0.006]
Haiku 5.5+0.092 [0.014, 0.170]−- 0.011 [ −- 0.024, 0.002]−- 0.018 [ −- 0.042, 0.006]
Sonnet 5.5at ceiling−- 0.011 [ −- 0.019, −- 0.003]+0.003 [ −- 0.005, 0.012]
Appendix
Table A2: The contrasts of Figure 3 for the scoped team, computed from unrounded rates. The first two are its pre-specified contrasts, with intervals at the corrected level. The last is exploratory, with 95 percent intervals.
ModelHmaskedH^{\mathrm{masked}}HarithH^{\mathrm{arith}}FF
Qwen3.5 9B+0.179 [0.105, 0.256]+0.163 [0.093, 0.230]+0.046 [0.012, 0.082]
Qwen3.5 27B+0.058 [0.004, 0.112]+0.088 [0.018, 0.159]+0.020 [0.000, 0.039]
Haiku 5.5+0.075 [0.027, 0.123]+0.044 [0.009, 0.079]+0.002 [ −- 0.005, 0.009]
Sonnet 5.50.000+0.003 [ −- 0.010, 0.017]0.000
Qwen3.5 9B, single call @Temp=0−- 0.037 [ −- 0.162, 0.086]+0.188 [0.065, 0.313]+0.005 [ −- 0.039, 0.051]
Qwen3.5 27B, single call @Temp=0+0.062 [ −- 0.045, 0.169]+0.175 [0.076, 0.284]+0.094 [0.052, 0.140]
Appendix
Table A3: Change in the single agent’s rates when the instruction is removed, the rate without it minus the rate with it, on the 100 paired memos. For the Claude models the single call and the three draws are pooled, and for the Qwen3.5 models the three draws at temperature 0.7. The last two rows give the single call of the Qwen3.5 models at temperature 0. 95 percent intervals.
ModelAll defects caughtFalse alarms
Qwen3.5 9B−- 0.053 [ −- 0.133, 0.024]−- 0.104 [ −- 0.158, −- 0.051]
Qwen3.5 27B+0.027 [ −- 0.026, 0.079]+0.059 [0.008, 0.113]
Haiku 5.5−- 0.040 [ −- 0.084, 0.004]−- 0.005 [ −- 0.024, 0.013]
Appendix
Table A4: The single agent without the instruction minus the team with it, one call of each arm on the 100 paired memos, for the three models below ceiling, with 95 percent intervals.
 Qwen3.5 9BQwen3.5 27BHaiku 5.5
dteam′−dsingle′d^{\prime}_{\mathrm{team}}-d^{\prime}_{\mathrm{single}} , all defects+0.01 [ −- 0.15, 0.17]−- 0.09 [ −- 0.29, 0.12]−- 0.03 [ −- 0.33, 0.23]
dteam′−dsingle′d^{\prime}_{\mathrm{team}}-d^{\prime}_{\mathrm{single}} , bare+0.13 [ −- 0.13, 0.40]not computednot computed
dteam′−dsingle′d^{\prime}_{\mathrm{team}}-d^{\prime}_{\mathrm{single}} , masked−- 0.11 [ −- 0.37, 0.14]+0.04 [ −- 0.23, 0.30]+0.17 [ −- 0.19, 0.51]
dteam′−dsingle′d^{\prime}_{\mathrm{team}}-d^{\prime}_{\mathrm{single}} , arithmetic+0.07 [ −- 0.16, 0.31]−- 0.21 [ −- 0.44, 0.04]−- 0.25 [ −- 0.57, 0.05]
dscoped′−dsingle′d^{\prime}_{\mathrm{scoped}}-d^{\prime}_{\mathrm{single}} , all defects+0.01 [ −- 0.14, 0.16]−- 0.06 [ −- 0.26, 0.14]+0.14 [ −- 0.20, 0.49]
dscoped′−dteam′d^{\prime}_{\mathrm{scoped}}-d^{\prime}_{\mathrm{team}} , all defects0.00 [ −- 0.14, 0.15]+0.02 [ −- 0.14, 0.19]+0.17 [ −- 0.08, 0.45]
Appendix
Table A5: Changes in d′d^{\prime} and in cc under the equal-variance model, with 95 percent intervals, on the three models below ceiling. By class, a common false-alarm rate is used. A class caught in full by either arm is not computed.
More in this topic

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

Explore similar work

CardsList

  1. Multi-agent Scaling Across Disjunctive and Compensatory Tasks

    Sep 25, 2026Carolina Fortuna, Blaz BertalanicMulti-Agent LLM SystemsMulti-Agent Systems

  2. Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness

    Mar 29, 2026Phat Nguyen, Thang PhamMulti-Agent LLMsLLM Agent Evaluation

    Georgia Institute of Technology, Atlanta GA 30332, USA · Adobe Inc., San Jose CA 95110, USA

  3. Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

    Sep 26, 2026Weicheng XueLLM Agent EvaluationAI Agent Auditing

    Virginia Tech