BudgetVerify: Budget-Tiered Verification for Financial QA

Janet Jenq; Hongda Shen

cs.AISep 27, 2026

BudgetVerify: Budget-Tiered Verification for Financial QA

Authors: Janet Jenq, Hongda Shen

Organizations: PitchBook

Abstract

Financial question answering often requires precise numerical extraction, unit handling, and arithmetic over tables and text, but applying expensive verification uniformly wastes test-time compute. We propose BudgetVerify, a budget-tiered generator-verifier framework that routes each generated answer to one of three verification tiers: no verification, lightweight check-and-revise, or higher-cost solve-first-then-compare verification. The router is trained from offline correctness and token-cost outcomes and, at test time, selects a verification tier using information available before verification, including the question, context statistics, the generated answer, and associated generator metadata. The selected tier either returns the generated answer directly or invokes the corresponding verifier. Across six commercial and open-weight base models, BudgetVerify consistently produces more efficient accuracy-cost Pareto frontiers than fixed verification policies by selectively allocating stronger verification only when it is useful. Although absolute performance varies across models, these efficiency gains and the resulting qualitative frontier shape are consistent across generator models.

Figures & tables

Figure 1: Generator–verifier workflow and FinQA verification conditions. The top panel shows the standard generator–verifier paradigm, where generated answers are checked or revised by a verifier. Green markers denote correct answers, while red markers denote incorrect answers. The bottom panels illustrate three conditions commonly observed in FinQA: direct extraction often needs no verification, simple errors can be fixed by lightweight checking, and harder numerical reasoning may require solve-and-verify computation.

Figure 2: Overview of BudgetVerify. A generator first produces an initial answer for each financial question. A lightweight router then uses pre-verification features from the question and generated answer to choose among three tiers: no verification, a low-cost MID verifier, or a higher-cost HIGH verifier. Green markers denote correct answers and red markers denote incorrect answers. The router allocates verification compute adaptively, aiming to preserve easy correct answers while sending harder or error-prone cases to stronger verification.

Statistics	Value
Total examples	8,281
Train/Validation/Test	6,251/883/1,147
Report pages	2,789
Avg. input text tokens	628.11
Avg. table tokens	59.42
Avg. total input tokens	687.53

Table 1: Basic statistics of the FinQA dataset. FinQA contains financial question-answer pairs with structured tables, unstructured text, executable answers, and annotated reasoning programs.

Figure 3: Oracle outcome decomposition across models on a 1,500-example diagnostic sample from the FinQA training split. The bars show which examples are already correct under the generator, fixed by Mid , fixed only by High , or not fixed by any verification tier. This diagnostic decomposition is used to analyze verifier outcome structure.

Panel of Figure 4 — Figure 4: Within-model accuracy-token-cost tradeoff curves across six base-model settings. Each panel should be interpreted as an independent replication of the same experiment, rather than as a comparison of absolute model performance across panels. The x-axis reports total token cost normalized by the generator-only Low baseline for that model, and the y-axis reports final answer accuracy on the FinQA test split. Colored curves correspond to supervised routing policies obtained by sweeping the cost parameter $\lambda$ . Each colored point represents a separately trained router $\pi_{\lambda}$ , since each $\lambda$ induces a different set of cost-aware oracle labels. Smaller $\lambda$ values favor accuracy and route more examples to Mid or High , while larger $\lambda$ values favor lower-cost Low decisions. Gray markers denote fixed verification baselines: Generator Only , Always Mid , and Always High . The upper envelope of the routed points gives the empirical accuracy–cost frontier.

LLM Answer Verification

Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.

See all 150 papers

Test-Time Scaling via Budgeted Multi-Attribute Verification
Sep 28, 2026Bo Xue, Ji Cheng, Shen-Huan Lyu +2Multi-Armed Bandits LLM Answer Verification
City University of Hong Kong · Hohai University · Zhejiang University
Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning
Jun 18, 2026Sajib Acharjee Dip, Dawei Zhou, Liqing ZhangLLM Inference Efficiency Cost-Aware Inference
Department of Computer Science, Virginia Tech · Fralin Biomedical Research Institute, Virginia Tech · FBRI Cancer Research Center, Washington, DC
Thinking Costs Tokens: When More Structure is Worth the Price
Aug 27, 2026Thomas Nolasque, John Grey, Calista Pham +1Efficient Language Model Reasoning Test-Time Scaling
Royal Bank of Canada

Sep 28, 2026J/K move · Enter open · S save

Test-Time Scaling via Budgeted Multi-Attribute Verification

Bo Xue, Ji Cheng, Shen-Huan Lyu, Yuanyu Wan+1

City University of Hong Kong · Hohai University · Zhejiang University

Multi-Armed Bandits LLM Answer Verification Cost-Aware Inference

Open paper AskPDF arXiv

Verifying LLM-generated answers under a shared computational budget requires jointly deciding which candidates to inspect and which verification attributes to evaluate. We formulate this problem as multi-attribute good-arm identification under a global budget: each candidate is an arm evaluated along several costly attributes, and the goal is to certify as many candidates as possible whose mean scores exceed the prescribed thresholds on all attributes. We propose BMA-GAI, an algorithm that combines cost-aware arm selection with adaptive sampling of attributes. Every observation serves both to guide adaptive allocation and to support anytime-valid certification, which removes the need for a separate confirmation stage. We establish an asymptotic coverage guarantee for BMA-GAI and derive a matching information-theoretic converse that characterizes the intrinsic complexity of the problem, thereby proving that BMA-GAI is first-order optimal away from critical budget levels. Experiments on synthetic benchmarks and an LLM answer-verification task show that BMA-GAI allocates the verification budget more efficiently and certifies more high-quality candidates than competing methods.

Abstract

LLM Answer Verification

Explore similar work

Test-Time Scaling via Budgeted Multi-Attribute Verification

Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning

Thinking Costs Tokens: When More Structure is Worth the Price