Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data

Kushagra Bhatnagar; Deekshith Marla

cs.AIOct 26, 2026

Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data

Authors: Kushagra Bhatnagar, Deekshith Marla

Abstract

Financial instruction-data pipelines must distinguish content validity from permission to release a particular dataset. Arya Forge combines typed records, current-content checks, and approval coverage bound to a dataset manifest. We evaluate its release boundaries in a financial-data systems case study. An integrity audit covers 100,000 synthetic instruction records; a separate arithmetic implementation agrees with all 15,000 numeric records across twelve formula families. In 240 scenarios with 2,400 treatment rows, adding financial and release-record checks to a short-instruction filtering recipe rejects all injected numeric faults and policy holds while retaining 220 of 240 clean controls. The twenty lost controls comprise every tested cash-conversion-cycle example. Nineteen persisted-release cases show that eleven selected negative cases are blocked, four equivalent representations are rejected, and changes to external evidence after binding are outside ordinary Gold promotion's checks. Local processing measurements and controlled checkpoint resumptions characterize cost and recovery through 100,000 input rows. The results expose a tradeoff between strict producer-contract enforcement and useful-representation retention, and identify the protected inputs required for manifest-bound approval. These findings apply to the tested implementation and workloads; they do not establish general financial correctness, authenticated approval, expert-validated quality, or downstream model improvement.

Figures & tables

Study	Evidence and decision unit	Relation to the present evaluation
in-toto	Signed supply-chain steps and artifact identity.	Establishes authenticated step binding; Forge trusts supplied authority assertions.
Attesting LLM Pipelines	Attested training/release claims at admission.	Gate design and evaluation blueprint; Forge reports executed record and release probes.
Reusing Assurance Evidence	Signed control evidence valid for a candidate and its dependencies.	Studies stale evidence reuse; Forge tests current financial records and approval scope.
Structure Before Intelligence	Validation and release of workflow outputs.	Measures safety, yield, and cost; Forge examines instruction-data representation and approval coverage.
Arya Forge	Typed instruction records and approval bound to a manifest.	Measures selected faults, clean-template losses, and equivalent-representation rejection; no authority authentication.

Table 1: Closest release-control comparisons. Entries describe the cited studies, not exhaustive limits of their software.

Panel of Figure 1 — Figure 1: Dataset stages and the ordinary approval path. Solid arrows denote processing; dashed arrows denote inputs to binding or promotion. Ordinary Gold promotion rechecks current records, parent integrity, and active coverage, but does not reopen external evidence. Supplemental promotion has a different evidence-read boundary (Section 3.2 ). Authority assertions, accepted identities, and evaluation time remain trusted inputs.

Operation	Required evidence and effect
Load Silver	Check stage and manifest-listed artifact hashes; parse typed records and check count.
Bind registry	Check external evidence bytes and decision activity; bind decisions to this manifest and require unique coverage.
Evaluate Gold	Rerun current record checks and quarantine; evaluate manifest pin and unique active approval coverage at the supplied time.
Write Gold	Record approval and parent lineage, embed registry and coverage report, and refuse an existing immutable destination.

Table 2: Checks in the ordinary governed promotion path. Registry binding and Gold promotion are separate operations.

Domain	Records	Additional fresh semantic check
Corporate credit	20,000	None
Numeric reasoning	15,000	Supported native arithmetic and canonical text; supported legacy EMI
Grounded regulation	15,000	None; grounding/citation flags are stored assertions
Financial analysis	15,000	None
SME credit	10,000	None
Trade finance	8,000	None

Table 3: Audited synthetic Gold mixture and shared promotion-gate coverage. All domains receive current schema, privacy-pattern, and unresolved-finding checks. “None” means no additional domain-semantic recomputation in this shared gate; domain-specific generation or qualification checks may have run earlier, and stored validation assertions are still consulted.

Component	Cases	Pre-repair conforming	Repaired conforming
Numeric factory	2	1	2
Promotion boundary	7	3	7
Overlap audit	3	0	3
Near deduplication	3	1	3
Total	15	5	15

Table 4: Conformance on author-selected component diagnostics. Counts are not population accuracy.

Panel of Figure 2 — Figure 2: Retention and acceptance on the 2,400 constructed test rows. Labels give exact accepted/total counts; all recipes accept 0/240 repetition probes. Clean retention is desirable, whereas accepting the injected numeric faults or release-policy holds is undesirable. Treatments are paired within 240 scenarios. Policy holds are eligibility conditions, not necessarily financial errors; four treatments alter metadata unavailable to the text-only baseline. The final arm is a composed record gate, not the full pipeline.

Formula group	Text retained	JSON retained	Total retained
Cash conversion cycle	0/10	0/10	0/20
Each of the other eleven families	10/10	10/10	20/20
All twelve families	110/120	110/120	220/240

Table 5: Clean-control retention by family and representation. The SFT recipe and both augmented arms have identical counts. These are the existing 240 controls, not an additional experiment.

Case category	Accepted	Rejected
Complete source/record approvals; JSON formatting	3	0
Partial, overlapping, expired, wrong-parent approvals	0	4
JSONL/Parquet byte tampering	0	2
Wrong answer with stale flag and fresh approval	0	1
Current or stored sensitive findings	0	2
Missing or changed evidence before binding	0	2

Table 6: Nineteen dependent persisted-release cases. Counts are case tallies; all semantic variants target the DSCR record.

Panel of Figure 3 — Figure 3: Local processing measurements on the authored workload, with three uninterrupted repetitions per stage and size. Left: median stage time and observed minimum–maximum bars (not confidence intervals); small ranges can be hidden by markers. Right: the largest observed peak RSS in each group of three runs, on a linear scale starting at zero. Time excludes interpreter startup/imports; peak RSS includes them. Qualification reads $0.9N$ rows after deduplication. Connecting lines guide the eye; they do not fit or extrapolate a scaling law.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Raw agreement	Cohen’s $\kappa$
Numeric correctness	80.0%	0.573
Evidence support	84.0%	0.698
Jurisdiction/date applicability	91.0%	0.832
Training usefulness	86.5%	0.716

Appendix

Table 7: Descriptive agreement of two same-model AI reviewers over 200 paired records. Uncertain and not-applicable labels remain categories; agreement is not accuracy.

$N$	Stage	Median seconds	Maximum RSS (MiB)	Output (MiB)
1,000	Audit	0.225	44.1	2.0
	Bronze	0.124	48.8	2.5
	Qualification	0.038	41.0	0.5
10,000	Audit	2.403	46.9	19.7
	Bronze	1.235	53.4	24.3
	Qualification	0.353	42.2	4.6

Appendix

Table 8: Three repetitions per cell on the authored laptop workload.

N

is raw input scale; qualification reads

0.9N

canonical rows. Time excludes interpreter startup, while peak RSS includes it.

Algorithmic Auditing

Algorithmic auditing evaluates whether automated systems are fair, reliable, robust, compliant, and accountable in their intended settings.

See all 143 papers

AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA
Jun 18, 2026Aravind Narayanan, Shaina RazaLLM Answer Verification Financial QA
Vector Institute
ERP Data Provisioning Financial Control Testing
Jun 23, 2026Anitha SamudralaSynthetic Tabular Data Generation Synthetic Data Evaluation
AMAZON CORPORATION
Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization
Oct 12, 2026Huihao Jing, Shaojin Chen, Wenbin Hu +5LLM Guardrails Tool-Use Authorization
HKUST

Jun 18, 2026J/K move · Enter open · S save

AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA

Aravind Narayanan, Shaina Raza

Vector Institute

LLM Answer Verification Financial QA Chart QA Multi-Agent Systems Finance

Open paper AskPDF arXiv

Financial chart question answering in regulated settings demands more than accuracy: practitioners must know which answers to trust before acting on them, and many institutions cannot send client data to external model providers. Yet existing chart-QA agents are accuracy-focused and opaque, and most assume proprietary API access; to our knowledge, none combines auditability with on-premise deployability without significant accuracy compromise. We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification, recording every step in a traceable Model Evaluation Packet (MEP) per sample. On FinMME, AgentFinVQA improves

+7.68

pp over a primary-backbone matched zero-shot baseline with a proprietary backbone (Gemini-3 Flash; 71.24% vs. 63.56%, McNemar

p \approx 1.1 \times 10^{-16}

), and

+4.84

pp with open-weights Qwen3.6-27B-FP8 served locally. The verifier's verdict also serves as a useful confidence signal (68.2% vs. 55.6% exact accuracy on confirmed vs. revised answers), enabling human-in-the-loop review routing. Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures and are the categories least detected by the verifier, identifying clear directions for future work. Together these results show that auditable, on-premise financial chart QA is practical and that the open-weights system keeps most of the accuracy gains while enabling full data residency. We release our code to support reproducible evaluation.

Abstract

Algorithmic Auditing

Explore similar work

AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA

ERP Data Provisioning Financial Control Testing

Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization