TxnQA: Benchmarking Analytical Reasoning and Evidence Grounding over Transaction Event Sequences

Rabimba Karanjai; Yiming Bian; Yun-Shiuan Chuang; Jenna L. Ballard; Zijie Pan; Shivani Shekhar; Yixi Li; Yirou Ge

cs.AIOct 19, 2026

TxnQA: Benchmarking Analytical Reasoning and Evidence Grounding over Transaction Event Sequences

Authors: Rabimba Karanjai, Yiming Bian, Yun-Shiuan Chuang, Jenna L. Ballard, Zijie Pan, Shivani Shekhar, Yixi Li, Yirou Ge,

Organizations: PayPal AI · University of Pennsylvania

Abstract

Analyzing merchant transaction histories requires dynamic temporal filtering, entity resolution, and aggregation over irregular event streams--operations poorly captured by static tabular or time-series forecasting benchmarks. We introduce TxnQA, an analytical QA benchmark comprising 27 task families (81 templates) alongside a 5-scenario fraud-risk reasoning suite that evaluates both verdicts and supporting evidence over synthetic, privacy-mitigated transaction data. Ground-truth answers are computed programmatically, and model performance is decomposed into format-following rate (FFR), conditional content accuracy (CA), and end-to-end correctness (E2E). Evaluating prompted proprietary baselines and six LoRA-adapted open-weight models shows that adaptation reliably enforces output schemas but leaves arithmetic bottlenecks: Gemma-4-31B-IT and Qwen3-8B attain E2E scores of 0.867 and 0.816, whereas evaluated MoE checkpoints reach only 0.632-0.713. Furthermore, on the 500-instance fraud suite, verdict accuracy frequently exceeds 80%, yet joint accuracy (requiring exact evidence attribution) collapses to at most ~23%. These findings identify schema compliance, semantic calculation, and evidence attribution as three distinct bottlenecks in transaction-sequence QA.

Figures & tables

Figure 1: Overview of the TxnQA benchmark framework. Starting from real merchant transaction data, we generate privacy-mitigated synthetic merchant profiles and irregular transaction sequences using REaLTabFormer. Each transaction sequence is converted into natural-language context and paired with automatically generated question-answer pairs. The resulting benchmark is used to evaluate multiple open-weight large language models under both base (zero-shot) and LoRA supervised fine-tuning settings. Model performance is assessed using three complementary metrics together with fine-grained analyses across question categories, answer formats, and model architectures.

Split	Users	Items	Use
Train	30,000	–	LoRA SFT
Validation	6,500	–	Model selection
Test – General QA	160	162	Main analytical results
Test – Fraud QA	500	500	Fraud-evidence results

Table 1: Dataset populations used in the evaluation.

Proprietary model	Reasoning	Precise QA			Categorical QA
		FFR	CA (strict)	E2E (approx.)	FFR	CA (match)
Claude Sonnet	Thinking-on	97.5 $\pm$ 1.1	85.0 $\pm$ 0.5	82.9 $\pm$ 1.1	99.1 $\pm$ 1.6	74.8 $\pm$ 0.4
Claude Sonnet	Thinking-off	85.1 $\pm$ 1.5	85.1 $\pm$ 2.3	72.4 $\pm$ 2.3	97.2 $\pm$ 0.0	70.5 $\pm$ 1.6
GPT-5.2	High	87.3 $\pm$ 0.5	88.7 $\pm$ 0.6	77.4 $\pm$ 0.7	95.4 $\pm$ 1.6	73.8 $\pm$ 3.0
GPT-5.2	None	89.8 $\pm$ 0.5	76.3 $\pm$ 2.5	68.5 $\pm$ 2.3	88.9 $\pm$ 2.8	70.8 $\pm$ 2.0
Gemini 2.5 Pro	High	91.4 $\pm$ 1.6	81.2 $\pm$ 0.3	74.2 $\pm$ 1.3	91.7 $\pm$ 0.0	63.6 $\pm$ 3.0

Table 2: Zero-shot performance of proprietary models, percentages with SD across 3 decoding seeds. E2E is approximated as

\mu_{FFR}\times\mu_{CA}/100

; SD via delta-method approximation. Bold indicates top value per column.

Fine-tuned checkpoint	Precise QA			Categorical QA		Decline Acc.
	FFR	CA (strict)	E2E (approx.)	FFR	CA (match)
DeepSeek-R1-Distill-7B	100.0 $\pm$ 0.0	79.0 $\pm$ 2.5	79.0 $\pm$ 2.5	100.0 $\pm$ 0.0	91.7 $\pm$ 2.8	100.0
Qwen3-8B	100.0 $\pm$ 0.0	81.6 $\pm$ 2.4	81.6 $\pm$ 2.4	100.0 $\pm$ 0.0	91.7 $\pm$ 2.8	100.0
Qwen3-30B-A3B-Instruct-2507	99.0 $\pm$ 0.0	72.1 $\pm$ 1.0	71.3 $\pm$ 1.0	100.0 $\pm$ 0.0	88.0 $\pm$ 1.6	100.0
Qwen3-30B-A3B-Thinking-2507	100.0 $\pm$ 0.0	63.2 $\pm$ 1.1	63.2 $\pm$ 1.1	100.0 $\pm$ 0.0	76.9 $\pm$ 3.2	100.0
Qwen3-32B	100.0 $\pm$ 0.0	82.5 $\pm$ 3.1	82.5 $\pm$ 3.1	100.0 $\pm$ 0.0	89.8 $\pm$ 1.6	100.0

Table 3: Fine-tuned performance on the General QA precise- and categorical-answer splits. E2E approximated as in Table 2 . † ChatTS evaluated with a single deterministic run.

Protocol	Checkpoint	Base FFR	Tuned FFR	$\Delta$ FFR	Base E2E	Tuned E2E	$\Delta$ E2E
Matched non-thinking	Qwen3-8B	0.981 $\pm$ 0.00	1.000 $\pm$ 0.00	+0.019	0.505 $\pm$ 0.02	0.816 $\pm$ 0.02	+0.311
	Gemma-4-31B-IT	0.956 $\pm$ 0.01	1.000 $\pm$ 0.00	+0.044	0.711 $\pm$ 0.01	0.867 $\pm$ 0.03	+0.156
	Qwen3-32B	0.978 $\pm$ 0.01	1.000 $\pm$ 0.00	+0.022	0.555 $\pm$ 0.03	0.825 $\pm$ 0.03	+0.270
	Qwen3-30B-A3B-Instruct	0.959 $\pm$ 0.01	0.990 $\pm$ 0.00	+0.031	0.581 $\pm$ 0.01	0.713 $\pm$ 0.01	+0.132
Native-thinking	DeepSeek-R1-Distill-7B	0.749 $\pm$ 0.02	1.000 $\pm$ 0.00	+0.251	0.546 $\pm$ 0.03	0.790 $\pm$ 0.03	+0.244
Native-thinking	Qwen3-30B-A3B-Thinking	0.956 $\pm$ 0.02	1.000 $\pm$ 0.00	+0.044	0.855 $\pm$ 0.02	0.632 $\pm$ 0.01	$-$ 0.223

Table 4: Base-to-tuned changes. Native-thinking deltas use a larger generation budget.

Model	Precise QA (%)			Evidence Quality
Model	FFR	Verdict Acc.	Joint Acc.	Rec. (%)	Prec. (%)	Over-citation
GPT-5 (thinking on)	$\mathbf{100.0\pm 0.0}$	$80.9\pm 1.2$	$22.4\pm 0.6$	$58.2\pm 0.9$	$34.0\pm 0.6$	$1.77\pm 0.05$
GPT-5 (thinking off)	$\mathbf{100.0\pm 0.0}$	$81.4\pm 0.4$	$0.9\pm 0.3$	$74.2\pm 1.4$	$34.5\pm 0.4$	$2.38\pm 0.01$
GPT-5 mini (thinking on)	$\mathbf{100.0\pm 0.0}$	$82.5\pm 0.6$	$15.4\pm 0.2$	$66.3\pm 1.2$	$\mathbf{34.7\pm 0.2}$	$1.83\pm 0.04$
GPT-5 mini (thinking off)	$\mathbf{100.0\pm 0.0}$	$\mathbf{87.7\pm 1.0}$	$1.0\pm 0.4$	$\mathbf{85.6\pm 0.2}$	$34.4\pm 0.3$	$2.75\pm 0.01$
Claude Opus 4.6 (thinking on)	$\mathbf{100.0\pm 0.0}$	$71.7\pm 0.3$	$16.5\pm 0.2$	$62.8\pm 0.4$	$28.8\pm 0.3$	$2.28\pm 0.01$

Table 5: FraudQA performance of proprietary models (zero-shot), mean

\pm

SD over 3 seeds. Left: answer-level accuracy over all rows ( Precise QA ). Right: Evidence Quality conditioned on the evidence-incorrect population (

P\neq G

). Bold indicates best per column.

Answering without reasoning
Model	SQL	Direct	Direct, 32k
Qwen3-8B	78.7	50.6	–
Qwen3-32B	76.7	55.6	–
Qwen3-30B-A3B-Instruct	87.6	59.1	–
Answering with reasoning
Qwen3-30B-A3B-Thinking	87.9	77.4	84.9

Table 6: End-to-end accuracy (E2E, %) on the expanded 71-type test set (426 questions). SQL: the model queries a database containing the same records. Direct: protocol with default 8,192-token budget, or 32,000 tokens for reasoning models. Values are means over 3 seeds.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Variant 1	Variant 2	Variant 3
1. merchant category	What category does this merchant operate in?	How would you classify this merchant’s business?	What kind of goods or services does this merchant sell?
2. merchant country	Where is this merchant based?	What country does this merchant operate from?	In which country is this merchant located?
3. busiest day	On which weekday is this merchant most active?	What’s the peak day of the week for transactions?	Which day sees the highest transaction volume?
4. busiest time of day	When during the day is this merchant busiest?	Which time of day sees the most activity?	What part of the day has peak volume?
5. merchant age	How old is this merchant’s account?	How long has this merchant been operating?	When was this merchant first established?
6. transaction count	What’s this merchant’s total transaction count?	How many payments has this merchant processed?	What’s the overall transaction volume?

Appendix

Table 7: 27 analytical question types, 3 wording variants each. † Not answerable from rendered text alone; gold label is unknown .

Type(s)	Regime	Example format instruction
1–4, 20–22	categorical	Category / country / day-code / part-of-day / clock-hour / ISO or prose date
5, 6–9, 13–19, 23, 26–27	precise	Exact count, ratio, coefficient of variation, or interval, per type
10–12, 15	precise	Exact dollar amount, full digits or K/M-suffixed
24, 25	precise	Exact age (years/months/days) or country count

Appendix

Table 8: 43 answer-format variants across the 27 question types (condensed), spanning precise and categorical regimes.

Checkpoint	Params	Type	Epochs	LR	Trainable
DeepSeek-R1-Distill-Qwen-7B	7.62B	Dense	3	$1{\times}10^{-4}$	80.7M
Qwen3-8B	8B	Dense	3	$1{\times}10^{-4}$	87.3M
Qwen3-32B	32B	Dense	2	$7{\times}10^{-5}$	268M
Qwen3-30B-A3B-Instruct	30B/3B	MoE	2	$7{\times}10^{-5}$	1.99B
Qwen3-30B-A3B-Thinking	30B/3B	MoE	2	$7{\times}10^{-5}$	1.99B
Gemma-4-31B-IT	31B	Dense	2	$5{\times}10^{-5}$	244M

Appendix

Table 9: LoRA configurations (rank 32,

\alpha=64

, dropout 0.00–0.05).

Open-weight checkpoint	Thinking	Precise QA			Categorical QA
		FFR	CA (strict)	E2E (approx.)	FFR	CA (match)
Qwen3-8B	Off	98.1	51.5	50.5	88.9	66.7
Qwen3-8B	On	93.7	87.1	81.6	95.4	66.0
Qwen3-32B	Off	97.8	56.8	55.5	99.1	68.2
Qwen3-32B	On	94.0	84.8	79.7	100.0	63.0
Qwen3-30B-A3B-Instruct-2507	Off	95.9	60.6	58.1	100.0	63.0

Appendix

Table 10: Zero-shot open-weight performance. † Reasoning-only checkpoint.

Model	Method	E2E	FFR	CA exact (strict)	CA categorical	Decline acc.
Qwen3-8B	SQL	78.7 $\pm$ 1.1	92.2 $\pm$ 1.3	84.6 $\pm$ 2.2	83.8 $\pm$ 3.1	44.1 $\pm$ 3.4
	SQL, no notes	75.9 $\pm$ 1.9	88.8 $\pm$ 2.0	84.9 $\pm$ 1.8	83.4 $\pm$ 3.4	35.0 $\pm$ 8.4
	Direct	50.6 $\pm$ 0.2	96.9 $\pm$ 0.4	46.4 $\pm$ 0.1	61.2 $\pm$ 1.0	18.1 $\pm$ 1.0
Qwen3-32B	SQL	76.7 $\pm$ 2.0	88.7 $\pm$ 0.3	86.2 $\pm$ 2.6	84.3 $\pm$ 0.6	61.6 $\pm$ 7.6
	Direct	55.6 $\pm$ 0.3	97.6 $\pm$ 0.4	49.4 $\pm$ 0.5	69.9 $\pm$ 1.1	51.4 $\pm$ 5.4
Qwen3-30B-A3B-Instruct	SQL	87.6 $\pm$ 0.4	95.5 $\pm$ 1.1	91.9 $\pm$ 0.8	89.6 $\pm$ 0.6	59.9 $\pm$ 2.0

Appendix

Table 11: SQL baseline and direct answers on the expanded 71-type test set, all metrics (%, mean

\pm

SD over 3 seeds). E2E and FFR are over 367 answerable questions; CA is over format-valid answers; decline accuracy is over 59 questions whose gold is unknown .

Financial Language Model Evaluation

See all 33 papers

How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions
Jun 6, 2026Donghao Huang, Tomas Drietomsky, Benjamin Barrett +1Financial Services Memory-Efficient Fine-Tuning
Research and Development, Mastercard, Arlington, VA, USA · School of Computing and Information Systems, Singapore Management University, Singapore · Economic Intelligence, Mastercard, Prague, Czech Republic
MINT: A Universal Zero-Shot Predictor for Transaction Data
Aug 14, 2026Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan +3Zero-Shot Learning Multimodal Foundation Models
Risk and Security AI Lab, Visa Inc. United Kingdom
Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline
Aug 10, 2026Morris LeeLLM Answer Verification LLM Agent Evaluation
Independent Researcher

Jun 6, 2026J/K move · Enter open · S save

How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions

Donghao Huang, Tomas Drietomsky, Benjamin Barrett, Zhaoxia Wang

Research and Development, Mastercard, Arlington, VA, USA · School of Computing and Information Systems, Singapore Management University, Singapore · Economic Intelligence, Mastercard, Prague, Czech Republic

Financial Services Memory-Efficient Fine-Tuning LLM Fine-Tuning Document Information Extraction LLM Information Extraction Efficient Language Model Inference+2

Open paper AskPDF arXiv

Merchant information extraction turns noisy financial transaction descriptors into structured fields at production scale. Our deployed LoRA-fine-tuned LLaMA~~3.1-8B reaches 96.95% F1, but its memory and throughput motivate smaller replacements. We evaluate 23 retained fine-tuning runs plus a separately trained production reference, spanning Gemma~~3 (270M--4B), Qwen~~3.5 (0.8B--4B), Aya~~3.35B, and LLaMA~~3.1-8B across LoRA ranks, prompts, training templates, and serving environments. A rank-8 LLaMA fine-tune reaches 96.75% F1, only 0.20 points below the rank-32 production reference. Qwen~~3.5~~4B with JSON-Only prompting reaches 96.60% F1 and strict record-level exact match of 91.67%, with a $3.8\times$ lower inverse-throughput time estimate than the rank-8 8B model. Qwen~~3.5~0.8B reaches 94.75% F1, and Qwen Think and Nothink templates differ by less than 0.004 F1. Across 14 Databricks endpoints, mean F1 change from local evaluation is

-0.0081

; Aya is the only family with a 2.7--5.1 point decline. These results show that compact fine-tuned models can preserve most extraction accuracy, but model selection must account for prompt choice, throughput, and serving-stack behavior.

Abstract

Financial Language Model Evaluation

Explore similar work

How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions

MINT: A Universal Zero-Shot Predictor for Transaction Data

Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline