STQA: A Benchmark for Stock-Focused Tabular Question Answering over Historical and Forecasted Data
Authors: Baoxu An, Wenmian Yang, Zhensheng Wang, Weijia Jia
Organizations: School of Artificial Intelligence, Beijing Normal University, Beijing, PR China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, PR China · Beijing Normal-Hong Kong Baptist University, Zhuhai, PR China
Stock market analysis inherently requires composite reasoning over historical records and future projections, yet existing benchmarks remain fragmented across isolated tasks. We introduce STQA (Stock-focused Tabular Question Answering), an end-to-end benchmark designed to systematically evaluate natural-language question answering over historical data, numerical forecasts, and forecast-based reasoning. Built on a large-scale financial dataset, STQA covers 4,417 stocks and contains 31,400 question-answer pairs derived from expert-crafted templates, accompanied by fine-grained intent and slot annotations. To operationalize this benchmark, we present SQFRS (Stock Query-Forecast-Reasoning System), an agent-based unified framework that orchestrates SQL retrieval and time-series forecasting tools. Experiments demonstrate that while current large language models perform well on historical queries, forecast-based reasoning poses a substantial challenge, revealing critical bottlenecks in tool coordination and reasoning under uncertainty. The dataset and code are available at https://github.com/xuxubaobaoan/STQA_Project. STQA thus serves as a rigorous testbed for future research on trustworthy, tool-augmented financial agents.
Table 1: Dataset comparison, where “-” indicates data that is currently inaccessible.
Figure 1: The overall framework of SQFRS. Given an input query, the SLU module predicts its intent and extracts slots. In the illustrated example, the Dynamic Routing Agent identifies the query as a forecast-based reasoning problem and selects the corresponding FR workflow to generate the final answer.
QR
NF
FR
WF
Model
Method
F1
Acc
MSE
MAE
MRE
F1
Acc
Acc
Qwen3 30B
SQFRS
0.7000
0.6924
81.3165
2.4583
0.0617
0.5536
0.5529
99.67%
Vanilla
0.6755
0.6406
128.6482
2.8026
0.0744
0.5111
0.4672
-
GPT-oss-20B
SQFRS
0.7186
0.7152
81.3120
2.4561
0.0616
0.6234
0.5929
99.93%
Vanilla
0.6259
0.6174
126.2028
2.7814
0.0742
0.5202
0.4706
-
GPT-oss-120B
SQFRS
0.7309
0.7178
80.5744
2.4140
0.0590
0.6319
0.6171
100%
Table 2: Overall performance comparison between the SQFRS framework and the Vanilla baseline, with boldface indicating the best result achieved by each method. “–” indicates that WF is not applicable.
Model
Method
QR
NF
FR
Acc
MRE
Acc
Qwen3 30B
SQFRS
0.6924
0.0617
0.5529
W/O SLU
0.6486
0.0704
0.5029
GPT-oss-20B
SQFRS
0.7152
0.0616
0.5929
W/O SLU
0.6705
0.0704
0.5509
GPT-oss-120B
SQFRS
0.7178
0.0590
0.6171
Table 3: Ablation Study Results for the SLU Module.
Figure 2: The impact of different GT labels.
QR
NF
FR
Model
Method
F1
Acc
MAE
MRE
MSE
F1
Acc
Qwen3 30B
Template
0.7000
0.6924
2.4583
0.0617
81.3165
0.5536
0.5529
Human
0.5303
0.5000
2.7960
0.0743
127.5079
0.4857
0.4750
GPT-oss-20B
Template
0.7186
0.7152
2.4561
0.0616
81.3120
0.6234
0.5929
Human
0.5962
0.5333
2.7714
0.0731
125.5648
0.4970
0.4667
GPT-oss-120B
Template
0.7309
0.7178
2.4140
0.0590
80.5744
0.6319
0.6171
Table 4: Result of Real-world Questions
Model
Setting
FR Acc (%)
Human
Stockholder
69.17
Human
Non-stockholder
63.33
Qwen3 30B
SQFRS
59.17
Qwen3 30B
DR
54.17
GPT-oss-20B
SQFRS
62.50
GPT-oss-20B
DR
50.88
Table 5: FR Accuracy of Humans and Models under SQFRS vs DR.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
History SQL
Model
ECR
Pass@1
Qwen3 30B
100%
99.05%
GPT-oss-20B
100%
99.25%
GPT-oss-120B
100%
99.60%
GEMMA 4-31B
100%
99.70%
Appendix
Table 6: History SQL ECR and Pass@1.
Model
Setting
QR
NF
FR
P
R
F1
Acc
MSE
MAE
MRE
P
R
F1
Acc
Qwen3 30B
SQFRS
0.7006
0.7003
0.7000
0.6924
81.3165
2.4583
0.0617
0.5539
0.5532
0.5536
0.5529
W/O SLU
0.6571
0.6523
0.6529
0.6486
116.9249
2.7243
0.0704
0.5123
0.5166
0.5145
0.5029
GPT-oss-20B
SQFRS
0.7236
0.7188
0.7186
0.7152
81.3120
2.4561
0.0616
0.5959
0.6535
0.6234
0.5929
W/O SLU
0.6771
0.6730
0.6734
0.6705
116.8996
2.7237
0.0704
0.5531
0.5800
0.5762
0.5509
GPT-oss-120B
SQFRS
0.7314
0.7316
0.7309
0.7178
80.5744
2.4140
0.0590
0.6160
0.6596
0.6319
0.6171
Appendix
Table 7: Experimental Results of the SLU Module.
Model
Method
QR
NF
FR
Acc
MRE
Acc
Qwen3 30B
SQFRS
0.6924
0.0617
0.5529
LLM-SLU
0.6810
0.0685
0.5400
GPT-oss-20B
SQFRS
0.7152
0.0616
0.5929
LLM-SLU
0.7029
0.0678
0.5886
GPT-oss-120B
SQFRS
0.7178
0.0590
0.6171
Appendix
Table 8: Robustness analysis via in - context learning.
Model
Threshold
Extreme Value
Trend Comparison
Qwen3-30B
76.0%
44.5%
54.8%
GPT-oss-20B
66.0%
56.0%
55.2%
GPT-oss-120B
80.0%
76.0%
55.6%
GEMMA 4-31B
87.0%
82.5%
52.8%
Appendix
Table 9: Fine-grained results on different types of forecast-based reasoning questions.
Price Fields
Forecast-15days
TimeXer
I-Transformer
TimeMixer
TimesNet
GPT-oss-120B
Time-MoE
open
MSE
252.2331
474.3625
147.4776
202.2346
202.2304
145.3490
MAE
4.1762
5.5079
3.4352
3.8685
4.0777
3.3174
MRE
0.0868
0.1154
0.0727
0.0810
0.0876
0.0686
close
MSE
216.6427
289.5159
364.7470
673.4907
258.6725
202.9997
MAE
4.0625
4.7266
5.0233
7.0094
4.4143
3.7858
MRE
0.0863
0.0998
0.1050
0.1464
0.0929
0.0799
Appendix
Table 10: Experimental Results of the Time Series Forecasting Model Based on Raw Data.
Model
Method
FR
NF
P
R
F1
Acc
MSE
MAE
MRE
Qwen3 30B
TimeXer
0.4768
0.4839
0.4803
0.4743
178.0094
2.8938
0.0855
I-Transformer
0.5536
0.5529
0.5515
0.5523
425.2180
4.3682
0.1049
TimeMixer
0.4847
0.4901
0.4874
0.4843
161.6847
2.8681
0.0819
TimesNet
0.5288
0.5334
0.5311
0.5219
114.4559
2.6444
0.0698
TimesMoE
0.5539
0.5532
0.5536
0.5529
81.3165
2.4583
0.0617
Appendix
Table 11: Comparative Experiment of Time Series Forecasting Models and Large Language Model.
Label
Statistics num
Number of templates
72
Query template count
42
Forecast template count
30
Number of QA pairs
31400
Single stock QA quantity
28000
Forecast QA quantity
20900
Appendix
Table 12: Dataset Metrics Statistics Table.
QR
NF
FR
Model
P
R
F1
Acc
MSE
MAE
MRE
P
R
F1
Acc
Qwen3 30B
0.7105
0.7091
0.7085
0.6990
81.312530
2.4576
0.061611
0.5676
0.5662
0.5683
0.5586
GPT-oss-20B
0.7317
0.7300
0.7298
0.7286
81.226003
2.4528
0.061578
0.6255
0.6579
0.6414
0.6100
GPT-oss-120B
0.7407
0.7409
0.7402
0.7381
80.348970
2.4127
0.058952
0.6472
0.6619
0.6607
0.6234
GEMMA 4-31B
0.7412
0.7442
0.7409
0.7392
81.404654
2.4823
0.062075
0.6821
0.6916
0.6816
0.6357
Appendix
Table 13: Ground Truth SLU.
QR
NF
FR
Model
P
R
F1
Acc
MSE
MAE
MRE
P
R
F1
Acc
Qwen3 30B
0.7168
0.7190
0.7177
0.7086
81.0282
2.4485
0.061560
0.5760
0.5659
0.5709
0.5597
GPT-oss-20B
0.7357
0.7309
0.7315
0.7296
80.9126
2.4452
0.061472
0.6430
0.6607
0.6557
0.6229
GPT-oss-120B
0.7409
0.7423
0.7416
0.7400
80.2875
2.4093
0.058217
0.6483
0.6632
0.6652
0.6359
GEMMA 4-31B
0.7612
0.7534
0.7533
0.7416
81.3612
2.4652
0.061848
0.6840
0.6932
0.6875
0.6374
Appendix
Table 14: Ground Truth Workflow.
Model
MSE
MAE
MRE
P
R
F1
Acc
Qwen3 30B
80.6790
2.4282
0.060553
0.6204
0.6224
0.6210
0.6029
GPT-oss-20B
80.8355
2.4346
0.060584
0.6741
0.6882
0.6712
0.6386
GPT-oss-120B
79.8980
2.4029
0.057562
0.6823
0.7043
0.6879
0.6542
GEMMA 4-31B
80.6177
2.4192
0.060550
0.6961
0.7080
0.7020
0.6629
Appendix
Table 15: Ground Truth History SQL.
Model
P
R
F1
Acc
MRE
Qwen3 30B
0.8593
0.8614
0.8603
0.8371
0.004563
GPT-oss-20B
0.8039
0.8515
0.8108
0.7971
0.004689
GPT-oss-120B
0.8186
0.8690
0.8335
0.8129
0.004153
GEMMA 4-31B
0.8783
0.8936
0.8859
0.8614
0.004931
Appendix
Table 16: Ground Truth Forecast Data.
Figure 3: The impact of different GT labels on final result accuracy and MRE (GPT-oss-20B).
Figure 4: The impact of different GT labels on final result accuracy and MRE (GPT-oss-120B).
Figure 5: The impact of different GT labels on final result accuracy and MRE (Qwen3-30B).
Figure 6: LLM Rewrite Effect: Before vs After.
Figure 7: Artificial Rewriting Effect: Before vs After.
Financial markets are characterized by extreme non-stationarity, low signal-to-noise ratios, and strong dependence on external information such as news, company fundamentals, and macroeconomic signals. Yet, existing approaches either abstract time-series into text or decouple forecasting from language-based reasoning, leading to a fundamental mismatch between qualitative reasoning and quantitative outcomes. To address this, we introduce StockR1, a time-series-enhanced LLM that unifies stock forecasting and financial reasoning through a verifiable forecast action. Based on a tool-call design, the model first emits a forecast action, which is a structured and interpretable representation of its qualitative market outlook. It then invokes a time-series decoder conditioned on this action to generate distributional future trajectories, leading to more informed question answering and financial reasoning. We optimize the full pipeline with reinforcement learning, where rewards jointly reflect answer validity, forecast accuracy, and consistency between generated actions and observed time-series dynamics. In addition, rewards are reweighted by a sample-level uncertainty scalar, encouraging the model to accommodate varying uncertainty in market dynamics. We evaluate StockR1 on financial question answering and stock forecasting over a large-scale 10-year benchmark. Our method consistently outperforms time-series baselines and general-purpose LLMs, improving reasoning accuracy by 17.7% (4B) and 25.9% (8B). These findings demonstrate that structuring the forecast actions establishes a powerful synergy between language reasoning and temporal prediction, enabling LLMs to reason through verifiable, interpretable, and numerically grounded decisions.
Jialin Chen, Aosong Feng, Harshit Verma +7
Yale University · Arizona State University · University of Texas Rio Grande Valley
Financial analysis requires reasoning over heterogeneous signals, including company fundamentals from regulatory filings and trading signals derived from market prices. However, existing financial benchmarks focus almost exclusively on regulatory filings, neglecting market dynamics and cross-signal interactions. We introduce FinTradeBench, a benchmark of 1,400 questions covering NASDAQ-100 stocks over a ten-year window. To holistically evaluate financial reasoning, FinTradeBench includes questions in three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions that require cross-signal reasoning. Our benchmark starts with 150 expert-authored seed questions, which are then scaled to 1400. To ensure reliable evaluation, we create a Core400 subset in which every item is expert-reviewed. In zero-shot (No-RAG) response generation, we observe that all models perform substantially worse on trading-focused questions compared to fundamental-focused questions. To identify failure causes, we evaluate LLMs under controlled settings including retrieval-based baselines, tool-assisted computation, oracle conditions, and time-series-specialized pipelines. LLMs' capabilities to do computation and evidence selection contribute most to reasoning failures. Additionally, retrieval significantly improves trading-signal reasoning but does not improve fundamental reasoning, where identifying relevant signals remains the dominant failure mode. Finally, rather than relying on a small, fine-tuned model for time-series reasoning, it is better to summarize the time-series data and leverage LLMs' reasoning capabilities.
Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan +2
Financial forecasting has traditionally relied on numerical time-series (TS) signals, while textual information such as news remains largely underutilized. We introduce XForecast, a benchmark and competition for news-grounded financial forecasting, where the task is to predict stock price movements four weeks ahead using both historical price data and paired news signals. XForecast combines stock price data with curated financial text generated using FinTexTS, a multimodal dataset generation method. It covers 100 large-cap stocks from 2019--2023, with training on 2019--2022 and evaluation on 2023. To better evaluate models that leverage textual signals, we adopt a weighted hit rate metric that places greater emphasis on instances where news information is expected to have stronger influence. The XForecast Challenge, hosted on Kaggle as part of KDD 2026 Finance Day, aims to benchmark methods for integrating financial TS and textual information.
Jaehoon Lee, Jun Seo, Seunghan Lee +10
LG AI Research, Seoul, South Korea · Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea