Beyond Keywords: Leveraging Generative LLMs and Label Aggregation to Classify Economic Policy Uncertainty in News Articles

Paul Trust

cs.LGSep 25, 2026

Beyond Keywords: Leveraging Generative LLMs and Label Aggregation to Classify Economic Policy Uncertainty in News Articles

Authors: Paul Trust

Abstract

This research describes the adaptation of Large Language Models (LLMs) for economic monitoring in the public sector to automatically determine whether an article discusses Economic Policy Uncertanity (EPU) and to identify its specific type. Previous studies either rely on keywords, which often result in a high count of false positives, or use machine learning approaches that require a large number of quality human labeled data that is costly and time consuming to acquire. In this study, we propose approaches based on weak supervision techniques, using generative LLMs to create synthetic labels through prompting, making the approach both cost-effective and scalable. Additionally, we propose methods for for multi-label and hierarchical classification of articles related to EPU.

Figures & tables

Category	Relevant Works
Unsupervised ML	Azqueta-Gavaldon et al. (2020) , Xu et al. (2023) , Azqueta-Gavaldón (2017) , Wang et al. (2024) , Miranda-Belmonte et al. (2023) , Kaveh-Yazdy and Zarifzadeh (2021) , Nyman and Ormerod (2020) , Azqueta-Gavaldón et al. (2023) , Jiang et al. (2023) , Baker et al. (2016)
Traditional ML	Lolić et al. (2022) , Tobback et al. (2018) , Sadorsky (2022) , Wang et al. (2022)
Deep Learning & LLM s	Rao et al. (2024) , Chen et al. (2024) , Qureshi et al. (2021) , Keith et al. (2020) , CHANG et al. (2024) , Audrino et al. (2024) , Yeh et al. (2024) , Ito et al. (2025)

Table 1: Relevant studies on automated methods for EPU classification

Panel of Figure 1 — Figure 1: EPU label generation prompt.

Subcategory	Count
Other policy matters that do not fit into pre-defined categories	1176
Political conflict and leadership changes	827
National security and terrorism	820
Trade policy (tariffs, quota, trade rules)	712
Taxes	659
Monetary policy	372

Table 2: Distribution of subcategories for news articles describing EPU

Label Generator	CK	MCC
Majority Vote	-0.0121	-0.0139
WAWA	-0.0011	-0.0013
Mistral3-7B	0.0460	0.0553
LLAMA3-8B	0.2970	0.3058
Qwen2-14B	0.2849	0.3486
Qwen2-7B	0.2849	0.3486

Table 3: Evaluation metrics ( CK and MCC ) for different label generators for the binary EPU classification task

Label Generator	Accuracy	F1	GM
Mistral3-7B	42.06	27.36	41.29
Majority Vote	55.25	46.65	49.44
WAWA	55.41	47.69	49.95
LLAMA3-8B	64.15	42.85	56.09
Qwen2-14B	68.89	61.46	62.85
Qwen2-7B	68.89	61.46	62.85

Table 4: Performance evaluation metrics (F1, accuracy, and GM ) for different label generators for the binary EPU classification task

Dataset	Model	Accuracy	F1	GM	AUC	AUPRC
Human	BERT	75.99	74.66	73.77	0.8423	0.7750
Human	RoBERTa	74.97	73.54	72.59	0.8440	0.7833
Human	LSTM	63.41	60.79	58.96	0.6452	0.5352
WS	DP	71.83	70.78	70.55	0.7338	0.6654
WS	LSTM	63.46	62.10	61.66	0.6482	0.5359
WS	BERT	71.93	71.33	71.83	0.7859	0.6877

Table 5: Performance comparison of different models and datasets on EPU binary classification. Dataset WS indicates that labels generated by DP were used to train the respective model.

Figure 2: Receiver Operating Curves ( ROC ) curves for different models for binary EPU classification task. WS in the case means that labels generated by DP were used to train the respective model.

Model	EMR	LRP	AUPRC	AUC
LSTM	58.15	75.40	0.0466	0.5931
BERT	61.70	83.70	0.1634	0.7727
RoBERTa	60.43	83.78	0.1676	0.7930

Table 6: Performance metrics for different models on the multi-label text classification task

Model	Hierarchical Strategy	Accuracy	F1	AUC	AUPRC	GM
BERT	basic	74.81	74.33	83.86	0.7727	74.54
	masking	74.72	74.30	84.16	0.7724	74.58
	two-stage	74.64	74.33	83.80	0.7641	74.80
	weighted	75.62	74.88	83.93	0.7741	74.90
RoBERTa	basic	76.03	75.48	84.68	0.7895	75.60
	masking	76.19	75.59	84.81	0.7905	75.66

Table 7: Performance evaluation of various models and hierarchical strategies on the binary EPU classification task

Model	Hierarchical Strategy	EMR	AUPRC	AUC	LRP
BERT	basic	59.45	0.0702	0.7080	74.41
	masking	59.45	0.0811	0.7189	74.10
	two-stage	61.94	0.1974	0.7526	81.21
	weighted	59.45	0.0678	0.7046	73.53
RoBERTa	basic	59.45	0.0765	0.7070	74.39
	masking	59.45	0.0963	0.7219	75.22

Table 8: Performance evaluation of various models and hierarchical strategies on the multi-label classification task

Hierarchical Classification

Hierarchical classification uses relationships among labels to predict categories at multiple levels, from broad groups to fine-grained classes.

See all 89 papers

AI Economist Agent: An Agentic Framework for Evidence-Based Economic and Financial Analysis with RAG, Knowledge Graphs, and Large Language Models
Jun 18, 2026Masahiro KatoAgentic RAG LLM Agents
The University of Tokyo
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
Apr 4, 2026Alexander Loth, Martin Kappes, Marc-Oliver PahlAI-Generated Text Detection LLM Evaluation
IMT Atlantique, UMR IRISA, Chaire Cyber CNI Rennes, France
Ideological Bias in LLMs' Economic Causal Reasoning
Apr 23, 2026Donggyu Lee, Hyeok Yun, Jungwon Kim +4Political Bias in Language Models Language Model Bias Evaluation
Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea +1

Jun 18, 2026J/K move · Enter open · S save

AI Economist Agent: An Agentic Framework for Evidence-Based Economic and Financial Analysis with RAG, Knowledge Graphs, and Large Language Models

Masahiro Kato

The University of Tokyo

Agentic RAG LLM Agents Evidence-Grounded Generation Finance

Open paper AskPDF arXiv

We propose an AI economist agent for economic and financial scenario analysis. Scenario design often requires analysts to assess emerging risks with limited historical precedent, combine information from many sources, and translate qualitative mechanisms into internally consistent quantitative paths. Large language models (LLMs) can search and synthesize this information, but fluent narratives alone do not establish the model-based calculations needed for economic conclusions. Our framework uses LLM agents to plan the analysis, retrieve relevant evidence, and organize economic mechanisms, while registered quantitative models generate numerical outcomes and predefined tests determine whether intermediate results can be used in the final report. We apply the framework to European macro-financial stress scenarios and bank capital analysis. The empirical analysis evaluates retrieval of economic mechanisms, scenario construction, model execution, and report generation under a historical information cutoff. The results show how the AI economist agent can combine flexible evidence retrieval and scenario construction while keeping the resulting analysis linked to identifiable sources and explicit model calculations.

Abstract

Hierarchical Classification

Explore similar work

AI Economist Agent: An Agentic Framework for Evidence-Based Economic and Financial Analysis with RAG, Knowledge Graphs, and Large Language Models

Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Ideological Bias in LLMs' Economic Causal Reasoning