PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

Han Bao; Penghao Zhang; Yue Huang; Zhengqing Yuan; Yanchi Ru; Rui Su; Yujun Zhou; Xiangqi Wang

cs.CLApr 14, 2026

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

Authors: Han Bao, Penghao Zhang, Yue Huang, Zhengqing Yuan, Yanchi Ru, Rui Su, Yujun Zhou, Xiangqi Wang,

Organizations: University of Notre Dame · Independent Researcher

Abstract

Large Language Models (LLMs) are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present PolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) Memorization: factual recall of policy knowledge, (2) Understanding: conceptual and contextual reasoning, and (3) Application: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose PolicyMoE, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models.

Figures & tables

Figure 1: Three levels of evaluating LLM in PolicyBench .

Figure 2: Selected examples from PolicyBench spanning three levels and two languages.

Level	ID	Task	Number
			CN	US
Mem.	1-1	Article/Date Memorization	4,498	3,351
	1-2	Terminology Recognition	237	126
	1-3	Organization Identification	268	228
Und.	2-1	Idea Understanding	1,219	1,013
	2-2	Interest Understanding	1,145	996

Table 1: Task list of PolicyBench and respective numbers (“Mem” = Memorization, “Und” = Understanding, “App” = Application).

Level	Region	GPT-4o	o4-mini	Gemini-2.5	Gemini-2.0	Claude-3.7	Claude-3.5	LLaMA-4	Gemma-3-27B	QwQ-32B	Deepseek-V3	Deepseek-R1
Level 1	CN	46.01%	45.93%	54.06%	47.87%	55.29%	53.77%	49.81%	41.75%	55.87%	48.61%	62.02%
Level 1	US	52.69%	54.90%	57.73%	53.71%	58.68%	58.76%	52.55%	49.91%	46.40%	50.12%	59.33%
Level 2	CN	56.34%	55.81%	60.57%	56.39%	60.47%	59.74%	56.56%	55.56%	59.79%	55.51%	62.92%
Level 2	US	63.40%	64.71%	64.91%	62.25%	68.23%	68.95%	61.17%	62.17%	57.71%	58.62%	65.37%
Level 3	CN	70.24%	79.49%	76.18%	73.80%	73.82%	72.83%	68.54%	71.51%	80.34%	72.33%	73.78%
Level 3	US	68.13%	77.00%	69.44%	66.55%	68.28%	68.47%	66.41%	68.37%	69.90%	69.39%	74.60%

Table 2: Performance (accuracy) of all models on different levels and regions. Gemini-2.5, Gemini-2.0, Claude-3.5 and Claude-3.7 denote Gemini-2.5-Flash, Gemini-2.0-Flash, Claude-3.5-Sonnet and Claude-3.7-sonnet respectively. Red and blue represent the highest and lowest scores in each row respectively.

Region	GPT-4o	o4-mini	Gemini-2.5	Gemini-2.0	Claude-3.7	Claude-3.5	LLaMA-4	Gemma-3-27B	QwQ-32B	Deepseek-V3	Deepseek-R1
Chinese	57.53%	60.41%	63.60%	59.35%	63.19%	62.11%	58.30%	56.27%	65.33%	58.82%	62.24%
US	61.41%	65.54%	64.03%	60.84%	65.06%	65.39%	60.04%	60.15%	58.00%	59.38%	66.43%
Average	59.47%	62.98%	63.82%	60.10%	64.13%	63.75%	59.17%	58.21%	61.67%	59.10%	64.34%

Table 3: Average accuracy (%) of all models across Chinese and US (red and blue represent the highest and lowest in each row).

Figure 3: Model performance in 10 subtasks (ID and the specific task are shown in Table 1 ).

Level	Region	Original	Training	$\Delta$
Level 1	CN	36.85%	41.83%	$\uparrow$ 13.51%
Level 1	US	23.35%	35.43%	$\uparrow$ 51.73%
Level 2	CN	45.68%	47.02%	$\uparrow$ 2.93%
Level 2	US	42.31%	42.78%	$\uparrow$ 1.11%
Level 3	CN	64.73%	69.12%	$\uparrow$ 6.78%
Level 3	US	46.65%	57.48%	$\uparrow$ 23.22%

Table 4: Qwen2.5-7B-Instruct performance across levels and regions before and after training (%)

Figure 4: Routing distributions over three experts for each level.

Figure 5: Human accuracy on three levels (%).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Categories of Chinese Policy Documents and Representative Keywords (Partial List).

Figure 7: Screenshot of human evaluation interface.

US Sources
Language & Content Type	Primary Collection Websites / Platforms
Official Policies	https://www.transportation.gov
	https://www.hhs.gov
	https://www.va.gov
	https://www.commerce.gov
	https://www.usda.gov

Appendix

Table 5: Sources for Policies and Supplementary Materials (By Language)

Model	Developer	Open-source	Version	Role
GPT-4o	OpenAI		/	Evaluated model
o4-mini			2025-04-16	Evaluated model; Judge (Level 3)
Gemini-2.5-Flash	Google DeepMind		preview-04-17	Evaluated model; Judge (Level 3)
Gemini-2.0-Flash			/	Evaluated model
Gemma-3-27B			27B Instruct	Evaluated model
Claude-3.7-Sonnet	Anthropic		20250219	Evaluated model; Judge (Level 3)

Appendix

Table 6: Overview of models evaluated or used as judges in this study.

Dimension	ID	Definition
Article/Date Memorization	1-1	Tests memory of specific articles, dates, numbers, etc. in the policy.
Terminology Recognition	1-2	Tests ability to recognize and understand policy terminology.
Organization Identification	1-3	Tests ability to identify organizations mentioned in the policy.
Idea Understanding	2-1	Examines the ideological foundation and value orientation behind the policy.
Interest Understanding	2-2	Assesses the identification and analysis of key stakeholders affected by or involved in the policy.
Institution Understanding	2-3	Evaluates understanding of the formal and informal rules, organizations, and mechanisms shaping policy implementation.

Appendix

Table 7: The detailed introduction of 10 dimensions.

Figure 8: Representative error cases across three cognitive levels in PolicyBench .

Level	Region	Clarity	Relevance	Faithfulness	Avg.
Level 1	CN	99.20%	98.01%	98.78%	98.66%
Level 1	US	99.15%	98.25%	97.94%	98.45%
Level 2	CN	98.86%	98.00%	98.12%	98.32%
Level 2	US	98.76%	97.60%	97.22%	97.86%
Level 3	CN	99.16%	96.02%	96.14%	97.11%
Level 3	US	97.66%	95.64%	96.32%	96.54%

Appendix

Table 8: Human evaluation results across levels and regions.

Annotation Dimension	Kappa ( $\kappa$ )	Agreement Level
Bloom Classification (L1/L2/L3)	0.856	Strong
Clarity (Pass/Fail)	0.841	Strong
Relevance (Pass/Fail)	0.877	Strong
Faithfulness (Pass/Fail)	0.827	Strong

Appendix

Table 9: Inter-Annotator Agreement statistics. Quality scores were binarized for

\kappa

calculation.

Metric	Score
Correctness Verification	96.1%
Content Validity (Rated “Appropriate”)	94.5%
Bloom-Label Agreement (Krippendorff’s $\alpha$ )	0.86

Appendix

Table 10: Expert Validation Results (

N=120

Level	Expert Accuracy	Comparison vs. SOTA LLM
Level 1 (Memorization)	82.3%	Comparable
Level 2 (Understanding)	88.4%	Significant Gap
Level 3 (Application)	90.1%	Significant Gap

Appendix

Table 11: Human Expert Performance (Open-Book) vs. Models.

Model	Baseline	Single-Examiner Setting			LOEO Setting			Avg.
Model	(Consensus)	Claude-Only	GPT-Only	Qwen-Only	Wo-Claude	Wo-GPT	Wo-Qwen	Avg.
Qwen-3	89.0	86.0	88.0	92.0	92.0	90.0	88.0	89.29
Llama-4	82.0	84.0	88.0	89.0	88.0	87.0	85.0	86.14
Qwen-2.5	83.0	81.0	85.0	90.0	83.0	86.0	87.0	85.00
GPT-4o-mini	66.0	59.0	74.0	85.0	74.0	71.0	67.0	82.67
GPT-4o	75.0	71.0	82.0	85.0	85.0	83.0	78.0	79.86

Appendix

Table 12: Complete Multi-Examiner Sensitivity Analysis. Scores represent accuracy (%). "Baseline" denotes the standard PolicyBench setting (3-Examiner Consensus). "Wo-X" denotes the Leave-One-Examiner-Out setting. The data reveals that relying on a single examiner (e.g., GPT-Only or Claude-Only) leads to drastic score fluctuations compared to the stable Baseline.

Model	Baseline Score	Own-Gen Score	$\Delta$	Bias Type
GPT-4o	75.0%	82.0%	+7.0%	Self-Leniency / Pattern Matching
Qwen-3	89.0%	92.0%	+3.0%	Moderate Inflation
Claude-4-Sonnet	84.0%	69.0%	-15.0%	Self-Strictness / Hyper-Critical

Appendix

Table 13: Analysis of Self-Scoring Bias. The results show that single-examiner benchmarks suffer from extreme variance (from +7 inflation to -15 deflation).

Comparison Pair	Spearman $\rho$	Kendall $\tau$	Interpretation
Baseline vs. Wo-Qwen	0.901	0.781	Highly Stable
Baseline vs. Wo-GPT	0.607	0.524	Moderately Stable
Baseline vs. GPT-Only	0.342	0.293	Unstable / Biased

Appendix

Table 14: Leaderboard Stability Analysis. High correlation in LOEO conditions confirms the robustness of the ranking system, while single-examiner rankings (especially GPT-Only) are highly unstable.

Case ID	Final Score (Run 1)	Final Score (Run 2)	Final Score (Run 3)	Mean	Std. Dev.
Case 1	4.00	4.50	4.00	4.17	0.29
Case 2	4.00	4.00	4.25	4.08	0.12
Case 3	3.00	3.25	3.00	3.08	0.12
Case 4	4.33	4.50	4.25	4.36	0.10
Case 5	2.25	2.25	2.50	2.33	0.12
Case 6	5.00	5.00	5.00	5.00	0.00

Appendix

Table 15: Stability analysis of the final LLM-as-a-Judge scores. The low standard deviation across three independent runs for each case demonstrates the reliability of our multi-judge evaluation protocol.

Metric	Value
Pearson Correlation ( $r$ )	0.87
Mean Absolute Error (MAE)	0.42
Agreement Rate	94%

Appendix

Table 16: Human–Model alignment statistics. High correlation, low error, and strong agreement indicate that the LLM judge closely matches expert evaluation.

Model	MMLU-Pro	LegalBench	PolicyBench (Avg)	PolicyBench (L2)
DeepSeek-V3	81.9%	80.1%	59.10%	57.68%
GPT-4o	80.3%	79.8%	59.47%	56.08%
Claude-3.7-Sonnet	80.3%	78.1%	64.13%	58.48%
Gemini-2.5-Flash	77.9%	81.7%	63.82%	64.06%
Pearson Correlation ( $r$ )	-0.69	-0.07	1.00	–

Appendix

Table 17: Performance comparison and correlation analysis across benchmarks.

Model	Level 1	Level 2	Level 3	Average
	(Memorization)	(Understanding)	(Application)
Base (Qwen2.5-7B)	30.10%	44.00%	55.69%	43.26%
Standard LoRA	34.82%	44.51%	59.45%	46.26%
PolicyMoE (Ours)	38.63%	44.90%	63.30%	48.94%
$\Delta$ (MoE vs. LoRA)	+3.81%	+0.39%	+3.85%	+2.68%

Appendix

Table 18: Performance comparison between the base model, Standard LoRA, and PolicyMoE on a controlled training subset. PolicyMoE achieves larger gains on Memorization (L1) and Application (L3) tasks.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
Sep 24, 2026Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan +1LLM Evaluation LLM Information Extraction
Reliable Intelligence Team, VTT Technical Research Centre of Finland Ltd., 02150 Espoo, Finland
Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit
Sep 22, 2026Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere KyamakyaAI-Assisted Decision Making LLM Auditing
Institute for Smart Systems Technologies University of Klagenfurt Klagenfurt, Austria · University of Klagenfurt / Inst. f. Smart Systems Technologies, Austria & Facult´e Polytechnique Universit´e de Kinshasa, DR-Congo
PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay
Mar 25, 2026Rohan Khetan, Ashna KhetanPolitical Bias in Language Models Language Model Bias Evaluation
Northville High School, Northville, USA · Department of Computer Science, Stanford University, Stanford, USA

Sep 24, 2026J/K move · Enter open · S save

From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring

Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan, Arash Hajikhani

Reliable Intelligence Team, VTT Technical Research Centre of Finland Ltd., 02150 Espoo, Finland

LLM Evaluation LLM Information Extraction

Open paper AskPDF arXiv

Science, technology, and innovation policies are crucial for competitiveness, yet their diversity and scale make them difficult to map and monitor consistently. Existing approaches rely heavily on manual survey efforts, which are costly and challenging to scale across countries. Large language models (LLMs) enable new possibilities for extracting and structuring information from long and unstructured policy documents. This paper presents an application of LLMs as "AI respondents" for generating structured survey responses from policy texts. We develop a data extraction pipeline based on long-context in-context learning to map information from public web sources into predefined survey categories, including policy instruments, target groups, and thematic areas. The pipeline integrates a validation step using a secondary LLM to assess relevance and evidence, alongside comparisons with human-provided responses. Using a multi-country dataset, we evaluate the alignment between LLM-generated and human-generated outputs through overlap measures and cross-validation. Results show that LLMs achieve high agreement for structured indicators (84-95%), while differences remain in free-text fields, where models tend to provide more detailed procedural descriptions. These findings highlight the potential of hybrid human-AI workflows for policy monitoring, improving both efficiency and scalability while maintaining the need for human validation and contextual interpretation.

Abstract

LLM Evaluation

Explore similar work

From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring

Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit

PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay