Dual-Axis Policy Entailment: A Benchmark for Policy-Change Reasoning

Charles Chang; Yuen Yuen Ang; Bolun Sun

cs.CLOct 31, 2026

Dual-Axis Policy Entailment: A Benchmark for Policy-Change Reasoning

Authors: Charles Chang, Yuen Yuen Ang, Bolun Sun

Organizations: Duke Kunshan University · Johns Hopkins University · Northwestern University

Abstract

Whether a policy shifts domains and, by extension, continues, broadens, narrows, or reverses previous policy is of paramount importance to the study of public policy. We propose Dual-Axis Policy Entailment (DAPE), which evaluates an ordered pair of policy directives on both dimensions at once: a binary judgment determines whether the two directives govern the same policy domain, and a five-way judgment determines whether the later directive continues, broadens, narrows, or reverses the earlier policy, or governs a different domain altogether. Read together, the two judgments form one five-category label, our primary metric. The two-axis design captures two central dimensions of policy change without conflating errors on either dimension into a generic "irrelevant" label, and it makes annotation more rigorous: a coder cannot dismiss a difficult pair as domain-irrelevant without an explicit, checkable decision. We introduce an expert-annotated benchmark of 1,000 ordered directive pairs drawn from a large corpus of Chinese government policy, together with an annotation-calibration protocol (an adjudication packet with pre/post agreement measurement) that we contribute as a methodological artifact. Fourteen open-weight models and four frontier API flagships fail the task zero-shot: even the best frontier flagship's chance-corrected agreement with expert labels is far below the human ceiling (Cohen's $κ\approx 0.29$ vs. $0.53$ ). A fine-tuned open-weight model (LoRA SFT of Qwen3-14B, five-fold cross-validation) closes this gap, reaching the coder-mutual human ceiling ( $κ= 0.524$ vs. $0.527$ ). Finally, we apply our method to China's AI policy (1980-2024) and map the direction of policy change across eras.

Figures & tables

Figure 1: Policy change decomposes into movement along two axes: a policy-domain axis (horizontal) and a policy-direction axis (vertical). (a) China’s Zero-COVID policy, established in 2020 as a restriction in the domain of epidemic control ( National Health Commission of the People’s Republic of China, 2020 ) and lifted in 2022 in the vocabulary of economic growth and social stability ( State Council Joint Prevention and Control Mechanism for COVID-19, Comprehensive Group, 2022 ) . (b) U.S. land-conservation measures, adopted as climate-change mitigation ( Executive Office of the President, 2021 ) and rolled back in the name of energy security ( Executive Office of the President, 2025 ) . In both cases the reversal is a diagonal move: the later directive shifts domain framing and action direction at once. A single-axis model cannot say which axis moved; DAPE scores them separately.

Label	What $B$ does to the policy in $A$	$n$
Consistency	Continues the same policy: renewal or parallel reissue, no material change.	255
Expansion	Broadens coverage, reach, resources, obligations, or stringency.	171
Restriction	Narrows coverage, relaxes obligations, lowers stringency.	176
Reversal	Rescinds or materially reverses the earlier policy.	127
Irrelevant	A different policy domain; formulaic or topical overlap alone does not count.	271

Table 1: The five policy-change labels and their counts in the 1,000-pair benchmark. The five rows form the policy-direction change axis; the binary policy-domain matching axis’s single mismatch outcome is definitionally the irrelevant row (

d=0\Leftrightarrow

unrelated ), so both axes are represented. Prose terms are used throughout; the stored labels are consistency , expansion , restriction , contradiction (reversal), and unrelated (irrelevant).

Model	Tier	Cov%	Acc	Macro-F1	Cohen’s $\kappa$
qwen3-32b	open	100	0.480	0.452	0.337
gemini-2.5-flash	API	100	0.452	0.430	0.297
gpt-5-mini	API	100	0.425	0.383	0.279
qwen2.5-14b-instruct	open	100	0.412	0.357	0.266
qwen3-8b	open	100	0.385	0.353	0.214
llama-3.3-70b	API	100	0.403	0.348	0.230

Table 2: Five-category policy-change classification on the 1,000-pair gold benchmark: open-weight and API models under the main prompt, with formulation variants and baselines. Coverage is the share of pairs with a parseable prediction; rows below full coverage are scored on parsed pairs and are not directly comparable. Macro-F1 and Cohen’s

\kappa

are the primary metrics. Script-generated; do not hand-edit.

slug	n_scored	coverage	macroF1	kappa	acc	F1_irrelevant	F1_consistency	F1_restriction	F1_expansion	F1_reversal
claude_sonnet_45__direct5__t1500	1000	0.999	0.412	0.265	0.432	0.511	0.411	0.217	0.446	0.474
deepseek_chat__direct5__t1500	1000	1.000	0.406	0.282	0.440	0.527	0.400	0.124	0.477	0.503
gemini_25_pro__direct5__t1500	1000	0.998	0.418	0.273	0.436	0.508	0.382	0.195	0.465	0.542
gpt_5__direct5__t1500	1000	1.000	0.420	0.256	0.417	0.495	0.200	0.357	0.433	0.618

Table 3: Frontier API flagships under the direct five-category prompt (1,000 gold pairs, canonical 1,500-character window). Few-shot, multi-seed, and context-length variants are in Appendix K .

Model	$n$	macro-F1	$\kappa$	acc.	F1 irrelevant	F1 consistency	F1 restriction	F1 expansion	F1 reversal
SFT Qwen3-14B LoRA	1000	0.634	0.524	0.626	0.586	0.710	0.544	0.551	0.780
DAPT + SFT	1000	0.621	0.509	0.614	0.561	0.698	0.548	0.513	0.786
Gold + model-silver SFT	1000	0.638	0.529	0.631	0.591	0.709	0.550	0.552	0.790

Table 4: Five-fold cross-validated fine-tuning results on the 1,000-pair expert gold set.

Model	$n$	macro-F1	$\kappa$	acc.	F1 irrelevant	F1 consistency	F1 restriction	F1 expansion	F1 reversal
SFT Qwen3-14B LoRA (five-way)	1000	0.634	0.524	0.626	0.586	0.710	0.544	0.551	0.780
SFT Qwen3-14B LoRA (dual-axis, collapsed)	1000	0.649	0.548	0.643	0.600	0.724	0.594	0.551	0.776

Table 5: Five-fold cross-validated fine-tuning with the collapsed five-way target versus the dual-axis (two-output) target, scored on the collapsed relation. Script-generated; do not hand-edit.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

System	$n$	m-F1	m-F1 obs.	$\kappa$	binary $\kappa$
Qwen3-32B zero-shot	300	0.403	0.503	0.411	0.357
GPT-5 zero-shot	300	0.329	0.411	0.240	0.307
Five-way SFT	300	0.474	0.593	0.581	0.665
Five-way SFT jurcorr	300	0.425	0.532	0.479	0.520
Dual-axis SFT	300	0.542	0.677	0.638	0.702
Dual-axis SFT jurcorr	300	0.481	0.602	0.484	0.539

Appendix

Table 6: Existing-model results on the initial 300-pair, less-enriched frame. m-F1 obs. averages over classes present in the frame’s gold labels; binary

\kappa

measures related versus irrelevant. This stratified frame is not a uniform natural sample. SFT predictions are out-of-fold.

Metric	BGE-large-zh	Qwen3-Embedding-8B
Gold-partner recall, top 10	87.3%	89.9%
Gold-partner recall, top 50	93.7%	95.2%
Median gold-partner rank	2.0	2.0
Cosine Spearman, BGE versus Qwen	0.823
Similarity-band agreement	40.4%

Appendix

Table 7: Alternate-embedder diagnostics on the existing 1,000 gold pairs. Retrieval ranks are computed within the 1,961 benchmark source documents, not the full corpus. On 520 comparable non-sentinel PKUlaw pairs, original and recomputed BGE cosines have Spearman 1.000 and median absolute difference 0.0000.

Embedder	Band	$n$	Qwen3-32B	Five-way SFT	Dual-axis SFT
BGE	high	443	0.465	0.629	0.639
BGE	moderate	378	0.358	0.553	0.570
BGE	low	178	0.420	0.603	0.677
BGE	below	1	0.200	0.200	0.200
Qwen	high	144	0.441	0.585	0.577
Qwen	moderate	407	0.449	0.586	0.591

Appendix

Table 8: Macro-F1 by embedding-similarity band on the same human-labeled gold pairs. Bands are diagnostics, not new train/test splits. The BGE below-band contains one pair and is not interpretable as a performance estimate.

Frame	$n$	Irrelevant	Consistency	Directional	Mean conf.
Uniform natural	1000	99.8%	0.2%	0.0%	1.000
BGE retrieval	1200	66.4%	30.6%	3.0%	0.996
Qwen retrieval	1200	47.4%	50.4%	2.2%	0.996

Appendix

Table 9: Predicted composition of the natural and matched retrieval frames. Directional pairs are expansion, restriction, or reversal. These are model predictions for sampling-sensitivity analysis, not human-validated accuracy. The BGE and Qwen frames overlap on 16 pairs; their cosines have Spearman correlation 0.873 across the alternate-frame union.

Evidence	Pair selection	$n$	Related	Directional
Human labels	Different agencies	27	18.5%	0.0%
Human labels	Same agency	264	70.1%	14.8%
Human labels	Other agency/year cells	72	48.6%	9.7%
Human labels	Same agency, 1–5 years	228	70.6%	14.9%
Model composition	Uniform natural, unrestricted	1000	0.2%	0.0%
Model composition	BGE retrieval, same agency, 1–5 years	600	41.7%	1.7%

Appendix

Table 10: Agency/year constraints increase the yield of related candidate pairs. The top panel uses existing human labels from the stratified early frame and is not a corpus-prevalence estimate; nine agency-missing rows are excluded from the same/different-agency comparison. The bottom panel reports model-predicted composition only, not human-validated accuracy.

Label	Definition and worked example
CONSISTENCY	Policy $B$ carries forward $A$ with no material change in scope or stringency, including renewals, transmissions, and parallel reports. Example. $A$ (2012): “Xuzhou Human Resources and Social Security Bureau, notice publishing the 2012 work-injury insurance data and the periodic benefit-adjustment measures”; $B$ (2016): the same notice issued for 2016. The instrument, coverage, and obligations are unchanged; only the reference year moves.
EXPANSION	$B$ broadens $A$ : more coverage, more obligations, or more stringent requirements. Example. $A$ (2003): “Ministry of Water Resources, opinions on strengthening the management of village and township water-supply projects”; $B$ (2004): “Ministry of Water Resources, notice issuing the Staffing Standards for Village and Township Water-Supply Stations ”. $B$ adds concrete staffing obligations to the program $A$ established.
RESTRICTION	$B$ narrows $A$ : removes coverage, relaxes obligations, or reduces stringency. Example. $A$ (2007): “Guangdong Price Bureau and provincial development-reform and economic-trade commissions, notice on further implementing the differential electricity-pricing policy for the steel industry”; $B$ (2011): the same agencies’ “reply approving the cessation of differential electricity pricing for Wenrong Steel Mill in Zijin County, Heyuan City, and other enterprises”. The policy stands, but named enterprises are removed from its scope.
CONTRADICTION	$B$ rescinds or reverses material provisions of $A$ , often via an explicit fèizhǐ (‘repeal’) marker citing $A$ ’s wenhao or policy-document ID. Example. $A$ (2020): “Lüliang Municipal People’s Government, opinions on accelerating enterprise listing and share-quotation work”; $B$ (2025): “Lüliang Municipal People’s Government, notice repealing certain documents”, which lists $A$ among the repealed items.
UNRELATED	$A$ and $B$ are not substantively related; topical-similarity-only matches fall here. Example. $A$ (1993): “Jilin City interim measures for the administration of urban piped gas”; $B$ (2013): “Benxi City measures for gas administration”. The vocabulary and policy area are similar, but two municipalities regulate their own gas systems, so there is no shared policy domain.

Appendix

Table 11: The five-class relation label set. Examples are actual benchmark pairs, with titles translated from Chinese. “Reversal” in the main text is the prose description of the stored contradiction label.

Comparison	Five-way		Binary relatedness
	acc.	$\kappa$	acc.	$\kappa$
Coder C vs gold	0.167	0.038	0.500	0.000
Coder D vs gold	0.133	0.008	0.533	0.000
Coder C vs Coder D (mutual)	0.667	0.519	0.833	0.667

Appendix

Table 12: Post-calibration blind test (

n=30

, boundary-loaded by design).

System	$n$	m-F1	m-F1 obs.	$\kappa$	binary $\kappa$
Qwen3-32B zero-shot	117	0.366	0.366	0.109	0.056
GPT-5 zero-shot	117	0.367	0.367	0.255	0.429
Five-way SFT	117	0.274	0.274	0.113	0.295
Five-way SFT jurcorr	117	0.296	0.296	0.168	0.362
Dual-axis SFT	117	0.251	0.251	0.074	0.213
Dual-axis SFT jurcorr	117	0.282	0.282	0.142	0.339

Appendix

Table 13: Existing-model agreement with the 117 nonblank strict-rule labels in the 120-pair recode. This single-coder, boundary-loaded reference is not independently adjudicated gold and not a natural sample. SFT predictions are out-of-fold.

System	Original gold		Corrected gold
	macro-F1	$\kappa$	macro-F1	$\kappa$
Qwen3-32B (zero-shot)	0.452	0.337	0.433	0.305
GPT-5 (zero-shot)	0.421	0.256	0.423	0.261
Gemini 2.5 Pro (zero-shot)	0.419	0.274	0.435	0.303
Claude Sonnet 4.5 (zero-shot)	0.412	0.265	0.416	0.271
DeepSeek-V3 (zero-shot)	0.406	0.282	0.412	0.293

Appendix

Table 14: Systems re-scored on original versus jurisdiction-corrected gold (

n=1{,}000

), without retraining.

Model	$n$	macro-F1	$\kappa$	acc.	F1 irrelevant	F1 consistency	F1 restriction	F1 expansion	F1 reversal
Five-way SFT, original-gold trained	1000	0.626	0.505	0.613	0.581	0.670	0.534	0.552	0.793
Five-way SFT, retrained on corrected gold	1000	0.634	0.529	0.637	0.675	0.690	0.488	0.527	0.790
Dual-axis SFT, original-gold trained	1000	0.635	0.524	0.624	0.590	0.685	0.565	0.546	0.789
Dual-axis SFT, retrained on corrected gold	1000	0.643	0.540	0.642	0.684	0.659	0.531	0.557	0.782

Appendix

Table 15: Five-fold results on the jurisdiction-corrected gold set (

n=1{,}000

). Original-gold-trained models are re-scored without retraining; retrained models use identical folds and hyperparameters.

frame	scope	documents	paragraphs	years
Published corpus (CAPC-CG)	central only	337,038	3,275,477	1949–2023
Extended central snapshot	central only	368,679	4,101,064	1949–2024
Issuer-matched universe	central + party organs	456,673	3.16M (corpus seg.)	within 1921–2026
Integrated multi-level corpus (ongoing)	all administrative levels	4,997,123	32.46M uids + 3.54M govweb (segs. differ)	1921–2026

Appendix

Table 16: The CAPC database and its ongoing expansion. Paragraph counts use each frame’s own segmentation and are not cross-comparable; compare documents.

component	status	documents	years	levels
Commercial policy database (backbone)	core	4,842,217	1921–2026	national–county
Government-website scrape	core	159,135	1993–2026	national–county (pref.-heavy)
gov.cn policy register (full pull)	register	120,355	1996–2026	—
gov.cn policy register (policy-labelled slice)	register	39,091	1996–2026	—
gov.cn State Council gazette register	register	15,141	2000–2026	—
National Laws and Regulations Database register	register	24,280	2000–2025	—

Appendix

Table 17: Components of the ongoing collection effort.

corpus	national	ministerial	provincial	prefectural	county	unknown	central (%)
Extended central snapshot	13,696	353,368	0	0	0	1,615	100.0
Commercial database backbone (labelled)	19,010	1,069,449	2,066,418	1,239,252	88,253	296,592	22.8
Government-website scrape	40	1,134	28,453	126,945	2,563	0	0.7
Integrated corpus (gross)	19,050	1,070,583	2,094,871	1,366,197	90,816	296,592	22.1

Appendix

Table 18: The central-only extended snapshot versus the all-levels integrated corpus (documents by administrative level). The government-website scrape is shown separately because its exact-title overlap with the backbone is subtracted only in the deduplicated total.

Figure 2: Administrative-level composition of the extended central snapshot, the commercial database backbone, the government-website corpus, and the integrated multi-level corpus.

Channel	$n$	Unrel.	Consist.	Expans.	Restr.	Reversal	Direct.
Same-agency neighbours	14,929	74.4	21.9	3.4	0.3	0.0	3.7
Reissue chains	2,326	1.6	78.0	13.8	6.6	0.0	20.4
Repeal-cue pairs	2,745	64.4	19.0	7.9	3.5	5.2	16.6
All pairs	20,000	64.5	28.0	5.2	1.5	0.7	7.4

Appendix

Table 19: Predicted five-way change categories for document pairs drawn from newly collected Chinese government-website documents (2015–2026), by pairing channel. Directional = expansion + restriction + reversal. Labels are the majority vote of five fine-tuned classifier folds applied to the first 1,500 characters of each document.

Era (by later doc)	Consistency	Expansion	Restriction	Reversal	Irrelevant	$n$
Deng	–	–	–	–	–	0
Jiang	7.7% (2)	0.0% (0)	0.0% (0)	0.0% (0)	92.3% (24)	26
Hu	17.9% (128)	5.0% (36)	1.4% (10)	0.0% (0)	75.7% (542)	716
Xi	4.8% (3109)	1.8% (1147)	0.1% (86)	0.0% (18)	93.3% (60951)	65311

Appendix

Table 20: Predicted five-way change categories for AI-policy pairs by era.

Figure 3: Predicted five-way change categories for AI-policy pairs by era and, for the Xi era, by year.

Frame	era	$n$	unrelated	consistency	expansion	restriction	reversal
Central-government subset	Jiang	1	100.0%	0.0%	0.0%	0.0%	0.0%
Central-government subset	Hu	197	67.0%	22.8%	7.6%	2.5%	0.0%
Central-government subset	Xi	3477	85.3%	6.9%	7.6%	0.1%	0.1%
Central-government subset	All	3675	84.3%	7.7%	7.6%	0.2%	0.1%
Provincial expansion	Jiang	25	92.0%	8.0%	0.0%	0.0%	0.0%
Provincial expansion	Hu	519	79.0%	16.0%	4.0%	1.0%	0.0%

Appendix

Table 21: AI-policy pair classification in the central-government subset versus the provincial expansion of the integrated corpus. Shares are within frame and era; directional change is expansion + restriction + reversal.

Level of $B$	$n$	Exp.	Res.	Rev.	Directional
National	597	9.5%	0.2%	0.0%	9.7%
Ministerial	3078	7.2%	0.2%	0.1%	7.6%
Provincial	62378	1.4%	0.1%	0.0%	1.6%
Province of $B$	$n$	Exp.	Res.	Rev.	Directional
Anhui	477	14.7%	0.8%	0.0%	15.5%
Beijing	801	7.7%	0.6%	0.0%	8.4%

Appendix

Table 22: AI-policy directional change by administrative level of the later document (top) and by province (bottom; provinces with

n\geq 300

pairs). Directional = expansion + restriction + reversal; remaining pairs are consistency or irrelevant.

Figure 4: Xi-era related AI-policy pairs in the central-government subset and the provincial expansion of the integrated corpus.

System	acc	$\kappa$	F1	m-F1	$\kappa$	acc	cond. acc	rec.
	domain match (binary)			direction (4-way)
Qwen3-14B (zero-shot)	0.751	0.161	0.851	0.330	0.248	0.479	0.490	97.7%
Qwen3-32B (zero-shot)	0.758	0.185	0.855	0.451	0.444	0.599	0.611	98.1%
GPT-5 (zero-shot)	0.673	0.262	0.758	0.345	0.216	0.353	0.501	70.4%
Qwen3-14B LoRA SFT (5-fold pooled)	0.774	0.431	0.845	0.556	0.536	0.639	0.759	84.2%

Appendix

Table 23: Post-hoc decomposition of five-way predictions into the two axes. Domain match is the binary judgment that the two directives govern the same policy domain (the binary axis), scored on all 1,000 pairs; direction is the four-way judgment on the 729 gold-related pairs, with irrelevant or unparsed predictions counted as errors. Rec. is the share of gold-related pairs the system keeps in the related set; cond. acc is direction accuracy restricted to those pairs.

System	band	$n$	domain	dir.	dir. $\|$ matched
Qwen3-14B zero-shot	[0.0, 0.6)	19	0.158	0.769	0.727
	[0.6, 0.7)	151	0.391	0.520	0.429
	[0.7, 0.8)	89	0.315	0.633	0.633
	[0.8, 0.9)	427	0.173	0.390	0.390
	[0.9, 1.0)	186	0.226	0.639	0.634
GPT-5 zero-shot	[0.0, 0.6)	19	0.158	0.462	0.300

Appendix

Table 24: Per-axis error rates by BGE cosine-similarity band. Domain = wrong binary domain-match judgment; dir. = wrong direction on gold-related pairs (boundary misses count as errors); dir.

|

matched = wrong direction given the pair was correctly kept as related.

System	total err.	pure dir.	boundary	% of errors
Qwen3-14B zero-shot	0.612	0.363	0.249	41%
GPT-5 zero-shot	0.583	0.256	0.327	56%
Qwen3-14B LoRA SFT	0.374	0.148	0.226	60%

Appendix

Table 25: Joint error taxonomy over all 1,000 pairs. Pure direction : domain judgment correct, collapsed label wrong. Boundary-involved : domain judgment wrong (the collapsed label is then almost necessarily wrong too). Shares are of all pairs, with the boundary share of all errors in the last column.

Model	elicitation	macro-F1	$\kappa$	acc
qwen3-32b	joint (two outputs, one pass)	0.447	0.333	0.477
	cascade (domain gate, then direction)	0.428	0.303	0.442
	direct (collapsed five-category)	0.462	0.322	0.474
	oracle domain gate (ceiling)	0.618	0.576	0.666
qwen3-4b	joint (two outputs, one pass)	0.238	0.107	0.324
	cascade (domain gate, then direction)	0.406	0.265	0.411

Appendix

Table 26: Elicitation variants of the dual-axis task, scored on the collapsed five-category metric. Joint predicts both outputs in one pass; cascade predicts the domain match first and the direction second; direct predicts only the collapsed relation; oracle replaces the domain judgment with gold.

slug	coverage	same_acc	same_kappa	same_F1	macroF1	kappa	acc	F1_irrelevant	F1_consistency	F1_restriction	F1_expansion	F1_reversal	constraint_viol_rate
claude_sonnet_45__dualaxis__t1500	0.999	0.755	0.170	0.854	0.332	0.216	0.397	0.242	0.518	0.180	0.410	0.312	0.000
claude_sonnet_45__dualaxis_cp__t1500	0.999	0.757	0.184	0.855	0.331	0.213	0.396	0.262	0.512	0.152	0.417	0.312	0.000
deepseek_chat__dualaxis__t1500	1.000	0.751	0.142	0.852	0.362	0.259	0.424	0.204	0.550	0.197	0.474	0.385	0.000
deepseek_chat__dualaxis_cp__t1500	1.000	0.759	0.202	0.855	0.368	0.266	0.429	0.294	0.552	0.183	0.468	0.344	0.001
gemini_25_pro__dualaxis__t1500	1.000	0.755	0.161	0.854	0.388	0.249	0.418	0.227	0.520	0.274	0.365	0.554	0.000
gemini_25_pro__dualaxis_cp__t1500	1.000	0.754	0.162	0.854	0.402	0.266	0.431	0.231	0.527	0.294	0.403	0.554	0.000

Appendix

Table 27: Frontier API flagships under the dual-axis (two-output) formulation: joint same-category and logical-relationship prediction on 1,000 gold pairs, collapsed to the five-way scheme for scoring. Each model appears under the original prompt and under a variant with the corrected related-pair prior (73%).

Regime	Model	$n$	coverage	macro-F1	$\kappa$	acc.
Zero-shot	Qwen3-32B	1000	1.000	0.452	0.337	0.480
Zero-shot	GPT-5	1000	1.000	0.420	0.256	0.417
Zero-shot	Gemini 2.5 Pro	1000	0.998	0.418	0.273	0.436
Zero-shot	Claude Sonnet 4.5	1000	0.999	0.412	0.265	0.432
Zero-shot	DeepSeek-V3	1000	1.000	0.406	0.282	0.440
Few-shot	GPT-5	995	0.999	0.445	0.279	0.436

Appendix

Table 28: Training-regime comparison on the expert gold set. Zero-shot rows use the main prompt for Qwen3-32B and the direct five-category prompt at 1,500 characters for frontier models; few-shot rows use the class-balanced corrected-prior prompt.

Setting	$n_{\mathrm{train}}$	cov%	macro-F1	$\kappa$	acc
Qwen3-14B zero-shot (main prompt)	0	100.0	0.342	0.202	0.388
Qwen3-14B LoRA SFT (n=50)	50	100.0	0.448	0.315	0.473
Qwen3-14B LoRA SFT (n=100)	100	100.0	0.510	0.361	0.498
Qwen3-14B LoRA SFT (n=200)	200	100.0	0.550	0.417	0.543
Qwen3-14B LoRA SFT (n=400)	400	100.0	0.587	0.460	0.575
Qwen3-14B LoRA SFT (full gold)	804	100.0	0.634	0.524	0.626

Appendix

Table 29: Expert-annotation learning curve for the five-way classifier under the unchanged LoRA recipe of the main paper.

Figure 5: Five-way classification performance by expert-annotation sample size.

		F1 [95% CI]
Class	$N_{gold}$	Qwen3-32B (zs)	GPT-5 (zs)	Qwen3-14B (SFT)
CONS	255	0.567 [0.522,0.610]	0.199 [0.145,0.253]	0.710 [0.664,0.753]
EXPAN	171	0.531 [0.476,0.585]	0.433 [0.378,0.486]	0.551 [0.487,0.612]
RESTR	176	0.425 [0.350,0.495]	0.357 [0.289,0.428]	0.544 [0.481,0.601]
CONTR	127	0.477 [0.375,0.568]	0.618 [0.537,0.690]	0.780 [0.716,0.833]
UNREL	271	0.262 [0.199,0.327]	0.495 [0.448,0.542]	0.586 [0.536,0.633]

Appendix

Table 30: Per-class F1 with 95% bootstrap confidence intervals for the best open-weight zero-shot model, a frontier flagship, and the fine-tuned model.

Figure 6: Predicted-label distribution shift from gold by model family. Lower TVD means the model’s predicted label marginal is closer to the expert gold marginal.

Model	Family	Cond.	Cov.	TVD	KL	MacroRec.	Over	Under
claude-sonnet-4.5	Claude	zs	0.999	0.193	0.116	0.411	EXPAN	RESTR
deepseek-r1-distill-qwen-14b	DeepSeek	zs	1.000	0.526	0.612	0.317	CONS	UNREL
deepseek-v3	DeepSeek	zs	1.000	0.450	0.487	0.362	CONS	UNREL
deepseek-v3-chat	DeepSeek	zs	1.000	0.268	0.216	0.428	EXPAN	RESTR
glm-4-9b-chat	GLM	zs	0.822	0.437	1.082	0.303	EXPAN	RESTR
gpt-5	GPT	zs	1.000	0.248	0.132	0.428	EXPAN	CONS

Appendix

Table 31: Label-distribution bias by model and family on the same 1,000-pair gold set. ‘Over’ and ‘Under’ name the classes with the largest positive and negative predicted-minus-gold marginal shifts.

Class	$N_{gold}$	Precision		Recall		F1
		ZS	SFT	ZS	SFT	ZS	SFT
CONS	255	0.340	0.740	0.918	0.682	0.496	0.710
EXPAN	171	0.469	0.536	0.222	0.567	0.302	0.551
RESTR	176	0.336	0.518	0.256	0.574	0.290	0.544
CONTR	127	0.800	0.825	0.252	0.740	0.383	0.780
UNREL	271	0.696	0.582	0.144	0.590	0.239	0.586

Appendix

Table 32: Qwen3-14B before and after expert fine-tuning on the same 1,000 pairs.

Family	Condition	Cov%	Macro-F1	$\kappa$	Acc	cons	expa	rest	reve	irre
Mistral-7B-Instruct-v0.2	zero-shot	84.2	0.180	0.182	0.226	0.373	0.043	0.029	0.000	0.457
	SFT (5-fold LoRA)	100.0	0.608	0.492	0.600	0.690	0.523	0.516	0.770	0.539
Qwen3-14B	zero-shot	100.0	0.342	0.202	0.388	0.496	0.302	0.290	0.383	0.239
	SFT (5-fold LoRA)	100.0	0.634	0.524	0.626	0.710	0.551	0.544	0.780	0.586
	SFT - zero-shot	15.8	0.428	0.310	0.374	0.317	0.480	0.487	0.770	0.082
	SFT - zero-shot	0.0	0.292	0.322	0.238	0.214	0.249	0.254	0.397	0.347

Appendix

Table 33: Cross-family correction from expert fine-tuning. Mistral-7B-Instruct-v0.2 and Qwen3-14B use the same five folds and LoRA recipe; lower-coverage zero-shot Mistral is scored under the zero-shot convention that counts unparsed outputs as errors, so coverage is reported alongside performance.

Figure 7: Zero-shot versus fine-tuned performance for Mistral-7B-Instruct-v0.2 and Qwen3-14B under the same five-fold recipe.

Technique	$n$	macro-F1	$\kappa$	acc.	note
SFT fold-0 baseline	196	0.635	0.533	0.633	replication
Self-consistency $k=8$	196	0.638	0.553	0.648	test-time vote
RAFT	196	0.623	0.524	0.622
DPO	196	0.632	0.538	0.638
GRPO	196	0.622	0.519	0.617
KTO	–	–	–	–	deterministic TRL failure

Appendix

Table 34: Training- and test-time technique comparison. Fold-0 validation arms use

n=196

; the canonical 80/20 row uses its held-out

n=204

split and is reported separately from the fold-0 comparison. KTO is included for completeness but produced no score because TRL 0.23.1 failed deterministically at training step 79/90 across two GPU nodes.

Quantity	Value
Pairs	5,000
Unanimous (5/5)	3,522 (70.4%)
4/5 agreement	765
3/5 agreement	645
2/5 agreement	68
Mean pairwise adapter agreement	0.849

Appendix

Table 35: Five-adapter teacher labeling of the 5,000-pair unlabeled slice.

slug	n_scored	coverage	macroF1	kappa	acc	F1_irrelevant	F1_consistency	F1_restriction	F1_expansion	F1_reversal
deepseek_chat__direct5__t0	1000	1.000	0.390	0.256	0.416	0.506	0.284	0.164	0.470	0.525
deepseek_chat__direct5__t4000	1000	0.999	0.391	0.258	0.420	0.510	0.322	0.141	0.475	0.506
deepseek_chat__direct5__t8000	1000	0.996	0.397	0.266	0.425	0.511	0.352	0.160	0.470	0.494
gemini_25_pro__direct5__t1500__temp0.7__seed1	1000	0.998	0.434	0.290	0.449	0.521	0.398	0.237	0.465	0.546
gemini_25_pro__direct5__t1500__temp0.7__seed2	1000	0.997	0.431	0.289	0.448	0.521	0.386	0.221	0.480	0.546
gpt_5__direct5__t1500__temp0.7__seed1	1000	1.000	0.425	0.262	0.423	0.514	0.216	0.337	0.420	0.638

Appendix

Table 36: Frontier flagship variants: few-shot, multi-seed (temperature 0.7), and context-length arms under the direct five-category prompt.

	five-way (all pairs)			direction (related pairs)
System	m-F1	$\kappa$	acc	m-F1	$\kappa$	cond. acc
Signal-only rule (obligations)	–	–	–	0.074	0.023	0.431
Signal-only rule (flexibility)	–	–	–	0.074	0.017	0.382
Prompt fusion (signal line in prompt)	0.638	0.530	0.629	0.565	0.558	0.755
Fine-tuned classifier (no signal)	0.649	0.548	0.643	0.572	0.576	0.759
Hard-rule override (neg. control)	0.631	0.524	0.623	0.552	0.545	0.725

Appendix

Table 37: Auxiliary policy-signal arms on the held-out folds. Signal-only rule maps the between-document shift in the share of obligation-hardening paragraphs (prohibiting + pressuring, within each document’s actionable paragraphs) to a direction label with a threshold tuned on the training folds; it never sees the texts and cannot express reversal. Prompt fusion appends one signal-profile line per document to the prompt of the fine-tuned classifier. Hard-rule override forces the classifier’s answer to expansion/restriction whenever the signal shift exceeds the same threshold (negative control). Direction metrics are four-way on gold-related pairs; abstentions count as errors.

Policy Evaluation

Policy evaluation estimates the outcomes, value, risks, and fairness of decision rules before or after deployment.

See all 61 papers

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models
Apr 14, 2026Han Bao, Penghao Zhang, Yue Huang +9LLM Evaluation Mixture-of-Experts Language Models
University of Notre Dame · Independent Researcher
Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error
Sep 13, 2026Hongliu CaoAI Agent Evaluation Automated Agent Evaluation
Amadeus SAS
PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs
Aug 10, 2026Ponkrit Kaewsawee, Chaklam Silpasuwanchai, Chutiporn AnutariyaKG Construction Document Information Extraction
Asian Institute of Technology, Pathum Thani, Thailand

Apr 14, 2026J/K move · Enter open · S save

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

Han Bao, Penghao Zhang, Yue Huang, Zhengqing Yuan+8

University of Notre Dame · Independent Researcher

LLM Evaluation Mixture-of-Experts Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present PolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) Memorization: factual recall of policy knowledge, (2) Understanding: conceptual and contextual reasoning, and (3) Application: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose PolicyMoE, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models.

Abstract

Policy Evaluation

Explore similar work

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs