Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations

Yonghong Zhang; Yong Xie; Isabel M. Parra; Ricardo Correia

cs.CLSep 25, 2026

Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations

Authors: Yonghong Zhang, Yong Xie, Isabel M. Parra, Ricardo Correia

Organizations: Department of Finance, Universidad Autónoma de Madrid, Madrid, Spain · Spanish National Research Council (CSIC), Madrid, Spain

Abstract

Difference-in-differences (DID) studies are widely used to evaluate climate policy, but assessing the evidence supporting their identification assumptions remains challenging. We introduce ARGUS, a structured language-model pipeline that audits reported evidence against an eleven-dimension assumption-implication-evidence rubric and abstains when relevant evidence cannot be retrieved. We evaluate ARGUS using injected flaws, economics papers, and a small pilot with reconciled labels. On the 11-flaw benchmark, ARGUS detects 73% of planted flaws, compared with 18% for a keyword-based pipeline. Across 26 economics papers, ARGUS abstains on about 40% of paper-dimension assessments for lack of retrievable evidence. In a five-paper pilot with labels reconciled by two annotators, it assigns a higher risk level than the labels on 25 of the 33 assessments it completes. A rule fixed before the labels arrived removes most of this in-sample; weighted agreement stays low. ARGUS provides evidence-linked risk reports that localize potential weaknesses for expert review, without adjudicating causal claims. Code and data: https://github.com/yonghongzhang-io/ARGUS

Figures & tables

Figure 1: Overview of ARGUS. Step 1 converts an empirical DID paper into an auditable structure using an 11-dimension identification rubric. Step 2 retrieves dimension-specific evidence, assesses reported-evidence adequacy with a bounded LLM audit module, and localizes risk in a human-reviewable report. Step 3 uses labels from two annotators, reconciled by them, to evaluate audit severity and score a pre-specified calibration rule offline; deployment requires only expert review of the report. Colours as in Figure 2 : blue, deterministic steps; green, model calls; orange, human steps.

Figure 2: ARGUS architecture. The top band is the audit pipeline, a fixed deterministic sequence; model calls are confined to two stages (extraction and assessment). As evaluated, these resolve per dimension to a fixed call sequence (lexical retrieval, one relevance-gate call, one adequacy call) with a step budget of one and a logged trace; the designed tools figure_parse and policy_lookup are not exercised (§ 3.2 ). The bottom band shows the flaw-injection evaluation loop, which injects one known flaw, audits the clean and injected versions, and scores detection, false alarm, and localization against the injected flaw as local ground truth, never against the unobserved true effect.

assessor	calls	detection	localization
keyword baseline	0	0.182	0.182
single-pass	1	0.818	0.818
per-dimension, no retrieval	11	1.000	1.000
two-stage ARGUS	22	0.727	0.727
(circular, pre-fix)	–	1.000	1.000

Table 1: Synthetic flaw injection on the clean_supported fixture, eleven flaws; all LLM rows gpt-4o, detection replicated on the dated 2024-11-20 snapshot; see Appendix F on run-to-run differences). Every row is an end-to-end pipeline with its own evidence path (keyword and two-stage do not share retrieved evidence, Appendix D ; Appendix K holds it fixed); the ablation arms share the model and rubric but remove retrieval, the gate, or both. The final row is the leaked, pre-fix circular setup. On the 33-variant benchmark the two-stage assessor rates one clean dimension medium, a

0.09

over-flag (§ 5.2 ). Wilson 95% intervals for the main rates are in Table 8 .

Figure 3: Keyword pipeline vs LLM evidence-adequacy pipeline (

n{=}11

flaws, structural injection). (a) End to end, the LLM pipeline raises detection and localization

0.182\,{\to}\,0.727

with false alarm

0

(the two pipelines retrieve evidence differently; Appendix K holds it fixed). (b) Per-flaw caught/missed: the keyword baseline is blind to every commission-type flaw , all of which the LLM catches; the LLM’s three misses are omissions on which it abstains. Table 2 reports the 33-variant benchmark.

variant type	$n$	detection	localization
commission	22	0.89	0.82
omission	11	0.45	0.33
all	33	0.75	0.66

Table 2: The 33-variant benchmark (gpt-4o, mean of three runs at temperature 0; detection decisions identical on 32/33 variants). Adequacy reasoning catches commission flaws; omission misses are abstentions to unknown , marking the evidence-grounding bottleneck rather than wrong judgements. Intervals in Table 8 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

flaw	dimension	threat named in
pretrend divergence	parallel trends	Roth (2022) ; Roth et al. (2023) ; Rambachan and Roth (2023)
anticipation effect	no anticipation	Callaway and Sant’Anna (2021) ; Roth et al. (2023)
forbidden comparison	staggered timing	Goodman-Bacon (2021) ; de Chaisemartin and D’Haultfœuille (2020) ; Sun and Abraham (2021)
spillover contamination	SUTVA / spillovers	Athey and Imbens (2017) ; Roth et al. (2023)
noncomparable controls	control group	Callaway and Sant’Anna (2021) ; Athey and Imbens (2017)
bad control	specification	Angrist and Pischke (2009)

Appendix

Table 3: Each injected flaw, its target dimension, and the literature that names the threat.

arm	calls/paper	det	fa	loc
full-paper single pass	1	0.727	0.000	0.727
per-dimension, no retrieval	11	0.879	0.000	0.879
two-stage (retrieval + gate)	22	0.747	0.091	0.657

Appendix

Table 4: Compute-graded ablation on the 33-variant benchmark (gpt-4o). det / fa / loc = detection, false alarm, localization. Calls per paper are the nominal maximum: when the gate finds nothing relevant the adequacy call is skipped, and retries are not counted. Localization takes the top-ranked flagged dimension; ties in risk level are broken by the fixed rubric order.

model	det	fa	loc	comm	omit
gpt-4o	0.75	0.09	0.66	0.89	0.45
Claude Opus 4.8	0.91	0.82	0.24	1.00	0.73
Gemini 2.5 Flash	0.79	0.18	0.36	0.91	0.55
Llama 3.1 8B (local)	0.94	0.27	0.27	0.95	0.91

Appendix

Table 5: Cross-model panel on the 33-variant benchmark (single run per model; gpt-4o is the three-run mean; single runs support the qualitative porting claim only). comm/omit = commission/omission detection. Intervals in Table 8 .

model	omission verdicts	clean $\geq$ med	agree
gpt-4o	6 / 3 / 2 / 0	2 / 22	–
Claude Opus 4.8	3 / 7 / 1 / 0	18 / 22	20 / 33
Gemini 2.5 Flash	4 / 3 / 3 / 1	4 / 22	26 / 33
Llama 3.1 8B (local)	1 / 10 / 0 / 0	6 / 22	19 / 33

Appendix

Table 6: Cross-model operating points on the 33-variant benchmark (single runs; gpt-4o run 1). Omission column: verdicts on the eleven deleted-section variants, in the order unknown / high / medium / low. Clean column: commission pairs (of 22) whose unmodified fixture is already rated medium or high. Last column: exact agreement with gpt-4o’s injected-variant verdict.

retrieval quality	unknown	high	medium	low
failed (22)	22	0	0	0
weak (22)	0	20	2	0
good (11)	0	5	5	1

Appendix

Table 7: Retrieval quality versus emitted risk on the 55 pilot-gold cells (re-run recording retrieval quality and evidence status). Failed retrieval always abstains; weak retrieval almost always becomes a substantive high , 19 of those 20 with evidence_status=missing .

benchmark	quantity	$k/n$	95% CI
11 flaws	keyword detection	2/11	0.05–0.48
	single-pass detection	9/11	0.52–0.95
	per-dimension detection	11/11	0.74–1.00
	two-stage detection	8/11	0.43–0.90
	two-stage false alarm	0/11	–
33 variants	single-pass detection	24/33	0.56–0.85

Appendix

Table 8: Wilson 95% confidence intervals (CI) for reported rates (top) and exact tests between arms (bottom; McNemar tests are paired on the same flaws or variants; discordant = detected by the first arm only / by the second only). False-alarm rows give counts only: every variant is paired with the same clean fixture, so they are repeated readings of one clean audit of eleven dimensions, not independent trials, and an interval would overstate what they show. Detection intervals describe this fixed test set (33 variants nested in eleven dimensions of one fixture), not real papers at large.

policy	$n$	exact agreement	weighted $\kappa$
before	33	0.24 (0.09–0.39)	0.08 ( $-0.05$ –0.25)
rule 1, demote $\to$ med	33	0.76 (0.61–0.88)	0.23 ( $-0.22$ –0.63)
rule 1, abstain $\to$ unk	13	0.62 (0.38–0.85)	0.29 ( $-0.18$ –0.77)
rules 1–4, demote $\to$ med	33	0.85 (0.73–0.97)	0.22 ( $-0.42$ –0.79)
rules 1–4, abstain $\to$ unk	13	0.85 (0.62–1.00)	0.32 ( $-0.52$ –1.00)

Appendix

Table 9: Cell-bootstrap 95% intervals for agreement with the reconciled gold (answered cells only; quadratic-weighted

\kappa

; 5,000 resamples). Rule 1 is the weak-retrieval rule; rules 1–4 add the three single-cell rules of Appendix J . Resampling cells ignores their nesting in five papers and may understate uncertainty.

venue / version	papers
American Economic Review	4
Quarterly Journal of Economics	3
Journal of Political Economy	2
Journal of Financial Economics	2
AEJ: Applied Economics	2
JF / REStat / REStud / J. Monetary Econ. / J. Int. Econ. (one each)	5

Appendix

Table 10: Venue of the 26 audited papers, as recorded for the version held in the source corpus. Two entries carry a corrupted venue field in the corpus index; they are counted under the journal that published them (Journal of Political Economy, 2003; Review of Economic Studies, 2002). AEJ: American Economic Journal; JF: Journal of Finance; REStat: Review of Economics and Statistics; REStud: Review of Economic Studies.

The Effect of the Banking Panic on the Supply of Credit to the Corporate Sector

The Effects of State-Level Banking Competition on Innovation

Do Credit Market Shocks Affect the Real Economy? Quasi-Experimental Evidence from the Great Recession and ‘Normal’ Economic Times

Cyclicality of Credit Supply: Firm Level Evidence

Does Stock Liquidity Enhance or Impede Firm Innovation?

Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania

Minimum Wage Effects Across State Borders: Estimates Using Contiguous Counties

Appendix

Table 11: The 26 audited studies, by title as indexed in the source corpus (the duplicated entry is listed once).

dimension	risk	rationale (abridged)
no anticipation	high	outcomes began changing between the policy announcement and its rollout, undermining no-anticipation
inference	medium	clustering and wild-cluster bootstrap reported, but the number of clusters is not stated
parallel trends	low	event-study with null pre-period coefficients and placebo tests
staggered timing	low	heterogeneity-robust estimators with a Goodman-Bacon decomposition
7 further dimensions	low	supporting evidence retrieved and judged adequate

Appendix

Table 12: ARGUS audit report (abridged) on the fixture with the injected anticipation flaw. Rationales are abridged from the evidence-cited report; the injected dimension is the one flagged high.

unknown policy	$n$	exact	$\kappa$	wt. $\kappa$
exclude (coverage)	33	0.242	0.031	0.128
mismatch (strict)	55	0.145	0.007	–

Appendix

Table 13: Two-stage ARGUS vs reconciled human gold (55 cells, 5 papers) under two unknown policies: coverage (drop abstentions) and strict ( unknown counts as mismatch). Pilot-scale: only 2 gold cells are high ; the robust signal is the one-directional over-severity and the abstention rate.

Figure 4: Two-stage ARGUS versus the reconciled gold on the 55 pilot cells (frozen protocol). (a) Every cell, eleven dimensions by five papers, coloured by outcome: ARGUS abstains on 22 and, of the 33 it answers, is more severe than the gold on 25, matches it on 8, and is less severe on none. (b) Distribution of risk labels in the gold and in ARGUS’s output, and the outcome of the 55 cells in the colours of panel (a). (c) ARGUS-by-gold confusion matrix (counts).

Figure 5: Per-dimension risk distribution on the 26 papers tagged DID in the source corpus. The keyword baseline (left) is almost entirely low ; the two-stage LLM (right) is differentiated, with a large unknown share that marks retrieval failure rather than over-flagging.

		rule 1		rules 1–4
metric	before	med	unk	med	unk
over-severe (answ.)	25	8	5	5	2
high precision	1/25	1/5	1/5	1/2	1/2
high recall	1/2	1/2	1/2	1/2	1/2
exact agr. (answ.)	0.24	0.76	0.62	0.85	0.85
wt. $\kappa$ (answ.)	0.08	0.23	0.29	0.22	0.32

Appendix

Table 14: Deterministic calibration vs the human gold (55 cells), in-sample. Two policies for a weak-retrieval high : demote to medium (med, keep coverage) or abstain to unknown (unk). Rule 1 alone is the result we claim; rules 1–4 add three single-cell exploratory rules (Appendix J ). Under abstention rule 1 changes no retained label: its higher agreement comes from dropping 20 answers. Pilot scale: only 2 gold cells are high ; a constant medium label scores

26/33

on the answered cells, so exact agreement carries little information here. The before column is the rich re-run of § 5.6 , which differs from the corpus run of Table 13 in one cell (a gold-low cell moves from medium to high , which is why weighted

\kappa

is 0.08 here and 0.13 there); intervals in Table 9 .

Figure 6: Calibration trade-off: severity reduction versus coverage against the reconciled human gold, for the weak-retrieval rule alone (in-sample; med and unk are the two policies of Table 14 , which also gives the exploratory four-rule set). Demoting weak-retrieval high s to medium preserves coverage (33 answered) while cutting over-severe judgements from 25 to 8; abstaining leaves fewer over-severe cells (5) and a higher weighted

\kappa

(

0.29

) only by discarding 20 answers, and leaves the remaining labels unchanged. Most of the gain in exact agreement reflects a gold that is

84\%

medium (§ 5.6 ); the remaining bottleneck is evidence grounding, not severity thresholding alone.

	11 flaws			33 variants
cell	det	fa	loc	det	fa	loc	off
K-kw	2/11	0/11	2/11	8/33	0/33	8/33	0.00
K-llm	10/11	1/11	8/11	31/33	3/33	22/33	0.15
S-kw	1/11	0/11	1/11	2/33	0/33	2/33	0.00
S-llm	11/11	0/11	11/11	31/33	0/33	22/33	0.16
gated two-stage	8/11	0/11	8/11	25/33	3/33	0.66	–

Appendix

Table 15: Shared-evidence control: evidence bundle (K, keyword chunks; S, ungated sections)

\times

judgement policy (kw, keyword scorer; llm, adequacy judge). det / fa / loc as in Table 4 ; off = share of the ten non-target dimensions rated

\geq

medium on the injected variants. The reference row is the gated pipeline on the gpt-4o alias (detection and false alarm from run 1, Table 8 ; localization is the three-run mean of Table 2 ).

Causal Inference

Causal inference studies how interventions and counterfactuals reveal cause-and-effect relationships beyond observed associations.

See all 314 papers

Position: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claims
May 8, 2026Zezheng Lin, Fengming LiuMechanistic Interpretability Causal Identifiability
Preprint.
ClimateCause: Complex and Implicit Causal Structures in Climate Reports
Apr 16, 2026Liesbeth Allein, Nataly Pineda-Castañeda, Andrea Rocci +1Relation Extraction Climate Science
Department of Computer Science, KU Leuven, Belgium · Institute of Argumentation, Linguistics, and Semiotics (IALS), Universit`a della Svizzera italiana, Switzerland
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Feb 24, 2026Ayush Sawarni, Jiyuan Tan, Vasilis SyrgkanisCausal Effect Estimation Benchmark Design
Stanford University

May 8, 2026J/K move · Enter open · S save

Position: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claims

Zezheng Lin, Fengming Liu

Preprint.

Mechanistic Interpretability Causal Identifiability Causal Interpretability Explainability Evaluation

Open paper AskPDF arXiv

Mechanistic interpretability papers increasingly use causal vocabulary: circuits, mediators, causal abstraction, monosemanticity. Such claims require explicit identification assumptions. A purposive audit of 10 papers across four methodological strands finds no dedicated identification-assumptions section and a recurring pattern: validation metrics such as faithfulness, completeness, monosemanticity, alignment, or ablation effects are reported as causal support without stating the assumptions that make them identifying. A two-human-coder audit on

n=30

reproduces the direction of the main finding: dedicated identification sections are absent, and validation-metric substitution is common, though exact Dim B/D counts are coding-rule sensitive. The paper proposes a disclosure norm: state whether the claim is causal, name the identification strategy, enumerate assumptions, stress at least one, and explain how conclusions shift if assumptions fail. Validation is not identification.

Abstract

Causal Inference

Explore similar work

Position: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claims

ClimateCause: Complex and Implicit Causal Structures in Climate Reports

CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation