Difference-in-differences (DID) studies are widely used to evaluate climate policy, but assessing the evidence supporting their identification assumptions remains challenging. We introduce ARGUS, a structured language-model pipeline that audits reported evidence against an eleven-dimension assumption-implication-evidence rubric and abstains when relevant evidence cannot be retrieved. We evaluate ARGUS using injected flaws, economics papers, and a small pilot with reconciled labels. On the 11-flaw benchmark, ARGUS detects 73% of planted flaws, compared with 18% for a keyword-based pipeline. Across 26 economics papers, ARGUS abstains on about 40% of paper-dimension assessments for lack of retrievable evidence. In a five-paper pilot with labels reconciled by two annotators, it assigns a higher risk level than the labels on 25 of the 33 assessments it completes. A rule fixed before the labels arrived removes most of this in-sample; weighted agreement stays low. ARGUS provides evidence-linked risk reports that localize potential weaknesses for expert review, without adjudicating causal claims. Code and data: https://github.com/yonghongzhang-io/ARGUS
Figures & tables
Figure 1: Overview of ARGUS. Step 1 converts an empirical DID paper into an auditable structure using an 11-dimension identification rubric. Step 2 retrieves dimension-specific evidence, assesses reported-evidence adequacy with a bounded LLM audit module, and localizes risk in a human-reviewable report. Step 3 uses labels from two annotators, reconciled by them, to evaluate audit severity and score a pre-specified calibration rule offline; deployment requires only expert review of the report. Colours as in Figure 2 : blue, deterministic steps; green, model calls; orange, human steps.
Figure 2: ARGUS architecture. The top band is the audit pipeline, a fixed deterministic sequence; model calls are confined to two stages (extraction and assessment). As evaluated, these resolve per dimension to a fixed call sequence (lexical retrieval, one relevance-gate call, one adequacy call) with a step budget of one and a logged trace; the designed tools figure_parse and policy_lookup are not exercised (§ 3.2 ). The bottom band shows the flaw-injection evaluation loop, which injects one known flaw, audits the clean and injected versions, and scores detection, false alarm, and localization against the injected flaw as local ground truth, never against the unobserved true effect.
assessor
calls
detection
false alarm
localization
keyword baseline
0
0.182
0.000
0.182
single-pass
1
0.818
0.000
0.818
per-dimension, no retrieval
11
1.000
0.000
1.000
two-stage ARGUS
22
0.727
0.000
0.727
(circular, pre-fix)
–
1.000
0.000
1.000
Table 1: Synthetic flaw injection on the clean_supported fixture, eleven flaws; all LLM rows gpt-4o, detection replicated on the dated 2024-11-20 snapshot; see Appendix F on run-to-run differences). Every row is an end-to-end pipeline with its own evidence path (keyword and two-stage do not share retrieved evidence, Appendix D ; Appendix K holds it fixed); the ablation arms share the model and rubric but remove retrieval, the gate, or both. The final row is the leaked, pre-fix circular setup. On the 33-variant benchmark the two-stage assessor rates one clean dimension medium, a 0.09 over-flag (§ 5.2 ). Wilson 95% intervals for the main rates are in Table 8 .
Figure 3: Keyword pipeline vs LLM evidence-adequacy pipeline ( n=11 flaws, structural injection). (a) End to end, the LLM pipeline raises detection and localization 0.182→0.727 with false alarm 0 (the two pipelines retrieve evidence differently; Appendix K holds it fixed). (b) Per-flaw caught/missed: the keyword baseline is blind to every commission-type flaw , all of which the LLM catches; the LLM’s three misses are omissions on which it abstains. Table 2 reports the 33-variant benchmark.
variant type
n
detection
localization
commission
22
0.89
0.82
omission
11
0.45
0.33
all
33
0.75
0.66
Table 2: The 33-variant benchmark (gpt-4o, mean of three runs at temperature 0; detection decisions identical on 32/33 variants). Adequacy reasoning catches commission flaws; omission misses are abstentions to unknown , marking the evidence-grounding bottleneck rather than wrong judgements. Intervals in Table 8 .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
flaw
dimension
threat named in
pretrend divergence
parallel trends
Roth (2022) ; Roth et al. (2023) ; Rambachan and Roth (2023)
anticipation effect
no anticipation
Callaway and Sant’Anna (2021) ; Roth et al. (2023)
forbidden comparison
staggered timing
Goodman-Bacon (2021) ; de Chaisemartin and D’Haultfœuille (2020) ; Sun and Abraham (2021)
spillover contamination
SUTVA / spillovers
Athey and Imbens (2017) ; Roth et al. (2023)
noncomparable controls
control group
Callaway and Sant’Anna (2021) ; Athey and Imbens (2017)
bad control
specification
Angrist and Pischke (2009)
Appendix
Table 3: Each injected flaw, its target dimension, and the literature that names the threat.
arm
calls/paper
det
fa
loc
full-paper single pass
1
0.727
0.000
0.727
per-dimension, no retrieval
11
0.879
0.000
0.879
two-stage (retrieval + gate)
22
0.747
0.091
0.657
Appendix
Table 4: Compute-graded ablation on the 33-variant benchmark (gpt-4o). det / fa / loc = detection, false alarm, localization. Calls per paper are the nominal maximum: when the gate finds nothing relevant the adequacy call is skipped, and retries are not counted. Localization takes the top-ranked flagged dimension; ties in risk level are broken by the fixed rubric order.
model
det
fa
loc
comm
omit
gpt-4o
0.75
0.09
0.66
0.89
0.45
Claude Opus 4.8
0.91
0.82
0.24
1.00
0.73
Gemini 2.5 Flash
0.79
0.18
0.36
0.91
0.55
Llama 3.1 8B (local)
0.94
0.27
0.27
0.95
0.91
Appendix
Table 5: Cross-model panel on the 33-variant benchmark (single run per model; gpt-4o is the three-run mean; single runs support the qualitative porting claim only). comm/omit = commission/omission detection. Intervals in Table 8 .
model
omission verdicts
clean ≥ med
agree
gpt-4o
6 / 3 / 2 / 0
2 / 22
–
Claude Opus 4.8
3 / 7 / 1 / 0
18 / 22
20 / 33
Gemini 2.5 Flash
4 / 3 / 3 / 1
4 / 22
26 / 33
Llama 3.1 8B (local)
1 / 10 / 0 / 0
6 / 22
19 / 33
Appendix
Table 6: Cross-model operating points on the 33-variant benchmark (single runs; gpt-4o run 1). Omission column: verdicts on the eleven deleted-section variants, in the order unknown / high / medium / low. Clean column: commission pairs (of 22) whose unmodified fixture is already rated medium or high. Last column: exact agreement with gpt-4o’s injected-variant verdict.
retrieval quality
unknown
high
medium
low
failed (22)
22
0
0
0
weak (22)
0
20
2
0
good (11)
0
5
5
1
Appendix
Table 7: Retrieval quality versus emitted risk on the 55 pilot-gold cells (re-run recording retrieval quality and evidence status). Failed retrieval always abstains; weak retrieval almost always becomes a substantive high , 19 of those 20 with evidence_status=missing .
benchmark
quantity
k/n
95% CI
11 flaws
keyword detection
2/11
0.05–0.48
single-pass detection
9/11
0.52–0.95
per-dimension detection
11/11
0.74–1.00
two-stage detection
8/11
0.43–0.90
two-stage false alarm
0/11
–
33 variants
single-pass detection
24/33
0.56–0.85
Appendix
Table 8: Wilson 95% confidence intervals (CI) for reported rates (top) and exact tests between arms (bottom; McNemar tests are paired on the same flaws or variants; discordant = detected by the first arm only / by the second only). False-alarm rows give counts only: every variant is paired with the same clean fixture, so they are repeated readings of one clean audit of eleven dimensions, not independent trials, and an interval would overstate what they show. Detection intervals describe this fixed test set (33 variants nested in eleven dimensions of one fixture), not real papers at large.
policy
n
exact agreement
weighted κ
before
33
0.24 (0.09–0.39)
0.08 ( −0.05 –0.25)
rule 1, demote → med
33
0.76 (0.61–0.88)
0.23 ( −0.22 –0.63)
rule 1, abstain → unk
13
0.62 (0.38–0.85)
0.29 ( −0.18 –0.77)
rules 1–4, demote → med
33
0.85 (0.73–0.97)
0.22 ( −0.42 –0.79)
rules 1–4, abstain → unk
13
0.85 (0.62–1.00)
0.32 ( −0.52 –1.00)
Appendix
Table 9: Cell-bootstrap 95% intervals for agreement with the reconciled gold (answered cells only; quadratic-weighted κ ; 5,000 resamples). Rule 1 is the weak-retrieval rule; rules 1–4 add the three single-cell rules of Appendix J . Resampling cells ignores their nesting in five papers and may understate uncertainty.
venue / version
papers
American Economic Review
4
Quarterly Journal of Economics
3
Journal of Political Economy
2
Journal of Financial Economics
2
AEJ: Applied Economics
2
JF / REStat / REStud / J. Monetary Econ. / J. Int. Econ. (one each)
5
Appendix
Table 10: Venue of the 26 audited papers, as recorded for the version held in the source corpus. Two entries carry a corrupted venue field in the corpus index; they are counted under the journal that published them (Journal of Political Economy, 2003; Review of Economic Studies, 2002). AEJ: American Economic Journal; JF: Journal of Finance; REStat: Review of Economics and Statistics; REStud: Review of Economic Studies.
The Effect of the Banking Panic on the Supply of Credit to the Corporate Sector
The Effects of State-Level Banking Competition on Innovation
Do Credit Market Shocks Affect the Real Economy? Quasi-Experimental Evidence from the Great Recession and ‘Normal’ Economic Times
Cyclicality of Credit Supply: Firm Level Evidence
Does Stock Liquidity Enhance or Impede Firm Innovation?
Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania
Minimum Wage Effects Across State Borders: Estimates Using Contiguous Counties
Appendix
Table 11: The 26 audited studies, by title as indexed in the source corpus (the duplicated entry is listed once).
dimension
risk
rationale (abridged)
no anticipation
high
outcomes began changing between the policy announcement and its rollout, undermining no-anticipation
inference
medium
clustering and wild-cluster bootstrap reported, but the number of clusters is not stated
parallel trends
low
event-study with null pre-period coefficients and placebo tests
staggered timing
low
heterogeneity-robust estimators with a Goodman-Bacon decomposition
7 further dimensions
low
supporting evidence retrieved and judged adequate
Appendix
Table 12: ARGUS audit report (abridged) on the fixture with the injected anticipation flaw. Rationales are abridged from the evidence-cited report; the injected dimension is the one flagged high.
unknown policy
n
exact
κ
wt. κ
exclude (coverage)
33
0.242
0.031
0.128
mismatch (strict)
55
0.145
0.007
–
Appendix
Table 13: Two-stage ARGUS vs reconciled human gold (55 cells, 5 papers) under two unknown policies: coverage (drop abstentions) and strict ( unknown counts as mismatch). Pilot-scale: only 2 gold cells are high ; the robust signal is the one-directional over-severity and the abstention rate.
Figure 4: Two-stage ARGUS versus the reconciled gold on the 55 pilot cells (frozen protocol). (a) Every cell, eleven dimensions by five papers, coloured by outcome: ARGUS abstains on 22 and, of the 33 it answers, is more severe than the gold on 25, matches it on 8, and is less severe on none. (b) Distribution of risk labels in the gold and in ARGUS’s output, and the outcome of the 55 cells in the colours of panel (a). (c) ARGUS-by-gold confusion matrix (counts).
Figure 5: Per-dimension risk distribution on the 26 papers tagged DID in the source corpus. The keyword baseline (left) is almost entirely low ; the two-stage LLM (right) is differentiated, with a large unknown share that marks retrieval failure rather than over-flagging.
rule 1
rules 1–4
metric
before
med
unk
med
unk
over-severe (answ.)
25
8
5
5
2
high precision
1/25
1/5
1/5
1/2
1/2
high recall
1/2
1/2
1/2
1/2
1/2
exact agr. (answ.)
0.24
0.76
0.62
0.85
0.85
wt. κ (answ.)
0.08
0.23
0.29
0.22
0.32
Appendix
Table 14: Deterministic calibration vs the human gold (55 cells), in-sample. Two policies for a weak-retrieval high : demote to medium (med, keep coverage) or abstain to unknown (unk). Rule 1 alone is the result we claim; rules 1–4 add three single-cell exploratory rules (Appendix J ). Under abstention rule 1 changes no retained label: its higher agreement comes from dropping 20 answers. Pilot scale: only 2 gold cells are high ; a constant medium label scores 26/33 on the answered cells, so exact agreement carries little information here. The before column is the rich re-run of § 5.6 , which differs from the corpus run of Table 13 in one cell (a gold-low cell moves from medium to high , which is why weighted κ is 0.08 here and 0.13 there); intervals in Table 9 .
Figure 6: Calibration trade-off: severity reduction versus coverage against the reconciled human gold, for the weak-retrieval rule alone (in-sample; med and unk are the two policies of Table 14 , which also gives the exploratory four-rule set). Demoting weak-retrieval high s to medium preserves coverage (33 answered) while cutting over-severe judgements from 25 to 8; abstaining leaves fewer over-severe cells (5) and a higher weighted κ ( 0.29 ) only by discarding 20 answers, and leaves the remaining labels unchanged. Most of the gain in exact agreement reflects a gold that is 84% medium (§ 5.6 ); the remaining bottleneck is evidence grounding, not severity thresholding alone.
11 flaws
33 variants
cell
det
fa
loc
det
fa
loc
off
K-kw
2/11
0/11
2/11
8/33
0/33
8/33
0.00
K-llm
10/11
1/11
8/11
31/33
3/33
22/33
0.15
S-kw
1/11
0/11
1/11
2/33
0/33
2/33
0.00
S-llm
11/11
0/11
11/11
31/33
0/33
22/33
0.16
gated two-stage
8/11
0/11
8/11
25/33
3/33
0.66
–
Appendix
Table 15: Shared-evidence control: evidence bundle (K, keyword chunks; S, ungated sections) × judgement policy (kw, keyword scorer; llm, adequacy judge). det / fa / loc as in Table 4 ; off = share of the ten non-target dimensions rated ≥ medium on the injected variants. The reference row is the gated pipeline on the gpt-4o alias (detection and false alarm from run 1, Table 8 ; localization is the three-run mean of Table 2 ).
Mechanistic interpretability papers increasingly use causal vocabulary: circuits, mediators, causal abstraction, monosemanticity. Such claims require explicit identification assumptions. A purposive audit of 10 papers across four methodological strands finds no dedicated identification-assumptions section and a recurring pattern: validation metrics such as faithfulness, completeness, monosemanticity, alignment, or ablation effects are reported as causal support without stating the assumptions that make them identifying. A two-human-coder audit on n=30 reproduces the direction of the main finding: dedicated identification sections are absent, and validation-metric substitution is common, though exact Dim B/D counts are coding-rule sensitive. The paper proposes a disclosure norm: state whether the claim is causal, name the identification strategy, enumerate assumptions, stress at least one, and explain how conclusions shift if assumptions fail. Validation is not identification.
Understanding climate change requires reasoning over complex causal networks. Yet, existing causal discovery datasets predominantly capture explicit, direct causal relations. We introduce ClimateCause, a manually expert-annotated dataset of higher-order causal structures from science-for-policy climate reports, including implicit and nested causality. Cause-effect expressions are normalized and disentangled into individual causal relations to facilitate graph construction, with unique annotations for cause-effect correlation, relation type, and spatiotemporal context. We further demonstrate ClimateCause's value for quantifying readability based on the semantic complexity of causal graphs underlying a statement. Finally, large language model benchmarking on correlation inference and causal chain reasoning highlights the latter as a key challenge.
Liesbeth Allein, Nataly Pineda-Castañeda, Andrea Rocci +1
Department of Computer Science, KU Leuven, Belgium · Institute of Argumentation, Linguistics, and Semiotics (IALS), Universit`a della Svizzera italiana, Switzerland
Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification, formulating a valid research design under stated assumptions, and estimation, implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 174 queries across 139 real-world datasets, curated from 73 peer-reviewed research papers and three widely used causal-inference textbooks. For each query, a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 81% of cases, full identification-specification correctness drops to only 38%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.