Agent Verifiers in Counterfactual Analysis: A Pilot Study Benchmarking Agentic Heterogeneous Treatment Effects
Organizations: Institute for Analytical Sociology, Linköping University, Linköping, Sweden · AI and Global Development Lab · Department of Statistics and Data Sciences, The University of Texas at Austin, Austin, TX, USA · Uppsala University, Uppsala, Sweden ·
Abstract
Agents can expand the covariate space of randomized experiments by retrieving fast-moving local context missed by slow-changing survey baseline covariates, helping characterize who benefits most from an intervention. Verifying such covariates is difficult: individual treatment effects are unobserved, retrieval leakage can pull post-treatment information into nominally pretreatment summaries, and encoder leakage can inject post-treatment pretraining knowledge into representations even after temporal filtering of inputs. Guided by a two-channel information bound separating summary provenance from encoder weights, we propose a three-layer benchmark protocol: (i) retrieval admissibility via pre-treatment date filtering; (ii) heterogeneity validation scored rank-weighted average treatment effects (RATEs); and (iii) encoder admissibility checked with chronologically vintaged encoders, future-fact probes, and weight perturbations. Across 11 datasets anchored by Uganda's Youth Opportunities Program, we compare session-isolated retrieval (sandboxed, bounded-budget agent runs per study) against a persistent workspace teammate (xAI's GrokBot) that builds one date-verified corpus per study while blinded to outcomes. The workspace teammate yields higher mean RATE ratios in 6/11 datasets for text alone and 7 for combined features, though paired tests show neither workflow universally dominates. Even on fixed retrieved text, encoder choice significantly shifts heterogeneity signal. Chronologically vintaged probes show no clear evidence of weight leakage, but single-record adapter perturbation analyses show that memorized future facts, once present, persist until injected noise degrades model output. Temporal filtering of retrieval is necessary but insufficient: verifying agent-generated covariates requires jointly examining source provenance, out-of-sample treatment-effect signal, and encoder weights.
Figures & tables
| Text | Tabular + text | |||||
|---|---|---|---|---|---|---|
| Study | Contrasts | Tabular | Session-isolated retrieval | Workspace teammate | Session-isolated retrieval | Workspace teammate |
| Uganda YOP | ||||||
| Kenya digital ID | ||||||
| Direct air capture | ||||||
| Just transition | ||||||
| COVID-19 boosters | ||||||
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Study | Contrasts | Analysis rows | Places |
|---|---|---|---|
| Uganda YOP | 5 | 1,710–1,715 | 296 |
| Kenya digital ID | 5 | 14,076 | 1,224 |
| Direct air capture | 7 | 29,068 | 494 |
| Just transition | 7 | 9,220 | 407 |
| COVID-19 boosters | 5 | 9,655 | 579 |
| Bangladesh masks | 2 | 595 | 34 |
| Dimension | Session-isolated retrieval | Workspace teammate (GrokBot) |
|---|---|---|
| Compute runtime | Ephemeral, read-only CLI sessions calling a pinned gpt-5.4-mini deployment | Persistent cloud Linux VM on grok.com with browser, PDF-parsing, and Python tools |
| Retrieval topology ( ) | Isolated per-context sessions (16-turn, 24-call budget; repeated draws on six public trials); no cross-context memory | Stateful study-wide pass over the canonical place registry; shared source cache and scripted place mapping |
| Source & archive tooling | Web search, Wayback lookup, and lightweight HTML/short-PDF text reader (no headless JS rendering or bulk PDF table parsing) | Browser rendering, bulk PDF table parsing, Wayback snapshots, PubMed/GitHub APIs, and candidate-URL re-verification |
| Temporal & leakage filter | Prompt window plus ex-post no-post date filter (retains undated rows) and published-trial self-citation regex filter | Ex-ante offline validator: known-pre date-interval bounds and study-outcome keyword exclusion |
| Outcome blinding | Worker inputs omit assignments and outcomes by script construction | Handoff package withholds evaluation outcomes until pre-evaluation corpus freeze |
| Governance trade-off | Pinned model deployment and fixed per-context prompt, but cross-session search noise and undated excerpts | Zero undated items and richer document parsing, but unpinned model weights and shared candidate URLs |
| AUTOC/SE | Qini/SE | |||||
|---|---|---|---|---|---|---|
| Outcome | Tabular | Session-isolated retrieval | Workspace teammate | Tabular | Session-isolated retrieval | Workspace teammate |
| Training hours | ||||||
| Vocational training | ||||||
| Skilled trade | ||||||
| Human capital | ||||||
| Income | ||||||
| Snapshot / condition | MAE | Spearman | Window bal. acc. | AUC | |
|---|---|---|---|---|---|
| 2007 / Clean | |||||
| 2007 / Prefix control | |||||
| 2014 / Clean | |||||
| 2014 / Prefix control |
| Snapshot / vintage / condition | MAE | Spearman | Bal. acc. | AUC | |
|---|---|---|---|---|---|
| 2007 snapshot / 2007 vintage / Clean | |||||
| 2007 snapshot / 2007 vintage / Prefix control | |||||
| 2007 snapshot / 2014 vintage / Clean | |||||
| 2007 snapshot / 2014 vintage / Prefix control | |||||
| 2007 snapshot / 2024 vintage / Clean | |||||
| 2007 snapshot / 2024 vintage / Prefix control |
| Primary linear probe | Endpoint projection | |||||
|---|---|---|---|---|---|---|
| Acc. [CI] | AUC | Acc. [CI] | AUC | |||
| 50.3 [48.8, 51.8] | 0.501 | 49.1 [47.6, 50.7] | 0.482 | |||
| 49.8 [48.3, 51.3] | 0.500 | 52.5 [51.0, 54.0] | 0.529 | |||
| 48.8 [47.3, 50.3] | 0.492 | 54.0 [52.4, 55.5] | 0.556 | |||
| 52.1 [50.6, 53.6] | 0.516 | 59.1 [57.6, 60.6] | 0.625 | |||
| 58.4 [56.9, 59.9] | 0.621 | 68.1 [66.6, 69.5] | 0.742 | |||
| Contrast and representation | AUTOC | CI lower | CI upper | Holm |
|---|---|---|---|---|
| Civic honesty / money vs. no money / wallet reported / area-pooled text | ||||
| Civic honesty / money vs. no money / wallet reported / tabular + area text | ||||
| Direct air capture / no community benefits / profile selected / tabular + area text | ||||
| Direct air capture / enhanced oil recovery storage / profile selected / area-pooled text | ||||
| Kenya digital ID / social protection / profile selected / text | ||||
| Kenya digital ID / social protection / profile selected / tabular + text |
| Text | Tabular + text | ||||
|---|---|---|---|---|---|
| Study / contrast / outcome | Tabular | Session-isolated retrieval | Workspace teammate | Session-isolated retrieval | Workspace teammate |
| Bangladesh masks / mask promotion / proper mask wearing | |||||
| Bangladesh masks / mask promotion / physical distancing | |||||
| COVID-19 boosters / 90% efficacy / profile accepted | |||||
| COVID-19 boosters / two-year duration / profile accepted | |||||
| COVID-19 boosters / variant protection / profile accepted | |||||
| Study / workflow / features | Tabular | Small, 4 PCs | Small (256-d) | Large (256-d) | MPNet (768-d) |
|---|---|---|---|---|---|
| Uganda YOP / session-isolated retrieval / text | |||||
| Uganda YOP / session-isolated retrieval / tabular + text | |||||
| Uganda YOP / workspace teammate / text | |||||
| Uganda YOP / workspace teammate / tabular + text | |||||
| Kenya digital ID / session-isolated retrieval / text | |||||
| Kenya digital ID / session-isolated retrieval / tabular + text |
LLM Agent Verification
LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.
See all 169 papers