AI-Generated Measurements for Identification and Inference with Missing Data: A Weak Shadow Variable Approach

Hongyu Chen; David Simchi-Levi; Ruoxuan Xiong

stat.MLFeb 17, 2026

AI-Generated Measurements for Identification and Inference with Missing Data: A Weak Shadow Variable Approach

Authors: Hongyu Chen, David Simchi-Levi, Ruoxuan Xiong

Organizations: Massachusetts Institute of Technology, Cambridge, MA 02139 · Emory University, Atlanta, GA 30322

Abstract

Across business and social science applications, outcomes are often missing in ways that depend on the unobserved outcomes themselves. In service systems, for example, whether a customer submits a rating depends on the rating they would have provided. Such missing-not-at-random (MNAR) mechanisms make population quantities difficult to identify without strong assumptions on the observation process. Meanwhile, rich unstructured data, such as customer interaction histories, are increasingly available and can be used to construct structured measurements using tools such as large language models (LLMs). In this work, we develop an assumption-lean partial identification framework that uses such measurements as weak shadow variables, defined as outcome-informative proxies that are conditionally independent of missingness given the true outcome and observed covariates. Importantly, they need not accurately predict missing outcomes or satisfy the completeness requirement in the classical shadow variable literature. For identification, we characterize sharp bounds on population quantities through a pair of linear programs. For estimation and inference, we propose a localized penalized estimator that remains feasible under sampling error, and a subsampling algorithm for constructing confidence intervals. In semi-synthetic experiments using real customer-service dialogues, weak-shadow-variable intervals are about 89% narrower than those without auxiliary information, while their midpoints have around 41% lower estimation error than classical MNAR methods.

Figures & tables

	$Y=1$	$Y=2$	$Y=3$	$Y=4$	$Y=5$	$Y=6$
$P(Y)$	0.05	0.10	0.20	0.30	0.25	0.10
$P(F=1\mid Y)$	0.05	0.20	0.35	0.50	0.65	0.70
$P(R=1\mid Y)$	0.90	0.60	0.25	0.20	0.55	0.90

Table 1: Simulation setup.

Figure 1: Finite-sample estimation of the lower endpoint. Panel (a) plots RMSE against

n

on log-log scales with an

n^{-1/2}

reference line. Panel (b) reports selection probabilities for the localized tuning index

K

, with population target

K_{*}=10

Figure 2: Lower-endpoint inference. Panel (a) compares the oracle root-

n

law with ordinary bootstrap and subsampling approximations. Panel (b) reports empirical coverage of one-sided

90\%

confidence intervals; colors denote the resampling method and markers distinguish upper and lower confidence bounds.

Rating	1	2	3	4	5
Count	2	144	725	2287	142
Percentage	0.1%	4.4%	22.0%	69.3%	4.3%

Table 2: Distribution of annotator ratings in the JDDC subset of the USS dataset (

n=3{,}300

)

Figure 3: Prompt-level relevance-exclusion frontier. The left panel plots

\mathrm{Spearman}(F,Y)

against the non-rejection rate of the exclusion test for

F\perp\!\!\!\perp R\mid Y

. The right panel plots the same relevance measure against the mean within-

Y

spread in

\mathbb{E}[R\mid F,Y]

. Each point averages over valid

F

-model and

R

-model pairings.

Figure 4: Estimator comparison in the semi-synthetic MNAR experiment. Each faint dot is one structural-cell average over

100

Bernoulli resamples, the large marker is the mean across cells, and whiskers show the interquartile range. Panel (a) compares the midpoint of the proposed interval with point estimators using absolute error; panel (b) compares the widths of the proposed and aggregated-LP identification intervals.

Figure 5: Prompt-design decomposition of identification-interval width. Each faint dot is one structural-cell average for the shadow-variable estimator, the large marker is the mean across cells, and whiskers show the interquartile range. Panel (a) compares prompt perspectives; panel (b) compares task framings and output types, including the binary branch.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel A: Prompt-side model generating $F$
Model	$\mathrm{corr}(F,Y)$	Non-rejection	Propensity gap	Midpoint MAE	Interval width
GPT-4.1-mini	0.323	0.750	0.0326	0.0561	0.1699
GPT-5.4-mini	0.349	0.722	0.0276	0.0582	0.2811
Grok-4-20-Reasoning	0.325	0.778	0.0327	0.0644	0.2457
Panel B: Missingness-side model generating response propensities
DeepSeek-V3.2	—	0.833	0.0462	0.0584	0.2333

Appendix

Table 4: Robustness across prompt-side and missingness-side language models. Midpoint MAE and interval width refer to the localized estimator. A higher non-rejection rate and a smaller propensity gap indicate closer agreement with the exclusion restriction.

Learning with Missing Data

Learning with missing data develops methods to estimate, predict, and make decisions when variables, labels, modalities, or time points are unobserved.

See all 120 papers

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements
Aug 18, 2026Naoki Egami, Sooahn ShinSemiparametric Inference Latent Variable Models
Department of Political Science and Statistics and Data Science Center, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139
Partial Identification with Multiple Nonlinear Measurements of a Latent Regressor
Jul 13, 2026Burhan Ogut, Michelle YinPartial Identification Linear Regression
American Institutes for Research. · Northwestern University, School of Education and Social Policy.
In-Context Learning for the Imputation of Public Opinion Data with Large Language Models
Jun 8, 2026Tobias Holtdirk, Georg Ahnert, Joseph W Sakshaug +1Incomplete Data Imputation Learning with Missing Data
LMU Munich · Munich Center for Machine Learning · University of Mannheim +2

Aug 18, 2026J/K move · Enter open · S save

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

Naoki Egami, Sooahn Shin

Department of Political Science and Statistics and Data Science Center, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139

Semiparametric Inference Latent Variable Models LLM-Assisted Annotation Learning with Noisy Labels

Open paper AskPDF arXiv

An increasing number of scholars use AI to measure variables they subsequently include in downstream analyses. Although AI-measured variables are often analyzed as if observed without error, ignoring prediction errors in automated measurement leads to substantial bias and invalid confidence intervals in downstream analyses, even if AI measurement accuracy is high, e.g., above 90%. Existing solutions, such as design-based supervised learning and prediction-powered inference, combine error-prone AI-based measurements with gold-standard labels, which may be costly and difficult to obtain in some application areas. In this paper, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). Under this assumption, we use semiparametric inference theory to prove that the DMM estimator is consistent and asymptotically normal, enabling valid inference for a wide range of downstream statistical analyses common in the social sciences. Our simulation results show that DMM yields valid inference and that adding accurate, though imperfect, measurements can improve efficiency. Focusing on common applications of large language model annotations, we also develop diagnostics to assess the conditional independence assumption.

Abstract

Learning with Missing Data

Explore similar work

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

Partial Identification with Multiple Nonlinear Measurements of a Latent Regressor

In-Context Learning for the Imputation of Public Opinion Data with Large Language Models