The Measurement Revolution? Credible Measurement and Inference in the Age of AI

Melissa Dell; Ashesh Rambachan

econ.GNAug 24, 2026

The Measurement Revolution? Credible Measurement and Inference in the Age of AI

Authors: Melissa Dell, Ashesh Rambachan

Organizations: National Bureau of Economic Research

Abstract

Artificial intelligence (AI) is transforming measurement in economics. AI models convert unstructured data, such as text and images, into structured variables at low cost, making previously prohibitive measurement feasible at scale. This shifts the bottleneck from finding any scalable measure of a phenomenon to choosing among many plausible ones, which may support different empirical conclusions. This review provides guidance for navigating that shift. We describe three stages at which AI enters the measurement pipeline---discovery, construct definition, and observation---and what each demands of researchers. We argue that credible inference with AI-generated variables requires appropriately designed validation: anchoring measurement to explicit criteria, rather than informal claims that a proxy is reasonable. We then examine how validation samples support valid inference even when AI predictions are arbitrarily biased, and what can be done when a random validation sample is unavailable.

Figures & tables

Panel of Figure 1 — Figure 1: The measurement pipeline. Measurement proceeds in three stages: discovery, construct definition, and observation. AI is transforming each stage, creating both opportunities and challenges for empirical research.

Prediction-Powered Inference

Prediction-Powered Inference uses model predictions and a smaller set of trusted labels to make efficient estimates with statistically valid uncertainty.

See all 30 papers

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements
Aug 18, 2026Naoki Egami, Sooahn ShinSemiparametric Inference Latent Variable Models
Department of Political Science and Statistics and Data Science Center, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139
Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components
May 23, 2026Ling Zhan, Xiaoyao Yu, Tao JiaAI for Science Uncertainty Quantification
College of Computer and Information Science, Southwest University, Chongqing, China · Chongqing Key Laboratory of Brain-Inspired Cognitive Computing and Educational Rehabilitation for Children with Special Needs, Chongqing Normal University, Chongqing, China · College of Computer and Information Science, Chongqing Normal University, Chongqing, China
Generative Augmented Inference
Apr 16, 2026Cheng Lu, Mengxin Wang, Dennis J. Zhang +1LLM-Assisted Annotation Causal Inference
Olin Business School, Washington University in St. Louis, USA · Naveen Jindal School of Management, University of Texas at Dallas, USA · W. P. Carey School of Business, Arizona State University, USA

Aug 18, 2026J/K move · Enter open · S save

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

Naoki Egami, Sooahn Shin

Department of Political Science and Statistics and Data Science Center, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139

Semiparametric Inference Latent Variable Models LLM-Assisted Annotation Learning with Noisy Labels

Open paper AskPDF arXiv

An increasing number of scholars use AI to measure variables they subsequently include in downstream analyses. Although AI-measured variables are often analyzed as if observed without error, ignoring prediction errors in automated measurement leads to substantial bias and invalid confidence intervals in downstream analyses, even if AI measurement accuracy is high, e.g., above 90%. Existing solutions, such as design-based supervised learning and prediction-powered inference, combine error-prone AI-based measurements with gold-standard labels, which may be costly and difficult to obtain in some application areas. In this paper, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). Under this assumption, we use semiparametric inference theory to prove that the DMM estimator is consistent and asymptotically normal, enabling valid inference for a wide range of downstream statistical analyses common in the social sciences. Our simulation results show that DMM yields valid inference and that adding accurate, though imperfect, measurements can improve efficiency. Focusing on common applications of large language model annotations, we also develop diagnostics to assess the conditional independence assumption.

Abstract

Prediction-Powered Inference

Explore similar work

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components

Generative Augmented Inference