Statistical Inference under Performativity: From Policy Feedback to Self-Evolving Systems
Authors: Tianwei Gao, Garrett G. Wen, Huiying Zhong, Lihua Lei, Weijie Su, Zhun Deng
Organizations: University of North Carolina at Chapel Hill · Yale University · Massachusetts Institute of Technology · Stanford University · University of Pennsylvania
Performativity arises when prediction-informed decisions change the data-generating process, creating feedback between deployment and learning. This phenomenon occurs in policy-making and in self-evolving AI systems that shape their own future training data. Statistical inference is challenging because estimation errors in earlier predictors affect subsequent data distributions and propagate into later updates. We develop an end-to-end framework for statistical inference under performativity. For repeated risk minimization and repeated gradient descent, we establish central limit theorems at fixed deployment rounds and derive covariance recursions that capture both current-round sampling variation and uncertainty inherited from earlier deployments. We develop consistent covariance estimators using policy perturbations and score matching. These results can yield asymptotically valid confidence regions for population iterates and the performative stable point. We further extend prediction-powered inference to this dynamic setting, combining labeled observations with predictions on additional unlabeled data to improve statistical efficiency. We evaluate our framework in synthetic, semi-synthetic educational, and self-evolving large language model routing settings. Our experiments demonstrate that ignoring propagated uncertainty can lead to substantial undercoverage even after repeated retraining. Together, these results provide statistical tools for learning systems that help determine the distributions from which they learn. Our work establish a complete statistical inference under performativity, introducing new challenges and inference settings that we believe will provide substantial value to policy-making, statistics, and machine learning.
Figures & tables
Method
Reporting round
Coverage of θt (%)
Coverage of θPS (%)
Trajectory interval width
Stable-point interval width
One-step ERM
1
94.65
81.98
0.026
0.026
Per-round M-estimator
6
94.78
95.23
0.025
0.026
RRM
6
94.78
95.24
0.025
0.026
RGD
78
94.99
97.09
0.016
0.017
Table 1: Empirical coverage and mean widths of nominal 95% confidence intervals for the population iterate θt and performative stable point θPS in the pure-synthetic experiment.
Figure 1: Pure-Synthetic Experiment. Left Panel: Empirical coverage of θt across rounds; Right Panel: Empirical coverage of performative stable point θPS across rounds. Vertical dashed line implies the fixed reporting round of RRM selected by Bt<10−3 .
Figure 2: Pure-Synthetic Controlled Experiment with Strong Feedback. Left Panel: Empirical coverage of θt across rounds; Right Panel: Empirical coverage of performative stable point θPS across rounds. Vertical dashed line implies the fixed reporting round of RRM selected by Bt<10−3 .
Method
Reporting round
Coverage of θt (%)
Coverage of θPS (%)
Trajectory interval width
Stable-point interval width
One-step ERM
1
95.04
4.43
0.012
0.012
Per-round M-estimator
20
88.77
92.11
0.011
0.013
RRM
20
94.67
96.60
0.014
0.015
RGD
39
94.90
98.14
0.010
0.012
Table 2: Empirical coverage and mean widths of interval in controlled synthetic experiment.
Method
Reporting round
Coverage of θt (%)
Coverage of θPS (%)
Trajectory region diameter
Stable-point region diameter
One-step ERM
1
95.16
80.35
0.046
0.046
Per-round M-estimator
24
94.82
96.42
0.046
0.048
RRM
24
95.05
96.58
0.046
0.048
RGD
31
95.02
96.61
0.043
0.045
Table 3: Empirical coverage and mean Euclidean diameters of nominal 95% confidence regions for the population iterate θt and performative stable point θPS in the semi-synthetic experiment.
Figure 3: Semi-Synthetic Experiment. Left Panel: Empirical coverage of θt across rounds; Right Panel: Empirical coverage of performative stable point θPS across rounds. Vertical dashed line implies the fixed reporting round of RRM selected by Bt<10−3 .
Figure 4: LLM routing with Best-of- 5 selection and n=10000 . Left: empirical coverage of θt ; right: empirical coverage of θPS . Both methods are shown through round 198 . The vertical dashed line marks the fixed reporting round for RRM 55 , selected by Bt<10−3 . Horizontal dashed lines indicate nominal 95% coverage; shaded bands are pointwise 95% exact binomial Monte Carlo confidence intervals.
Method
Reporting round
Coverage of θt (%)
Coverage of θPS (%)
Trajectory region diameter
Stable-point region diameter
One-step ERM
1
95.25
84.86
0.015
0.015
Per-round M-estimator
55
94.71
98.60
0.015
0.016
RRM
55
94.74
98.61
0.015
0.016
RGD
198
94.75
98.95
0.014
0.016
Table 4: Empirical coverage and mean Euclidean diameters of nominal 95% confidence regions in the LLM routing experiment.
Question subset
Slow-mode routing probability (%)
Mean routing deviation (%)
Initial utility
Stable-point router utility
All questions
52.99
3.00
0.483
0.500
GSM8K
56.46
6.46
0.413
0.452
CommonsenseQA
50.21
0.22
0.539
0.539
Table 5: Routing deviation and utility in the LLM routing experiment, evaluated at the population performative stable point. Routing deviation is the mean total variation distance from the initial equal-mixture policy. Utility is the expected value of answer correctness minus a token-cost penalty of 0.1 per 1,000 generated tokens, evaluated for one routed response-bank entry.
Dataset
Mean fast utility
Mean slow utility
Single-draw advantage
Best-of- 5 advantage
GSM8K
0.1125
0.7128
0.6003
0.6535
CommonsenseQA
0.5074
0.5709
0.0635
0.0186
Table 6: Utility comparison. The single-draw advantage is mean slow utility minus mean fast utility. The Best-of- 5 advantage is the expected difference between the best slow and fast candidate utilities under the initial equal-mixture policy, conditional on both modes appearing.
Setting
Score
Round
Mean population Jacobian norm
Jacobian RMSE
Per-round M-estimator coverage of θPS (%)
Synthetic ( κ=1 )
Basis
6
0.0079
0.0061
95.23
Synthetic ( κ=80 )
Basis
20
0.5719
0.0329
92.11
Semi-synthetic
Analytic
24
0.1312
0.0441
96.42
LLM routing
Analytic
55
0.0661
0.0105
98.60
Table 7: Feedback strength, Jacobian estimation error, and coverage of the performative stable point by the per-round M-estimator, using 10,000 Monte Carlo repetitions.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
N/n
Population λ⋆
Mean λ
RMSE of λ
1
0.107863
0.107762
0.011998
2
0.143818
0.143778
0.015899
5
0.179772
0.179700
0.019544
10
0.196115
0.196447
0.021505
20
0.205454
0.205390
0.022777
Appendix
Table 8: Population-optimal and empirically selected PPI weights in the pure-synthetic experiment, with n=2000 and fixed round t=6 . Means and RMSEs use 10,000 Monte Carlo repetitions. For each N/n , λ⋆ is the common reference evaluated at the deterministic population deployment for round 6.
Figure 5: Prediction-powered inference in the pure-synthetic experiment. From left to right: mean stable-point interval width, mean selected λ , and stable-point coverage as N/n varies. The gray dashed curve shows the population-optimal λ⋆ . The horizontal dashed line marks nominal 95% coverage; shaded bands are pointwise exact 95% binomial Monte Carlo confidence intervals.
Score method
m
Score RMSE
Coverage of θt (%)
Coverage of θPS (%)
Exact
–
0.000
94.56
96.59
Zero
–
14.268
88.77
92.11
Basis
1,000
3.568
94.49
96.56
Basis
10,000
1.122
94.67
96.63
Basis
100,000
0.359
94.67
96.60
Misspecified basis
1,000
16.657
93.69
95.96
Appendix
Table 9: Score estimation and coverage for RRM in the κ=80 strong feedback experiment, with n=10,000 , η=0.01 , and 10,000 Monte Carlo repetitions, evaluated at t=20 . Each fitted score uses m independent auxiliary observations per distribution, totaling 2m . Score RMSE is the square root of the average integrated squared error across repetitions. Trajectory coverage uses nominal 95% Wald intervals; stable-point intervals are enlarged at each endpoint by Bt=6ρt , with the round selected by Bt<10−3 .
Score method
m
Score RMSE
Coverage of θt (%)
Coverage of θPS (%)
Exact
–
0.000
94.96
98.13
Zero
–
14.342
83.96
92.59
Basis
1,000
3.518
95.01
97.92
Basis
10,000
1.106
94.96
98.14
Basis
100,000
0.346
94.90
98.14
Misspecified basis
1,000
16.714
93.22
97.13
Appendix
Table 10: Score estimation and coverage for RGD in the κ=80 strong feedback experiment, with n=10,000 , η=0.01 , and 10,000 Monte Carlo repetitions, evaluated at t=39 . Each fitted score uses m independent auxiliary observations per distribution, totaling 2m . Score RMSE is the square root of the average integrated squared error across repetitions. Trajectory coverage uses nominal 95% Wald intervals; stable-point intervals are enlarged at each endpoint by Bt=6ρt , with the round selected by Bt<10−3 .
Figure 6: Coverage under different score estimators. RRM and RGD are evaluated at fixed rounds 20 and 39, respectively. Filled circles show trajectory coverage using ordinary Wald intervals; open squares show stable-point coverage after bias enlargement. Error bars are pointwise 95% Wilson Monte Carlo intervals. The dashed line marks nominal 95% coverage.
Performative prediction studies feedback loops that arise when predictive models are deployed in consequential domains. In these settings, deploying a model can change the population whose patterns the model aims to predict, inducing a distribution shift that is endogenous to the learning system. This perspective departs from classical treatments of distribution shift, where shifts are typically modeled as exogenous changes in the data-generating process. Yet, in practice, distribution shift is rarely one or the other. Predictive models may influence future data through the decisions they support, while the world itself continues to drift for reasons beyond the learner's control. We study partially performative prediction, a framework that captures both endogenous and exogenous sources of distribution shift. The framework generalizes performative prediction by allowing the data distribution to evolve both in response to the deployed model and according to an external, time-varying process. We extend the central notions of performative stability and performative optimality to this setting by defining their online analogues that track the evolving partially performative environment. We analyze practical learning heuristics, including repeated retraining, and characterize when they successfully adapt to partially performative environments.
Jaewook Lee, Tijana Zrnic
Department of Electrical Engineering, Stanford University · Departments of Statistics and Management Science & Engineering, Stanford University
When algorithmic predictions inform people's decisions, the models we deploy are performative and actively shape the data we see. This feedback loop between algorithms and their broader environments introduces a challenge in the mechanics of social prediction: If different predictive models induce different distributions, is it possible to efficiently learn a prediction rule that is optimal for the distribution that it induces? Formally, this solution concept is known as performative stability. A core challenge in learning a performatively stable predictor is that, unlike supervised learning where distributions are fixed, the learner must deploy different predictors and observe their induced distributions. The main contribution of our work is a new algorithmic procedure that, in the high-accuracy regime, finds a performatively stable model in nearly the minimum number of model deployments without making any assumptions regarding how predictions shape distributions. In particular, our procedure succeeds at finding a randomized performatively stable predictor using exponentially fewer model deployments than prior approaches. Our second main contribution is a structural result showing how this recent randomized notion of stability achieved by our algorithm can be derandomized into a single predictor satisfying the prior deterministic notion if one is willing to assume that the loss is well-conditioned and that performative effects are weak, as in early work in this area. On a technical level, our results come from building on an underexplored technical connection between performative stability and expected variational inequalities.
U-statistics play a central role in statistical inference. In many modern applications, however, acquiring the labels required for U-statistics is costly. Motivated by recent advances in active inference, we develop an active inference framework for U-statistics that selectively queries informative labels to improve estimation efficiency under a fixed labeling budget, while preserving valid statistical inference. Our approach is built on the augmented inverse probability weighting U-statistic, which is designed to incorporate the sampling rule and machine learning predictions. We characterize the optimal sampling rule that minimizes its variance and design practical sampling strategies. We further extend the framework to U-statistic-based empirical risk minimization. Experiments on real datasets demonstrate substantial gains in estimation efficiency over baseline methods, while maintaining target coverage.
Xiaoning Wang, Yuyang Huo, Liuhua Peng +1
School of Statistics and Data Sciences, LPMC, KLM-DASR and LEBPS, Nankai University, Tianjin, China · School of Mathematics and Statistics, The University of Melbourne, Melbourne, Australia