PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments

Avinash Amudala

stat.MEApr 15, 2026

PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments

Authors: Avinash Amudala

Organizations: Rochester Institute of Technology

Abstract

Online A/B testing at scale relies on proxy metrics -- short-term, easily-measured signals used in place of slow-moving long-term outcomes. When the proxy-outcome relationship is heterogeneous across user segments, aggregate correlation can mask directional failures akin to Simpson's Paradox, leading to costly ship/no-ship errors. We introduce PROXIMA (Proxy Metric Validation Framework for Online Experiments), a lightweight diagnostic framework that scores proxy reliability through a composite of three complementary dimensions: normalised effect correlation, directional accuracy, and segment-level fragility rate. Unlike surrogate-index approaches that predict long-term treatment effects, PROXIMA directly audits whether a candidate proxy leads to correct launch decisions and flags the user segments where it fails. We validate PROXIMA on two public datasets -- the Criteo Uplift corpus (14M observations, advertising) and KuaiRec (7K users, video recommendation) -- using 80 simulated A/B tests. Early engagement metrics achieve a composite reliability of 0.80 on Criteo and 0.62 on KuaiRec, yielding 98.4% average decision agreement with an oracle policy. Fragility analysis reveals that recommendation domains exhibit substantially higher segment-level heterogeneity (68% fragility) than advertising (13%), yet directional accuracy remains above 96% in both cases. A sensitivity analysis over the weight space confirms that no single component suffices and that the composite provides substantially better discrimination between reliable and unreliable proxies than correlation alone. Code and reproduction scripts are available at: https://github.com/Avinash-Amudala/PROXIMA

Figures & tables

Figure 1: High-level architecture of PROXIMA. Historical experiment data is processed by the core engine (composite scoring, fragility detection, decision simulation) and surfaced via an API and dashboard.

Panel of Figure 2 — Figure 2: End-to-end PROXIMA workflows: data processing (a), composite scoring (b), decision simulation (c), and fragility analysis (d).

Proxy	$R$	$C$	DA	FR	$N$
early_starts	0.80 [.77,.83]	0.71	1.00	0.13	50
early_ctr	0.80 [.77,.83]	0.71	1.00	0.13	50
early_watch	0.65 [.62,.68]	0.42	1.00	0.00	50
rebuffer_rate	0.35 [.31,.39]	0.58	0.00	1.00	50

Table 1: Proxy reliability on Criteo (13.9M obs., 50 experiments). 95% bootstrap CI for

R

shown in parentheses.

Proxy	Win	FNR	Regret	Ships
early_starts	1.00	0.00	0.000	50
early_ctr	1.00	0.00	0.000	50
early_watch	1.00	0.00	0.000	50
Oracle	1.00	0.00	0.000	50
rebuffer_rate	0.00	1.00	$-$ 0.003	0

Table 2: Decision quality on Criteo.

Figure 3: Stacked decomposition of the composite reliability score. Each bar shows the contribution of the normalised correlation ( $w_{C}\cdot C$ , blue), directional accuracy ( $w_{\text{DA}}\cdot\text{DA}$ , green), and segment stability ( $w_{\text{FR}}\cdot(1-\text{FR})$ , orange) components.

Proxy	$R$	$C$	DA	FR	$N$
early_starts	0.62 [.59,.66]	0.61	0.97	0.68	30
early_ctr	0.62 [.59,.66]	0.61	0.97	0.68	30
early_watch	0.62 [.58,.66]	0.64	0.83	0.67	30
rebuffer_rate	0.28 [.24,.32]	0.40	0.03	0.82	30

Table 3: Proxy reliability on KuaiRec (7.2K users, 30 experiments). 95% bootstrap CI for

R

shown.

Proxy	Win	FPR	FNR	Regret	Ships
early_starts	0.97	0.03	0.00	0.002	30
early_ctr	0.97	0.03	0.00	0.002	30
early_watch	0.97	0.04	0.14	$-$ 0.005	26
Oracle	1.00	0.00	0.00	0.000	29
rebuffer_rate	0.03	0.00	1.00	$-$ 0.115	0

Table 4: Decision quality on KuaiRec.

Dataset	Size	$R$	$C$	DA	Win	FR
Criteo	13.9M	0.80	0.71	1.00	1.00	0.13
KuaiRec	7.2K	0.62	0.61	0.97	0.97	0.68
Average	–	0.71	0.66	0.98	0.98	0.41

Table 5: Best proxy ( early_starts ) across datasets.

Figure 4: Proxy metric reliability across datasets. Left: composite reliability scores by proxy and dataset. Right: effect correlation versus directional accuracy for each proxy–dataset pair.

Figure 5: Decision simulation results. Left: win rate by proxy metric across datasets. Right: false-positive versus false-negative rate trade-offs. The oracle achieves (0, 0); the pathological proxy rebuffer_rate lies at the extreme.

Method	Criteo	KuaiRec	Avg.
	Win	Win	Win
PROXIMA	1.00	0.97	0.98
Correlation-only	1.00	0.97	0.98
Random (expected)	0.75	0.72	0.74
Oracle	1.00	1.00	1.00

Table 6: Comparison of proxy selection strategies. “Selected” is the proxy each method ranks first; “Win” is the resulting decision win rate.

Weights $(w_{C},w_{\text{DA}},w_{\text{FR}})$	Criteo	KuaiRec	Avg.
(1.0, 0.0, 0.0)	0.71	0.61	0.66
(0.0, 1.0, 0.0)	1.00	0.97	0.98
(0.0, 0.0, 1.0)	0.87	0.32	0.60
(0.5, 0.5, 0.0)	0.86	0.79	0.82
(0.6, 0.2, 0.2)	0.80	0.62	0.71
(0.4, 0.4, 0.2)	0.86	0.70	0.78

Table 7: Sensitivity to weight parameters

(w_{C},w_{\text{DA}},w_{\text{FR}})

. Reliability of the best proxy ( early_starts ) is shown.

Figure 6: Composite reliability $R$ for each proxy under ten weight configurations. Warm colours indicate high reliability. The chosen weights $(0.6,0.2,0.2)$ produce the widest spread between reliable and unreliable proxies.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Proxy versus long-term treatment effects across 50 experiments. Green quadrants indicate directional agreement (correct ship/no-ship decision); red quadrants indicate disagreement. Regression lines show the linear relationship.

Figure 8: Bootstrap distribution of composite reliability $R$ (1,000 resamples over experiments). Vertical dashed lines indicate 95% confidence interval bounds.

Figure 9: Segment-level fragility profiles. Each bar shows the sign-flip rate for a specific region/device/tenure segment. Left: a moderate proxy with localised fragility. Right: a pathological proxy with widespread fragility.

A/b Testing

A/B testing compares product variants through controlled experiments to estimate causal effects and guide decisions.

See all 29 papers

proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference
Jul 27, 2026Alexandra N. M. Darmon, Deeksha Sinha, Steve Wilkins-Reeves +1Causal Inference Uncertainty Calibration
Meta
From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI
Sep 21, 2026Xuanyi Li, Vaskar Nath, Hossein Amirkhani +2A/b Testing Multi-Turn Dialogue Evaluation
Meta Platforms, Inc. Menlo Park, California, USA
Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
Aug 3, 2026Stefan Hut, Lorenzo MasoeroCausal Effect Estimation AI Agent Evaluation
Amazon

Jul 27, 2026J/K move · Enter open · S save

proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference

Alexandra N. M. Darmon, Deeksha Sinha, Steve Wilkins-Reeves, Caner Gocmen

Meta

Causal Inference Uncertainty Calibration

Open paper AskPDF arXiv

Proxy outcomes (such as short-term behavioral signals, model predictions, or surrogate endpoints) are frequently used in place of primary outcomes that are too slow to mature, rare, or challenging to measure directly. But valid inference on a proxy does not guarantee valid inference on the primary estimate as proxy-based estimates can be systematically biased in ways that are difficult to predict, leading to improperly calibrated confidence intervals. We present proxymate, a framework and open-source Python package for proxy validation and adjustment. proxymate organizes into four levels: The Representativity Level (population validity), the Unit Level (measurement quality), the Estimate Level (decision validity), and the Domain Level (cross-domain transportability). Within each level, proxymate provides diagnostic checks, and targeted adjustment strategies that map specific failures to appropriate corrections. At Meta, proxymate has been adopted by many different use cases, spanning experimentation, prevalence estimation, and monitoring use cases, all facing different proxy challenges (limited human review time, long maturation window of outcomes, low detectability) and showcasing the modularity of the framework. Across all products, proxymate assessed and corrected millions of proxy, primary unit comparisons. It has facilitated launches across multiple work streams including enabling quick decision making on thousands of experiments.

Abstract

A/b Testing

Explore similar work

proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference

From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI

Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation