ProxyGuard: Direct Reliability Inference for Randomized Data Release Mechanisms with Shared Targets

Dipesh Tharu Mahato; Pramod Dhungana

cs.LGAug 19, 2026

ProxyGuard: Direct Reliability Inference for Randomized Data Release Mechanisms with Shared Targets

Authors: Dipesh Tharu Mahato, Pramod Dhungana

Organizations: New York University · Queens College

Abstract

Researchers often choose a proxy dataset from many releases, transformations, or seeds. Search can make an invalid release appear adequate, while one adequate release does not establish that its generator is reliable. ProxyGuard controls both errors using prespecified bounded risks and a sealed target set. Named-release mode corrects for multiplicity and certifies specific releases. Direct shared-target mode evaluates independent mechanism draws on a common target, lower-bounds their favorable-score rate, and subtracts a bound on favorable scores contributed by invalid releases. Conditional on the target, release scores are independent, yielding a finite-sample mechanism-reliability guarantee without independent target batches or assumptions on release-level $p$ -value dependence. We show that the mean-only penalty is sharp and derive a smooth-score certificate with additive target concentration. In a registered three-requirement study, direct mode raises power from $5.6\%$ to $64.2\%$ at reliability $0.95$ , while named mode remains stronger under high-signal evidence. Prospective audits span full-pipeline Rice--TVAE, which retrains on every draw, and a non-tabular text mechanism.

Figures & tables

Panel of Figure 1 — Figure 1: ProxyGuard workflow. Development ends before the target audit opens. Named-release mode identifies particular releases; direct shared-target mode uses favorable-score frequency after correcting for invalid false passes.

Object	Meaning	Observed?	Role
$A(Q)$	true validity of release $Q$	no	defines reliability
$p_{Q}$	evidence against release invalidity	yes	named-release mode
$S_{\theta}(Q,Z)$	favorable score on the shared target	yes	direct shared-target mode
$V_{\theta}^{\mathrm{Holm}}$	individually recognized releases	yes	named mechanism bound
$\eta_{\theta}$	probability that a fresh release is valid	no	mechanism estimand

Table 1: Latent and observed objects. “Recognized,” “scores favorably,” and “truly valid” refer to different events.

Method	$\eta=0.80$	$\eta=0.90$	$\eta=0.95$
Named-release Holm	0.2	4.2	5.6
Direct shared-target	0.0	56.4	64.2
Preregistered hybrid	0.0	2.4	57.2

Table 2: Validation rates (%) over 500 confirmatory repetitions with

R=n=1{,}000

and

\eta_{0}=0.8

. The boundary column measures false validation; the other columns measure power.

Figure 2: Direct minus named power at reliability 0.95 after we planned both methods on the same pilot. Positive cells favor direct shared-target mode; negative cells favor named-release mode.

Dataset	High fidelity	Moderate evidence	Degraded
CDC	both validate (.938/.917)	both validate (.948/.931)	violation ( $U=.280$ )
Covertype	both validate (.938/.910)	both validate (.948/.885)	violation ( $U=.062$ )
Online Shoppers	unresolved (.449/.363)	unresolved (.002/.017)	violation ( $U=.703$ )

Table 3: All nine corrected lightweight-neural configurations. Parentheses give named/direct lower bounds;

U

is the violation-side upper bound.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Mode or direction	Budget	Error event controlled
Named lower bound	$\alpha_{\mathrm{R}}$	false recognition among named releases
Named lower bound	$\alpha_{\mathrm{M}}$	false validation among mechanisms
Direct lower bound	$\alpha_{\mathrm{Q}}$	conditional score-probability coverage
Direct lower bound	$\alpha_{\mathrm{Z}}$	shared-target false-pass contamination
Violation upper bound	$\beta_{\mathrm{R}}$	false release-level violation
Violation upper bound	$\beta_{\mathrm{M}}$	false mechanism-level violation

Appendix

Table 4: Error budgets and the events they control.

Regime	$c_{Q}/c_{Z}$	Mode	$n$	$R$	Projected lower bound
Moderate	0.01	direct	5,000	1,000	0.935
Moderate	0.10	direct	5,000	1,000	0.935
Moderate	1	direct	5,000	1,000	0.935
Moderate	10	direct	5,000	500	0.927
Moderate	100	direct	5,000	30	0.779
High signal	0.01	named	250	1,000	0.969

Appendix

Table 5: Cost-normalized development plans. One target record costs one unit;

c_{Q}/c_{Z}

varies by row.

$c_{Q}/c_{Z}$	Mode switch (%)	Mean regret	Maximum regret
0.01	6.1	0.010	0.161
0.10	6.1	0.010	0.161
1	6.1	0.010	0.161
10	6.1	0.010	0.165
100	8.2	0.013	0.165

Appendix

Table 6: Sensitivity of the development-only planner to misspecified pilot probabilities. Regret is the oracle projected lower bound minus the achieved projection under the fixed true rates.

Audit design	False validations	Rate	95% Wilson interval
Select, then reuse target	2,173/5,000	0.435	[0.421, 0.448]
Select, then seal target	154/5,000	0.031	[0.026, 0.036]
Test full family with Holm	118/5,000	0.024	[0.020, 0.028]

Appendix

Table 7: False validation after adaptive proxy selection. Every candidate is invalid. We froze the registry and seed before the 5,000-trial run.

Figure 3: Controlled release study. Left: the chance of validating at least one of 20 invalid candidates. The dashed line is the requested level. Right: the fraction of valid candidates validated as the audit sample grows.

Held-out regret family	Point FWER	ProxyGuard power
Continuous beta	0.985	0.841
Rare-subgroup mixture	0.987	0.000
Correlated candidates	0.703	0.924

Appendix

Table 8: Held-out regret distributions at

n=1{,}000

. FWER is the chance of validating at least one boundary-invalid candidate. Power is the fraction of valid candidates validated.

Audit $n$	Random rare $n$	Random FWER	Random power	Strat. rare $n$	Strat. FWER	Strat. power
500	10.0	0.000	0.000	250	0.029	0.969
1,000	20.0	0.000	0.000	500	0.049	1.000
2,500	50.0	0.018	0.090	1,000	0.047	1.000

Appendix

Table 9: Registered subgroup-specific audit. FWER is familywise over six boundary candidates; we average power over six valid candidates. Monte Carlo standard errors are at most 0.31 percentage points. Stratified sampling fixes the rare-group labeled count but does not include recruitment or screening cost.

Study block	Purpose	Confirmatory?	Shared target?
Registered simulation	baseline mechanism calibration	yes	no
Initial direct study	direct-mode sanity check	yes	yes
Primary comparison	direct versus named-release inference	yes	yes
False-pass diagnostic	necessity of the contamination term	yes	yes
Smooth concentration	structured target-side correction	yes	yes
Collective grid	independent-batch baseline	yes/amended	no

Appendix

Table 10: Roadmap for the mechanism experiments in this appendix.

Method	False mechanism validation	Power
Plug-in fraction	0.997	1.000
Point rule + binomial	0.059	1.000
Per-release IUT + binomial	0.006	0.996
Two-level ProxyGuard	0.006	0.996
Collective partial-conjunction ProxyGuard	0.006	0.996
Oracle release labels	0.025	1.000

Appendix

Table 11: Mechanism simulation with 50 releases and audit size 500. False mechanism validation is familywise over five mechanisms with

\eta=\eta_{0}=0.8

; power uses

\eta=0.98

Figure 4: Mechanism calibration at audit size 500. The left panel uses boundary mechanisms; the right uses reliability 0.98. A plug-in passing fraction carries no uncertainty correction.

True reliability	Named Holm	Direct shared-target	Oracle labels
$0.80$	$2.2\pm 0.5$	$0/1{,}000$	$4.6\pm 0.7$
$0.90$	$79.9\pm 1.3$	$69.0\pm 1.5$	$1{,}000/1{,}000$
$0.95$	$80.1\pm 1.3$	$\mathbf{89.2\pm 1.0}$	$1{,}000/1{,}000$

Appendix

Table 12: Validation rates (%) over 1,000 repetitions in the initial one-requirement shared-target confirmation. The boundary row measures false validation; the remaining rows measure power. The direct boundary count is shown explicitly; its exact one-sided 95% upper rate is 0.30%.

Method	False validation (%)	Events	Mean lower bound
Score-only, no correction	8.05	161/2,000	0.741
Direct shared-target	0.00	0/2,000	0.000
Oracle release labels	4.00	80/2,000	0.754

Appendix

Table 13: Boundary diagnostic for the direct contamination correction. The score-only row uses the same conditional binomial lower bound as direct mode but omits

\kappa/\alpha_{\mathrm{Z}}

. It is an intentionally invalid diagnostic, not a proposed method.

Method	$\eta=0.80$	$\eta=0.95$	$\eta=0.99$
Hard score + Markov	0.00	0.00	0.00
Ramp score + Markov	0.00	0.00	0.00
Ramp + target concentration	0.00	0.00	86.45
Oracle release labels	4.25	100.00	100.00

Appendix

Table 14: Validation rates (%) in the smooth-score confirmation. The first column is the reliability boundary. Both expectation-only rows use the same error split as the smooth concentration certificate.

Figure 5: Power gained by tail-Simes over named-release Holm with independent target batches. Releases have reliability 0.95. Each cell reports the power difference; $nR$ is the target-observation cost per mechanism.

Mechanism input	Boundary FWER	Power	Mean count	Gain over Holm
Named-release Holm	0.0000 (0.0000)	0.1266 (0.0039)	83.29	0.00
Tail-Simes	0.0005 (0.0005)	0.6501 (0.0055)	90.69	7.40
Tail-Fisher	0.0005 (0.0005)	0.9035 (0.0036)	92.71	9.42

Appendix

Table 15: Collective-method comparison at valid-release risk 0.04,

n=500

R=100

, and power reliability 0.95. Boundary FWER uses

\eta=\eta_{0}=0.8

. We registered tail-Simes; tail-Fisher is a post hoc benchmark.

Bad-release risk	Inner rule	Any false release label	False mechanism validation
0.100	Separate release IUTs	0.9765	0.0237
0.100	Inner Holm	0.0131	0.0170
0.101	Separate release IUTs	0.9520	0.0233
0.101	Inner Holm	0.0113	0.0171
0.102	Separate release IUTs	0.9325	0.0237
0.102	Inner Holm	0.0079	0.0188

Appendix

Table 16: Near-boundary release recognition. “Any false release label” is familywise over all releases from five boundary mechanisms. The last column is familywise over the five outer mechanism decisions.

$\lambda$	$\alpha_{\mathrm{R}}$	$\alpha_{\mathrm{M}}$	False mechanism validation	Power
0.10	0.005	0.045	0.020	0.645
0.25	0.013	0.037	0.019	0.643
0.50	0.025	0.025	0.021	0.602
0.75	0.037	0.012	0.000	0.000
0.90	0.045	0.005	0.000	0.000

Appendix

Table 17: Sensitivity to the release-layer share of the total 0.05 error budget. False validation is familywise over five boundary mechanisms; power uses reliability 0.98.

Reliability target	Mechanisms screened	All-recognized releases needed
80%	1	17
80%	5	24
80%	9	27
90%	1	36
90%	5	51
90%	9	56

Appendix

Table 18: Minimum number of recognized-valid releases needed when Holm recognizes every sampled release as valid.

Rounds	Fixed- $\alpha$ false validation	Spending false validation	Spending power
1	0.034	0.020	1.000
10	0.271	0.030	0.914
25	0.549	0.031	0.813
50	0.798	0.032	0.651

Appendix

Table 19: Adaptive search with fresh target batches. Power is for a valid candidate arriving in the named round under quadratic spending.

Dataset	$\epsilon$	Releases	Release V/U/F	One-sided bounds	Mechanism decision
German	1	5	0/5/0	[0.00, 1.00]	Unresolved
German	5	5	0/5/0	[0.00, 1.00]	Unresolved
German	10	5	0/5/0	[0.00, 1.00]	Unresolved
Heart	1	5	0/5/0	[0.00, 1.00]	Unresolved
Heart	5	5	0/5/0	[0.00, 1.00]	Unresolved
Heart	10	5	0/5/0	[0.00, 1.00]	Unresolved

Appendix

Table 20: Two-level retrospective sensitivity analysis of nine AIM configurations at reliability target

\eta_{0}=0.8

. Release counts are validated/unresolved/violation. The displayed bounds include inner release uncertainty and outer multiplicity.

Mechanism	Releases	Release V/U/F	One-sided bounds	Mean Cost5x regret	Decision
AIM, $\epsilon=1$	24	0/0/24	[0.00, 0.14]	+0.035	Violation

Appendix

Table 21: Informed AIM same-table replication after the pre-outcome amendment. Release-count columns list validated/unresolved/violation. The upper endpoint is a nominal 95% simultaneous one-sided bound; record overlap prevents a theorem-backed prospective interpretation.

Requirement	Form	Limit	Mean	Largest upper bound
Proxy Brier risk	Absolute risk	0.180	0.138	0.157
Proxy Cost5x risk	Absolute risk	0.160	0.113	0.136
Proxy log-loss risk	Absolute risk	0.070	0.032	0.039
Brier transfer	Relative regret	0.040	0.004	0.017
Cost5x transfer	Relative regret	0.040	0.004	0.027
Log-loss transfer	Relative regret	0.040	0.001	0.010

Appendix

Table 22: Informed bootstrap same-table control. “Largest upper bound” is the largest simultaneous release-level upper bound among 30 releases. All six limits came from the pilot, and we fixed them before evaluating the new split.

Reference	Brier	Clipped log loss	Cost5x
Source procedure	0.135	0.031	0.109
Constant 0.5	0.250	0.050	0.156
Training prevalence	0.172	0.038	0.156
Registered ceiling	0.180	0.070	0.160

Appendix

Table 23: Target risks used to interpret the absolute ceilings in the informed positive control. Constant policies choose their decision threshold on validation Cost5x before researchers open the target.

Study	Audit $n$	In pilot train	In pilot validation	In pilot audit
AIM informed rerun	6,000	3,623	1,169	1,208
Bootstrap informed rerun	6,000	3,598	1,184	1,218

Appendix

Table 24: Record lineage for the informed Taiwan reruns. The three overlap columns partition each replication audit set: every audit record had already appeared in the pilot split. The pilot seed was 3609; the AIM and bootstrap rerun seeds were 1155954205 and 4921. Neither rerun has a record-disjoint target reserve.

Experiment	Evidence status	Target lineage	Registered	Guarantee
Release calibration and target-reuse studies	Confirmatory simulation	Fresh simulated audit draws	Yes	Yes
Mechanism calibration and planning studies	Confirmatory simulation	Fresh simulated audit draws	Yes	Yes
Direct and smooth shared-target confirmations	Confirmatory simulations	Fresh simulated targets shared across releases	Yes	Yes
Adaptive candidate stream	Confirmatory simulation	Fresh simulated batch each round	Yes	Yes
Temporal site and schema proxies	Descriptive real-data audit	Public benchmarks used earlier in project	Full-run settings only	No
Repeated AIM releases	Retrospective sensitivity	Existing public benchmark splits and releases	No	No

Appendix

Table 25: Provenance and formal status of every reported experiment. An untouched target is independent, across the project, from choices informed by earlier analyses; a different split seed is not sufficient. The guarantee column is “Yes” only when the design supports the conditions of Assumption 1 or the registered stratified extension in Proposition 11 .

Metric	Rel. mean	Rel. max UCB	Limit	Abs. mean	Abs. max UCB	Limit
Brier	0.0031	0.0158	0.040	0.0973	0.1143	0.141
Clipped log loss	0.0006	0.0100	0.040	0.0232	0.0301	0.072
Cost5x	0.0027	0.0216	0.040	0.0675	0.0871	0.108

Appendix

Table 26: Prospective MAGIC bootstrap audit. Means are across 30 releases; “max UCB” is the largest simultaneous release-level upper bound.

Metric	Rel. limit	Rel. mean	Viol.	Abs. limit	Abs. mean	Viol.
Brier	0.060	0.1467	30/30	0.174	0.2558	30/30
Clipped log loss	0.060	0.0356	0/30	0.080	0.0621	0/30
Cost5x	0.060	0.0592	1/30	0.091	0.0998	1/30

Appendix

Table 27: Prospective Spambase–AIM component audit. Means are across 30 independent full-pipeline releases; “Viol.” counts releases with a corrected requirement-level violation.

Dataset	Configuration	$R$	Named $V/W$	Scores	$L_{\rm named}$	$L_{\rm dir}$
CDC	high	100	100/0	99	0.938	0.917
CDC	moderate	300	295/0	292	0.948	0.931
CDC	degraded	100	0/85	0	0	0
Covertype	high	100	100/0	100	0.938	0.910
Covertype	moderate	300	295/0	288	0.948	0.885
Covertype	degraded	100	0/100	0	0	0

Appendix

Table 28: Corrective stratified analysis of the prospectively collected lightweight-neural audits.

V/W

gives named validations and detected violations; “Scores” counts favorable direct scores.

Component	Frozen implementation
Input map	source-min–max scaling; registered integer/categorical encodings
Encoder	one random $\tanh$ layer; 4–48 hidden features
Training	Gaussian corruption (0.01–0.35); ridge reconstruction solve ( $10^{-3}$ –0.25)
Sampling	balanced labels; class-matched source anchors; Gaussian perturbation; decoder clipping and rounding
Configuration grid	nine settings jointly vary width, corruption, sampling noise, label fidelity, and ridge penalty
Redrawn per release	hidden weights, corruption, anchors, synthetic table, and proxy fit seed

Appendix

Table 29: Architecture of the lightweight neural generator. This is a random-feature denoising autoencoder, not an adversarial or diffusion synthesizer.

Configuration	$R$	Named $V/W$	Scores	$L_{\rm named}$	$L_{\rm dir}$	Decision
High fidelity (200 epochs)	40	40/0	40	0.852	0.864	both validate
Moderate (80 epochs)	120	65/1	67	0.405	0.428	unresolved
Degraded	40	2/34	2	0.001	0	violation ( $U=0.375$ )

Appendix

Table 30: Prospective Rice–TVAE audit.

V/W

gives named validations and detected violations. All bounds are simultaneous one-sided 95% statements over the three registered configurations.

Requirement	Score cutoff	Validity limit	Mean	Range across releases
Brier transfer	0.060	0.100	0.0474	[0.0433, 0.0519]
Absolute Brier	0.162	0.182	0.1467	[0.1427, 0.1513]
Cost5x transfer	0.041	0.081	0.0225	[0.0165, 0.0320]
Absolute Cost5x	0.097	0.117	0.0794	[0.0733, 0.0889]

Appendix

Table 31: Prospective Secondary Mushroom direct audit. The range is over 100 post-processing releases evaluated on the same 10,000 target records.

Proxy relation	$n$	$\Delta$ AUC	Brier	Cost5x	Decision
Bank Marketing temporal	2,620	-0.030	-0.135	-0.317	Rel. validated
Heart Disease site	323	-0.062	+0.014	+0.104	Unresolved
Adult reduced schema	8,141	-0.044	+0.030	+0.012	Violation

Appendix

Table 32: Descriptive audits without row correspondence. AUC is diagnostic; Brier and Cost5x entries are mean proxy-minus-source regrets.

Dataset	$n$	$\Delta$ AUC	Brier	Cost5x	Decision
Heart	54	+0.015	-0.015	-0.011	Unresolved
German	200	-0.035	+0.011	+0.021	Unresolved
Taiwan	6,000	-0.044	+0.037	+0.010	Violation

Appendix

Table 33: Realized-release TabDDPM audit. Brier and Cost5x are mean regrets; AUC remains diagnostic.

Generator	Dataset	Releases	Highest AUC	TPR@1%	Decision counts
AIM	German	3	0.505 (Gen-LRA)	0.023	0/3/0
AIM	Heart	3	0.626 (Gen-LRA)	0.093	0/3/0
AIM	Taiwan	3	0.502 (density-only KDE)	0.008	0/3/0
TabDDPM	German	1	0.533 (Gen-LRA)	0.016	0/1/0
TabDDPM	Heart	1	0.702 (DCR)	0.346	0/1/0
TabDDPM	Taiwan	1	0.510 (Gen-LRA)	0.016	0/1/0

Appendix

Table 34: No-box membership results, aggregated by generator and dataset. We select the highest AUC only for display. The final column counts validated/unresolved/violation decisions for the registered attack claim.

Figure 6: Mean membership AUC over ten registered subsamples. Rows are realized releases; Density denotes density-only KDE. Values near 0.5 indicate little discrimination for that attack, not a general privacy guarantee.

Dataset	Audit $n$	Validated / 3	Remaining state
Adult Income	6,513	1	2 unresolved
Australian Credit	138	0	3 unresolved
Bank Marketing	9,043	2	1 unresolved
Breast Cancer WDBC	114	0	3 unresolved
COMPAS	1,443	0	3 unresolved
German Credit	200	0	3 unresolved

Appendix

Table 35: Risk-controlled transformation results. The audit detects no corrected violation.

Configuration	$R$	Named $V$	$L^{\mathrm{named}}$	$L^{\mathrm{dir}}$	Decision
High signal	250	250	0.9810	0.9743	Both validate
Moderate evidence	500	500	0.9905	0.9869	Both validate
Degraded	250	54	0.1569	0.0701	Unresolved

Appendix

Table 36: Prospective non-tabular text audit at

\eta_{0}=0.95

V

is the number of named recognitions. Bounds are simultaneous over the three registered configurations.

Algorithmic Auditing

Algorithmic auditing evaluates whether automated systems are fair, reliable, robust, compliant, and accountable in their intended settings.

See all 143 papers

PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments
Apr 15, 2026Avinash AmudalaA/b Testing
Rochester Institute of Technology
proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference
Jul 27, 2026Alexandra N. M. Darmon, Deeksha Sinha, Steve Wilkins-Reeves +1Causal Inference Uncertainty Calibration
Meta
Subgroup Membership Inference Audits of Differentially Private Synthetic Text
Sep 9, 2026Yidan Sun, Viktor Schlegel, Srinivasan Nandakumar +2Privacy Auditing Membership Inference Attacks
Imperial College London, Imperial Global Singapore · Nanyang Technological University, Singapore · Imperial College London, United Kingdom

Apr 15, 2026J/K move · Enter open · S save

PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments

Avinash Amudala

Rochester Institute of Technology

A/b Testing

Open paper AskPDF arXiv

Online A/B testing at scale relies on proxy metrics -- short-term, easily-measured signals used in place of slow-moving long-term outcomes. When the proxy-outcome relationship is heterogeneous across user segments, aggregate correlation can mask directional failures akin to Simpson's Paradox, leading to costly ship/no-ship errors. We introduce PROXIMA (Proxy Metric Validation Framework for Online Experiments), a lightweight diagnostic framework that scores proxy reliability through a composite of three complementary dimensions: normalised effect correlation, directional accuracy, and segment-level fragility rate. Unlike surrogate-index approaches that predict long-term treatment effects, PROXIMA directly audits whether a candidate proxy leads to correct launch decisions and flags the user segments where it fails. We validate PROXIMA on two public datasets -- the Criteo Uplift corpus (14M observations, advertising) and KuaiRec (7K users, video recommendation) -- using 80 simulated A/B tests. Early engagement metrics achieve a composite reliability of 0.80 on Criteo and 0.62 on KuaiRec, yielding 98.4% average decision agreement with an oracle policy. Fragility analysis reveals that recommendation domains exhibit substantially higher segment-level heterogeneity (68% fragility) than advertising (13%), yet directional accuracy remains above 96% in both cases. A sensitivity analysis over the weight space confirms that no single component suffices and that the composite provides substantially better discrimination between reliable and unreliable proxies than correlation alone. Code and reproduction scripts are available at: https://github.com/Avinash-Amudala/PROXIMA

Abstract

Algorithmic Auditing

Explore similar work

PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments

proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference

Subgroup Membership Inference Audits of Differentially Private Synthetic Text