Web systems infer user preferences from behavior, ratings, text, and LLM-generated profiles, yet explaining a proxy may reveal its measurement process rather than the preference of interest. We develop an audit framework distinguishing point identification, partial identification, and abstention relative to an explicit reference distribution. Under feature-dependent, arbitrarily signed proxy bias, a strict majority of uncontaminated proxy attributions lets the coordinatewise median recover latent-attribution support and direction and, with externally justified positive scales, magnitude. We prove this threshold sharp using response-valid indistinguishable worlds, characterize sharp minimax error and exact compatible-target sets under approximate cleanliness and scale uncertainty, and propagate clean-proxy attribution error. A frozen known-truth benchmark spans two response families, four proxy counts, all contamination counts, three bias strengths, and eleven methods, with a focal comparison confirmed on twenty unseen seeds. Under correct budgets, a direct attribution-tensor audit achieved 100% coverage, 95.40%-97.05% active-driver sign decisions, and no false signs. In a temporally separated Amazon study, median aggregation reduced comparator-relative conditional sign error for all four targets; six of eight prespecified target-metric contrasts had Bonferroni intervals excluding zero. A controlled real-covariate perturbation favored the median on the prespecified focal metric for all four targets. Compatible sets showed that the current proxy configuration rarely licensed strict-sign conclusions. Weak comparator calibration limits the Amazon evidence to fitted-surface alignment rather than latent-preference recovery.
Figures & tables
Figure 1. Controlled strict-majority phase diagram at λ=1 . The vertical axis reports bias leakage: mean absolute attribution on the designated bias coordinate divided by the average mean absolute attribution over the two true driver coordinates. It is shown for the signed local median at every contamination count and K∈{3,4,5,7} in the additive and logistic-link DGPs. Bands are pointwise 95% Student- t intervals over ten exploratory seeds. Leakage remains small when 2m<K and rises under the shared-direction construction when the uniform guarantee ends. The theorem is worst-case: losing the guarantee does not require every beyond-boundary instance to fail. Two panels plot median bias leakage against contamination count for four proxy counts. Leakage is low under a strict clean majority and rises at the first boundary-violating count.
DGP
Median
RPCA
Δ
95% CI
Family-2 CI
Add.
0.115
0.142
-0.0277
[-0.0364, -0.0189]
[-0.0379, -0.0175]
Logit.
0.050
0.058
-0.0079
[-0.0110, -0.0047]
[-0.0116, -0.0042]
Table 1. Controlled confirmation-cohort bias leakage and seed-paired median-minus-RPCA effects at K=5,m=2,λ=1 (20 unseen seeds; lower is better). Family-two intervals apply a reporting-stage Bonferroni correction across the two fixed DGP contrasts. This is the only frozen confirmatory method contrast; the remaining methods belong to the descriptive grid.
b
Declared (r,ϵ)
Coverage
Mean diam.
Empty
Decision/false (%)
0.00
(2,0.010)
100.00%
0.00997
0.00%
95.40/0.00
0.35
(2,0.010)
100.00%
0.01349
0.00%
97.05/0.00
0.00
(2,0.005)
12.49%
0.00377
50.36%
48.85/0.36
0.35
(2,0.005)
23.06%
0.00421
13.62%
85.98/0.00
0.00
(1,0.010)
0.00%
–
100.00%
0.00/0.00
0.35
(1,0.010)
0.00%
–
100.00%
0.00/0.00
Table 2. Direct known-truth compatible-target-set audit at true (r,ϵ)=(2,.01) . b is the shared clean-proxy offset as a fraction of ϵ . Coverage, diameter, and emptiness use all 512×8 case–coordinate cells per seed; decision/false uses the two active drivers. Ten seeds are the replicates. Misspecified rows are descriptive post-primary stress tests; “–” means all sets are empty.
Target
Metric
Behavior
Median
Rel. red.
Effect
95% CI
Bonf. CI
n
W/T/L
Pop
DNAE
0.1396
0.1291
7.6%
-0.0106
[-0.01293, -0.00818]
[-0.01383, -0.00736]
1000
633/0/367
Pop
CSE
0.3413
0.3068
10.1%
-0.0344
[-0.04623, -0.02204]
[-0.05126, -0.01667]
1000
376/404/220
Rock
DNAE
0.1722
0.1709
0.7%
-0.0013
[-0.00431, 0.00175]
[-0.00552, 0.00296]
1000
534/1/465
Rock
CSE
0.4152
0.3611
13.0%
-0.0541
[-0.06770, -0.04014]
[-0.07325, -0.03494]
1000
408/354/238
Jazz
DNAE
0.1521
0.1493
1.8%
-0.0028
[-0.00558, 0.00003]
[-0.00661, 0.00109]
1000
527/5/468
Jazz
CSE
0.3560
0.3199
10.1%
-0.0361
[-0.04831, -0.02368]
[-0.05356, -0.01882]
1000
329/472/199
Table 3. Amazon point-attribution results (lower is better). Panel (a) is the frozen paired K=3 median-versus-behavior comparison; Bonferroni intervals cover its eight cells. Panel (b) lists every frozen point method without cell filtering or new inference. Rows differ in scale or objective, K=4 is sensitivity only, and the primary contrast remains panel (a).
Target
r
Covered
Exact 95% CI
Empty
Sign
Abstain
Elig. sign/coords
Discord./elig. sign
Pop
0
961/1000
[0.947, 0.972]
0.00%
0.53%
99.47%
34/5270
8/34
Pop
1
1000/1000
[0.996, 1.000]
0.00%
0.00%
100.00%
0/5270
–
Rock
0
949/1000
[0.933, 0.962]
0.06%
0.60%
99.34%
34/5231
22/34
Rock
1
1000/1000
[0.996, 1.000]
0.00%
0.00%
100.00%
0/5231
–
Jazz
0
935/1000
[0.918, 0.949]
0.04%
1.41%
98.54%
95/5310
25/95
Jazz
1
990/1000
[0.982, 0.995]
0.00%
0.16%
99.84%
11/5310
1/11
Table 4. RQ3 comparator-relative point-scale compatible-target sets. Coverage is an all-coordinate user event with an exact Clopper–Pearson 95% interval; sign and abstention use all coordinate sets. Discordance uses eligible decisions under the fixed comparator margin; the comparator is not latent truth, and “–” means no eligible decision.
Large language models are increasingly used as proxies for human subjects in social science research, yet external validity requires matching the response distributions of target human populations. We study population-level survey alignment: reconstructing aggregate survey responses from limited public survey data, without individual-level demographic profiles or model finetuning. We formalize this problem as preference reconstruction: rather than matching proxy agents to demographic profiles, we construct a functional basis of proxy agents and recover population preferences by weighted aggregation. We instantiate this idea via Prompts to Proxies (P2P), a two-stage inference-time system. Stage 1 uses structured attribute-based prompting with entropy-guided adaptive sampling to construct a diverse proxy pool spanning the latent preference space. Stage 2 employs L1-regularized regression to select a compact weighted ensemble matching observed target-population responses. Across 14 American Trends Panel waves, P2P achieves an average test MSE of 0.014 at approximately 0.8 USD per survey, improving over prompting and demographic-conditioning baselines. On the World Values Survey, cross-locale transfer experiments show that basis expressiveness can outperform locale matching on average, while locale-specific generation helps on culturally divergent questions. A stress test against an SFT-aligned survey model shows competitive performance using less than 3% of the training data. These results position preference reconstruction as a lightweight, externally verifiable alternative for survey-based population alignment. Code is available at https://github.com/bingchen-wang/P2P.
Bingchen Wang, Zi-Yu Khoo, Jingtan Wang
Independent Researcher, China · AI Singapore, Singapore · School of Computing, National University of Singapore, Singapore
Online A/B testing at scale relies on proxy metrics -- short-term, easily-measured signals used in place of slow-moving long-term outcomes. When the proxy-outcome relationship is heterogeneous across user segments, aggregate correlation can mask directional failures akin to Simpson's Paradox, leading to costly ship/no-ship errors. We introduce PROXIMA (Proxy Metric Validation Framework for Online Experiments), a lightweight diagnostic framework that scores proxy reliability through a composite of three complementary dimensions: normalised effect correlation, directional accuracy, and segment-level fragility rate. Unlike surrogate-index approaches that predict long-term treatment effects, PROXIMA directly audits whether a candidate proxy leads to correct launch decisions and flags the user segments where it fails. We validate PROXIMA on two public datasets -- the Criteo Uplift corpus (14M observations, advertising) and KuaiRec (7K users, video recommendation) -- using 80 simulated A/B tests. Early engagement metrics achieve a composite reliability of 0.80 on Criteo and 0.62 on KuaiRec, yielding 98.4% average decision agreement with an oracle policy. Fragility analysis reveals that recommendation domains exhibit substantially higher segment-level heterogeneity (68% fragility) than advertising (13%), yet directional accuracy remains above 96% in both cases. A sensitivity analysis over the weight space confirms that no single component suffices and that the composite provides substantially better discrimination between reliable and unreliable proxies than correlation alone. Code and reproduction scripts are available at: https://github.com/Avinash-Amudala/PROXIMA
Large language models are increasingly used to judge people, and their answers can depend on inputs correlated with a protected attribute such as race, even when that attribute is never shown. Interpreting this reliance requires knowing how much predictive information those inputs add, but audits often report only whether a decision changes. We introduce an evidence-referenced audit that compares proxy reliance with two references, the known outcome rule and a Bayesian linear regression fitted to the same labeled examples, while measuring ranking accuracy separately. We apply it to four models in a simulated patient-ranking task with known field weights. We find four results. First, at zero predictive value, the models still relied on fields that carried no information, while the same-example learner did not. Second, as proxies became predictive, reliance grew far less than the reference. Third, display format mattered. Ordered labels drew more reliance than arbitrary labels, a direction that held in all four models. Fourth, ranking accuracy did not consistently track these changes in reliance. Based on these findings, we argue that decision changes and accuracy cannot show whether proxy reliance is warranted by the evidence. Audits should report reliance against an explicit predictive reference, alongside accuracy. Ordered fields whose values move with group membership, such as scores or income bands, are a useful starting point. Together, these results make it possible to measure how far reliance departs from the evidence, rather than infer it from observed choices alone.