Counterfactual Evaluation of Temporal Observation Protocols

Xizhe Zhang

cs.LGAug 23, 2026

Counterfactual Evaluation of Temporal Observation Protocols

Authors: Xizhe Zhang

Abstract

We study counterfactual protocol evaluation: whether data collected under a realised observation protocol determine the predictive value of alternatives that were never deployed. Protocol value is the population $R^2$ of the Bayes-optimal predictor of a fixed trajectory-level target from the measurements an alternative would collect. We show that even infinite benchmark data need not determine this value: distinct latent covariance structures can induce the same benchmark measurement--target law while assigning different values to the same alternative. We develop a value-specific identification theory in which only latent ambiguity that changes the alternative's value matters. For linear targets, invisible covariance directions certify non-identification, while targeted measurements can restore identification without recovering the full latent covariance; an exact permutation construction extends the result to nonlinear aggregate targets. With finite dense calibration data, uniform error bounds control protocol-selection regret and distinguishable value gaps. Exact marginal gains then support cost-constrained, target-aware observation design. Simulations and retrospective analyses of Sleep-EDF and Long-Term AF show that broad temporal-layout differences can be more reliably distinguished than fine placements selected from finite data. Together, these results connect identification, calibration resolution and observation design for undeployed protocols.

Figures & tables

Figure 1: Four-point instance of theorem 3 . The profiles

\rho_{\pm}=\rho_{0}\pm\varepsilon\delta

are observationally equivalent under

A=\{Z_{0}\}

but assign values 0.6817 and 0.8274 to

B=\{Z_{1},Z_{2}\}

Figure 2: Finite-calibration error, selection regret and candidate-family resolution. (a) Family-uniform protocol-value error

\varepsilon_{m}

for the temporal-mean and occupation-at-zero targets over all 495 four-action protocols under a stationary OU Gaussian model on a

p=128

grid. (b) True regret of the protocol selected from estimated values, together with the

2\varepsilon_{m}

envelope from section 4 , for four-action selection from a 14-action catalogue, averaged over two covariance models and three targets: the temporal mean and occupation above thresholds zero and one. (c,d) Selection regret and family-uniform error for four nested candidate families of increasing granularity, with sizes 2, 5, 75 and 568, under a nonstationary

p=64

covariance. Calibration sizes are

m\in\{25,50,100,250,500,1000\}

. Panels (a,b) use 200 replications and panels (c,d) use 30; curves show replication means and bands show standard errors.

Method	minimum	mean	median
Latent-state mutual information	$0.601$	$0.864$	$0.852$
Integrated posterior variance	$0.802$	$0.947$	$0.987$
Actual-noise linear target	$0.767$	$0.982$	$1.000$
Noiseless kernel quadrature	$0.231$	$0.899$	$0.988$
Target-aware greedy	$0.912$	$0.992$	$1.000$
Target-aware greedy plus one-swap	$0.987$	$0.999$	$1.000$

Table 1: Relative efficiency of alternative design objectives and search procedures against the exhaustive target-specific optimum over 25 kernel–action–target instances.

Figure 3: Held-out prediction from reconstructed Sleep and AF observation protocols. (a) Pooled baseline-adjusted cross-fitted

R^{2}

for the Sleep targets under target-aware, learned kernel-quadrature, uniformly dispersed and contiguous supports. (b) Pooled cross-fitted

R^{2}

for AF burden under dispersed and contiguous window layouts.

Figure 4: Stability of the learned REM support at

N=16

. (a) Target-aware held-out advantage as the support-selection sample increases. (b) Target-aware advantage over learned kernel quadrature and fixed dispersion across 1000 study-stratified 80% subject subsamples.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Latent and calibration model	Actions, family and repetitions
Uniform error, fig. 2 (a)	$T=20$ , $p=128$ , OU $\tau=1$ , $\alpha=0$ ; known calibration-noise variance $0.25$ , action-noise variance $0.5$ ; mean and occupation-at-zero targets	12 equally spaced point actions, exactly four selected ( $495$ protocols); $m\in\{25,50,100,250,500,1000\}$ and $200$ replications.
Selection regret, fig. 2 (b)	$T=20$ , $p=128$ ; trait–OU $(\alpha,\tau)=(0.2,2)$ and two-scale OU $(\tau_{1},\tau_{2},w)=(0.5,6,0.5)$ ; noiseless calibration and action-noise variance $0.5$ ; mean and occupation targets at thresholds zero and one	Points at $\{1,3,\ldots,19\}$ , width-three windows centred at 5 and 15, and pre-specified three-replicate point packages at 2 and 18; exactly four of 14 actions; the same $m$ grid and replication count as panel (a).
Nested protocol classes, fig. 2 (c,d)	$T=20$ , $p=64$ ; noiseless calibration; occupation-at-zero target; action-noise variance $0.4$ ; $K_{jk}=0.15+0.85\exp\{-\|t_{j}-t_{k}\|/[\tau(t_{j})\tau(t_{k})]^{1/2}\}$ , with $\tau(t)$ linear from $0.3$ to $3$	Exactly four actions are selected. L1 compares contiguous with dispersed layouts; L2 adds early, middle, late and full-horizon phases; L3 uses supports on eight coarse bins; L4 adds 495 fine-grid four-point subsets. The cumulative, deduplicated class sizes are 2, 5, 75 and 568; the same $m$ grid and 30 replications.

Appendix

Table 2: Calibration and nested protocol-class configurations.

Target at $N=16$	pooled adjusted	SC adjusted	ST adjusted
REM	$+0.079[-0.010,+0.168]$	$+0.221[-0.010,+0.428]$	$+0.161[-0.445,+0.650]$
N3	$+0.151[+0.058,+0.290]$	$-0.293[-0.430,-0.155]$	$+0.554[+0.286,+0.900]$
Wake	$+0.019[+0.005,+0.040]$	$+0.321[+0.088,+0.529]$	$-0.123[-0.624,+0.890]$

Appendix

Table 3: Dispersed-minus-contiguous cross-fitted

R^{2}

differences and conditional 2.5–97.5 percentile ranges at the middle Sleep budget.

		$N=4$ (4.17%)		$N=16$ (16.67%)
Cohort	records	contiguous	dispersed	contiguous	dispersed
All records	$84$	$+0.696$	$+0.971$	$+0.851$	$+0.998$
$0<\Theta_{\rm AF}<1$	$71$	$+0.596$	$+0.958$	$+0.748$	$+0.998$
$0.05\leq\Theta_{\rm AF}\leq 0.95$	$33$	$+0.180$	$+0.829$	$+0.430$	$+0.990$

Appendix

Table 4: Cross-fitted

R^{2}

in the primary AF cohort and two outcome-defined sensitivity cohorts.

Counterfactual Evaluation

Counterfactual evaluation tests how models, explanations, and policies behave under hypothetical changes to inputs, evidence, actions, or environments.

See all 97 papers

Held-out evidence resolves follow-up measurement decisions in biological screens
Jul 30, 2026Jia Bi, Samuel Pinilla, Chenyang ZhuAdaptive Sampling
Scientific Computing Department, Science and Technology Facilities Council, Rutherford Appleton Laboratory, Didcot, UK · Diamond Light Source, Harwell Science and Innovation Campus, Didcot, UK · School of Electronics and ComputerSouthampton,Science,UK University of Southampton
Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation
Oct 2, 2026Mahsa Mohammadi, Sareh RowlandsBenchmark Auditing Temporal Video Understanding
University of Exeter Exeter, United Kingdom
Counterfactual Action Evaluation, Observation Bottlenecks, and Representation Geometry in Joint-Embedding Predictive World Models
Oct 2, 2026Arjun SubramanianWorld Model Evaluation Action-Conditioned World Models
Computer Science and Artificial Intelligence Laboratory (CSAIL) Massachusetts Institute of Technology

Jul 30, 2026J/K move · Enter open · S save

Held-out evidence resolves follow-up measurement decisions in biological screens

Jia Bi, Samuel Pinilla, Chenyang Zhu

Scientific Computing Department, Science and Technology Facilities Council, Rutherford Appleton Laboratory, Didcot, UK · Diamond Light Source, Harwell Science and Innovation Campus, Didcot, UK · School of Electronics and ComputerSouthampton,Science,UK University of Southampton

Adaptive Sampling

Open paper AskPDF arXiv

Machine learning determines which follow-up measurements biological screens collect. In a six-rule Cell Painting battery, the highest-value rule would re-image 96.01% of the library and had a 97.14% false-activation upper bound, showing why predicted value alone cannot justify replacing a fixed plan. We developed OPAL, a held-out decision test that freezes a rule and judges unnecessary measurement, coverage and value after cost against archive-specific criteria fixed before final evaluation. A development-selected sparse Cell Painting rule had 18.2-fold lower added-well burden, but its false-discovery bound exceeded 35%, so the fixed plan remained. LINCS--LJP favored broad acquisition under point-estimate criteria set during development, not selective saving. CTRP required fallback because its frozen score missed measured opportunity. OPAL separates optimization from evidence sufficient to change an experiment.

Abstract

Counterfactual Evaluation

Explore similar work

Held-out evidence resolves follow-up measurement decisions in biological screens

Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation

Counterfactual Action Evaluation, Observation Bottlenecks, and Representation Geometry in Joint-Embedding Predictive World Models