StreamLTT: Risk Certification for Policy Selection under Stream Dependence
Organizations: School of ECE, University of Georgia, Athens, GA 30602, USA
Abstract
Learn-then-Test (LTT) provides a general framework for certifying deployment policies against a prescribed risk target, but its standard formulation targets a fixed population risk estimated from calibration losses. In sequential deployment, losses may instead be dependent and nonstationary, so the risk of a fixed policy can evolve with the observed history. We introduce StreamLTT, which replaces this fixed population risk with average predictable risk, defined by conditioning each decision's loss on the information available before its outcome is revealed. Conditional centering turns empirical deviations from this target into a martingale sum, enabling both a test that adapts to predictable variance and a finite-sample Hoeffding alternative. Combined with family-wise error rate control, StreamLTT yields a simultaneously certified policy region that supports subsequent deployment selection. Across seven forecasting datasets and seven deployment risks, StreamLTT places the certified frontier substantially closer to the prescribed risk target than IID, robust-variance, block-bootstrap, and concentration-based alternatives, while empirical family-wise error closely follows the prescribed level. A finite-sample ablation further shows that the predictable-risk formulation drives the tighter risk frontier, while adaptation to predictable variance makes more effective use of the available family-wise error budget. StreamLTT therefore extends risk-certified policy selection to dependent deployment streams by addressing both what risk should be certified and how it should be tested.
Figures & tables
| Risk | Structure | Error event | |
| Interval miss | Pointwise | .10 | |
| Lower-tail miss | Pointwise | .10 | |
| Severe miss | Pointwise | .10 | |
| Task-regime miss | Regime | .05 | |
| Opposite direction | Temporal | .10 | |
| Missed reversal | History | .10 |
| Risk | Method | Electricity | Appliances | Household | PEMS04 | PEMS08 | Beijing | Jena |
| Interval ( ) | IID-LTT | .051/.028 | .046/.013 | .048/.020 | .051/.017 | .044/.015 | .047/.019 | .045/.012 |
| LTT-HB-Holm | .050/.006 | .045/.004 | .047/.009 | .051/.006 | .043/.006 | .046/.005 | .045/.004 | |
| HAC-CLT-Holm | .051/.035 | .046/.014 | .048/.025 | .051/.021 | .044/.019 | .047/.024 | .045/.017 | |
| BlockBootstrap-Holm | .051/.034 | .046/.015 | .048/.025 | .052/.021 | .044/.019 | .047/.023 | .045/.018 | |
| EB-Howard-Bonf. | .050/.000 | .045/.000 | .047/.000 | .050/.000 | .043/.000 | .046/.000 | .044/.000 | |
| StreamLTT-A | .096/.084 | .096/.082 | .097/.102 | .096/.094 | .095/.087 | .096/.089 | .096/.083 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Domain | Forecasting task |
| Electricity | Energy | Load forecasting |
| Appliances | Energy | Energy use forecasting |
| Household Power | Energy | Power forecasting |
| PEMS04 | Traffic | Traffic forecasting |
| PEMS08 | Traffic | Traffic forecasting |
| Beijing | Temperature | Weather forecasting |
| Risk | Method | Electricity | Appliances | Household | PEMS04 | PEMS08 | Beijing | Jena |
| Interval miss ( ) | IID-LTT | .051/.028 | .046/.013 | .048/.020 | .051/.017 | .044/.015 | .047/.019 | .045/.012 |
| LTT-HB-Holm | .050/.006 | .045/.004 | .047/.009 | .051/.006 | .043/.006 | .046/.005 | .045/.004 | |
| HAC-CLT-Holm | .051/.035 | .046/.014 | .048/.025 | .051/.021 | .044/.019 | .047/.024 | .045/.017 | |
| BlockBootstrap-Holm | .051/.034 | .046/.015 | .048/.025 | .052/.021 | .044/.019 | .047/.023 | .045/.018 | |
| EB-Howard-Bonf. | .050/.000 | .045/.000 | .047/.000 | .050/.000 | .043/.000 | .046/.000 | .044/.000 | |
| StreamLTT-A | .096/.084 | .096/.082 | .097/.102 | .096/.094 | .095/.087 | .096/.089 | .096/.083 |
| Risk | Method | Electricity | Appliances | Household | PEMS04 | PEMS08 | Beijing | Jena |
| Interval miss ( ) | IID-LTT | .052/.054 | .046/.022 | .049/.038 | .052/.035 | .044/.032 | .048/.033 | .046/.029 |
| LTT-HB-Holm | .050/.016 | .046/.009 | .048/.016 | .051/.011 | .043/.011 | .046/.011 | .045/.008 | |
| HAC-CLT-Holm | .053/.060 | .047/.029 | .049/.046 | .053/.044 | .045/.041 | .048/.038 | .046/.038 | |
| BlockBootstrap-Holm | .053/.061 | .047/.028 | .049/.046 | .053/.044 | .045/.042 | .048/.038 | .046/.038 | |
| EB-Howard-Bonf. | .050/.000 | .045/.000 | .047/.000 | .050/.000 | .043/.000 | .046/.000 | .044/.000 | |
| StreamLTT-A | .097/.185 | .096/.178 | .097/.211 | .097/.186 | .095/.181 | .097/.177 | .097/.182 |
| Risk | FWER | Frac. | FWER | Frac. | ||
| Interval miss | .092 | .000 | .322 | .093 | .001 | .325 |
| Lower tail miss | .094 | .000 | .699 | .094 | .001 | .704 |
| Severe miss | .070 | .000 | .981 | .070 | .001 | .982 |
| Task regime miss | .042 | .000 | .647 | .043 | .000 | .652 |
| Opposite direction | .086 | .000 | .947 | .086 | .001 | .953 |
Sequential Hypothesis Testing
Sequential hypothesis testing makes decisions as evidence arrives, controlling errors while adapting how much data to collect and when to stop.
See all 66 papers