State-Matched Sequence Conformal Prediction
Organizations: Department of Mathematics and Statistics, Georgia State University, Atlanta, GA 30303
Abstract
Sequential conformal prediction needs to decide which past errors should calibrate uncertainty for the current target. Recency and online error feedback are useful when recent observations remain relevant, but the recent past can be a poor reference when comparable states recur at nonadjacent times, with the recurrence pattern generally unknown. We propose State-Matched Sequence Conformal Prediction (SMSCP), which constructs an explicit one-sided local-linear state from pre-target information, uses state similarity to select a matched reference set of past calibration scores, and then applies the ordinary unweighted conformal order statistic. In contrast to existing approaches, SMSCP uses state similarity to identify calibration observations relevant to the target score quantile. This motivates a theoretical analysis of whether nearby states have comparable target score quantiles under temporal dependence. We establish control of the resulting conformal cutoff and characterize predictive information not captured by the chosen state. Simulations show that state matching is most effective when score-relevant states recur across nonadjacent times, while revealing settings in which its advantage weakens or reverses. Across solar irradiance and bike-demand benchmarks, SMSCP improves interval score over recency, weighted, adaptive, and HopCPT comparators, and achieves coverage closer to the nominal 0.90 level than ResCP on five of six benchmarks.
Figures & tables
| SMSCP absolute metrics | versus SMSCP | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Design | Size | Cov. | Width | IS | Rolling | Recent- | Tuned-Recent | Tuned-NexCP | ACI-tuned | ECI |
| HomoSin | S | 0.928 | 2.727 | 2.946 | +0.201 | +0.339 | +0.217 | +0.203 | +0.212 | +0.611 |
| HomoSin | L | 0.933 | 2.625 | 2.817 | +0.315 | +0.414 | +0.317 | +0.317 | +0.319 | +0.527 |
| DualFreq | S | 0.930 | 3.312 | 3.555 | +0.332 | +0.486 | +0.359 | +0.337 | +0.342 | +0.680 |
| DualFreq | L | 0.939 | 3.160 | 3.340 | +0.573 | +0.718 | +0.573 | +0.575 | +0.575 | +0.770 |
| HeteroSin | S | 0.905 | 3.568 | 4.206 | -0.019 | +0.147 | +0.006 | -0.015 | +0.005 | +0.366 |
| SMSCP absolute metrics | Non-neural | HopCPT | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Station | Cov. | Width | IS | Rolling | Recent- | Tuned-Recent | Tuned-NexCP | ACI-tuned | ECI | Median | Mean |
| Alachua | 0.910 | 264.0 | 361.0 | +32.7 | +33.4 | +32.7 | +32.7 | +30.4 | +30.0 | +30.9 | +130.6 |
| Phoenix | 0.920 | 188.2 | 252.8 | +31.5 | +31.5 | +31.5 | +29.9 | +30.4 | +23.4 | +21.1 | +145.8 |
| Seattle | 0.914 | 208.4 | 286.2 | +32.4 | +27.3 | +27.0 | +22.9 | +19.9 | +19.1 | +23.0 | +29.2 |
| Houston | 0.914 | 249.3 | 332.9 | +30.2 | +31.7 | +29.5 | +30.2 | +28.7 | +28.5 | +24.5 | +24.8 |
| Minneapolis | 0.909 | 214.3 | 295.6 | +22.1 | +19.7 | +19.6 | +17.5 | +16.8 | +13.1 | +15.3 | +45.2 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Design | Method | Coverage | Mean width | Mean |
|---|---|---|---|---|
| HomoSin | SMSCP | 0.928 | 2.727 | 2.946 |
| Rolling | 0.901 | 2.823 | 3.147 | |
| Recent-k | 0.921 | 3.010 | 3.286 | |
| Tuned-Recent | 0.903 | 2.843 | 3.163 | |
| Tuned-NexCP | 0.904 | 2.836 | 3.149 | |
| ACI-tuned | 0.902 | 2.834 | 3.159 |
| Design | Method | Coverage | Mean width | Mean |
|---|---|---|---|---|
| HomoSin | SMSCP | 0.933 | 2.625 | 2.817 |
| Rolling | 0.901 | 2.807 | 3.132 | |
| Recent-k | 0.920 | 2.966 | 3.231 | |
| Tuned-Recent | 0.901 | 2.809 | 3.133 | |
| Tuned-NexCP | 0.902 | 2.815 | 3.134 | |
| ACI-tuned | 0.902 | 2.813 | 3.135 |
| Design | Size | Rolling | Recent- | Tuned-Recent | Tuned-NexCP | ACI-tuned | ECI |
|---|---|---|---|---|---|---|---|
| HomoSin | S | +0.201 [+0.188,+0.213] | +0.339 [+0.325,+0.354] | +0.217 [+0.204,+0.230] | +0.203 [+0.191,+0.215] | +0.212 [+0.200,+0.225] | +0.611 [+0.574,+0.649] |
| HomoSin | L | +0.315 [+0.308,+0.323] | +0.414 [+0.406,+0.422] | +0.317 [+0.309,+0.324] | +0.317 [+0.310,+0.324] | +0.319 [+0.311,+0.326] | +0.527 [+0.519,+0.535] |
| DualFreq | S | +0.332 [+0.315,+0.348] | +0.486 [+0.468,+0.504] | +0.359 [+0.342,+0.377] | +0.337 [+0.320,+0.353] | +0.342 [+0.325,+0.359] | +0.680 [+0.636,+0.724] |
| DualFreq | L | +0.573 [+0.565,+0.581] | +0.718 [+0.710,+0.726] | +0.573 [+0.565,+0.581] | +0.575 [+0.567,+0.584] | +0.575 [+0.567,+0.583] | +0.770 [+0.762,+0.779] |
| HeteroSin | S | -0.019 [-0.034,-0.003] | +0.147 [+0.128,+0.166] | +0.006 [-0.011,+0.023] | -0.015 [-0.031,+0.000] | +0.005 [-0.011,+0.022] | +0.366 [+0.330,+0.402] |
| HeteroSin | L | +0.129 [+0.121,+0.137] | +0.183 [+0.173,+0.192] | +0.131 [+0.124,+0.139] | +0.131 [+0.124,+0.139] | +0.136 [+0.128,+0.143] | +0.434 [+0.423,+0.444] |
| SMSCP | HopCPT | [95% CI] | ||||||
|---|---|---|---|---|---|---|---|---|
| Design | Size | Cov. | Width | IS | Cov. | Width | IS | |
| HomoSin | S | 0.928 | 2.704 | 2.919 | 0.886 | 2.786 | 3.218 | +0.299 [+0.225,+0.372] |
| HomoSin | L | 0.933 | 2.613 | 2.807 | 0.892 | 2.782 | 3.289 | +0.482 [+0.266,+0.699] |
| DualFreq | S | 0.927 | 3.297 | 3.549 | 0.899 | 3.497 | 4.013 | +0.464 [+0.296,+0.632] |
| DualFreq | L | 0.939 | 3.157 | 3.341 | 0.891 | 3.512 | 4.268 | +0.927 [+0.463,+1.391] |
| HeteroSin | S | 0.901 | 3.548 | 4.238 | 0.880 | 3.419 | 4.341 | +0.103 [-0.023,+0.228] |
| Design | Method | Coverage | Mean width | Mean |
|---|---|---|---|---|
| AR1 | SMSCP | 0.899 | 3.524 | 4.460 |
| Rolling | 0.900 | 3.520 | 4.447 | |
| Recent-k | 0.901 | 3.553 | 4.469 | |
| Tuned-Recent | 0.900 | 3.535 | 4.458 | |
| Tuned-NexCP | 0.902 | 3.551 | 4.450 | |
| ACI-tuned | 0.901 | 3.550 | 4.463 |
| Station | Method | Coverage | Mean width | Mean | |
|---|---|---|---|---|---|
| Alachua | SMSCP | 0.910 | 264.0 | 361.0 | 22887 |
| Rolling | 0.900 | 266.9 | 393.7 | 22887 | |
| Recent-k | 0.907 | 282.7 | 394.4 | 22887 | |
| Tuned-Recent | 0.900 | 266.9 | 393.7 | 22887 | |
| Tuned-NexCP | 0.900 | 266.9 | 393.7 | 22887 | |
| ACI-tuned | 0.901 | 272.2 | 391.4 | 22887 |
| Station | Rolling | Recent- | Tuned-Recent | Tuned-NexCP | ACI-tuned | ECI |
|---|---|---|---|---|---|---|
| Alachua | +32.7 [+28.1,+37.2] | +33.4 [+28.8,+38.3] | +32.7 [+28.5,+36.9] | +32.7 [+28.4,+36.8] | +30.4 [+26.0,+35.0] | +30.0 [+25.0,+34.8] |
| Phoenix | +31.5 [+28.7,+34.1] | +31.5 [+28.4,+34.6] | +31.5 [+28.5,+34.4] | +29.9 [+27.2,+32.5] | +30.4 [+27.1,+33.4] | +23.4 [+21.0,+25.7] |
| Seattle | +32.4 [+28.0,+37.8] | +27.3 [+21.6,+33.2] | +27.0 [+22.3,+32.2] | +22.9 [+17.9,+27.8] | +19.9 [+15.1,+24.9] | +19.1 [+14.4,+23.4] |
| Houston | +30.2 [+26.0,+34.8] | +31.7 [+27.5,+36.5] | +29.5 [+25.6,+34.1] | +30.2 [+26.2,+34.6] | +28.7 [+24.9,+32.8] | +28.5 [+24.0,+32.9] |
| Minneapolis | +22.1 [+19.1,+25.0] | +19.7 [+16.2,+23.0] | +19.6 [+16.1,+22.8] | +17.5 [+14.1,+20.7] | +16.8 [+13.5,+20.0] | +13.1 [+9.9,+16.3] |
| SMSCP | HopCPT across initializations | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Station | Cov. | Width | IS | Mean cov. | Median cov. | Mean width | Median width | Mean IS | Median IS |
| Alachua | 0.910 | 264.0 | 361.0 | 0.844 | 0.898 | 244.8 | 261.0 | 491.6 | 391.9 |
| Phoenix | 0.920 | 188.2 | 252.8 | 0.849 | 0.901 | 196.0 | 205.9 | 398.6 | 273.9 |
| Seattle | 0.914 | 208.4 | 286.2 | 0.881 | 0.895 | 206.4 | 205.8 | 315.4 | 309.2 |
| Houston | 0.914 | 249.3 | 332.9 | 0.897 | 0.897 | 239.2 | 239.1 | 357.7 | 357.4 |
| Minneapolis | 0.909 | 214.3 | 295.6 | 0.867 | 0.894 | 200.2 | 206.1 | 340.8 | 310.9 |
| SMSCP absolute metrics | versus SMSCP | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Cov. | Width | IS | Rolling | Recent- | Tuned-Recent | Tuned-NexCP | ACI-tuned | ECI | HopCPT |
| 0.894 | 326.4 | 419.6 | +315.4 | +284.7 | +296.2 | +276.3 | +286.9 | +199.5 | +48.2 |
| Method | Coverage | Mean width | Mean |
|---|---|---|---|
| SMSCP | 0.894 | 326.4 | 419.6 |
| Rolling | 0.868 | 397.3 | 734.9 |
| Recent- | 0.907 | 502.4 | 704.3 |
| Tuned-Recent | 0.899 | 494.3 | 715.8 |
| Tuned-NexCP | 0.912 | 507.4 | 695.9 |
| ACI-tuned | 0.906 | 501.9 | 706.4 |
| Comparator | Coverage [95% CI] | Width [95% CI] | [95% CI] |
|---|---|---|---|
| Rolling | -0.0261 [-0.0368,-0.0175] | +70.8 [+44.3,+95.7] | +315.4 [+302.3,+344.7] |
| Recent- | +0.0127 [+0.0024,+0.0227] | +176.0 [+163.8,+190.4] | +284.7 [+274.6,+306.4] |
| Tuned-Recent | +0.0048 [-0.0038,+0.0133] | +167.9 [+145.2,+190.1] | +296.2 [+285.8,+312.8] |
| Tuned-NexCP | +0.0178 [+0.0089,+0.0297] | +181.0 [+168.2,+203.3] | +276.3 [+262.4,+304.5] |
| ACI-tuned | +0.0120 [+0.0029,+0.0232] | +175.5 [+161.2,+192.3] | +286.9 [+277.7,+307.2] |
| ECI | +0.0091 [-0.0005,+0.0233] | +107.2 [+97.2,+126.2] | +199.5 [+190.1,+220.1] |
| Comparator | 7-day 95% CI | 14-day 95% CI | |
|---|---|---|---|
| Rolling | +315.4 | [+302.3,+344.7] | [+298.4,+348.1] |
| Recent- | +284.7 | [+274.6,+306.4] | [+274.2,+310.0] |
| Tuned-Recent | +296.2 | [+285.8,+312.8] | [+284.9,+313.9] |
| Tuned-NexCP | +276.3 | [+262.4,+304.5] | [+266.2,+307.5] |
| ACI-tuned | +286.9 | [+277.7,+307.2] | [+276.3,+309.4] |
| ECI | +199.5 | [+190.1,+220.1] | [+191.9,+220.6] |
| SMSCP | ResCP | [95% CI] | |||||
|---|---|---|---|---|---|---|---|
| Dataset | Cov. | Width | IS | Cov. | Width | IS | |
| Alachua | 0.910 | 264.0 | 361.0 | 0.886 | 202.4 | 334.0 | -27.0 [-31.9,-21.8] |
| Phoenix | 0.920 | 188.2 | 252.8 | 0.883 | 90.8 | 188.5 | -64.4 [-69.9,-58.4] |
| Seattle | 0.914 | 208.4 | 286.2 | 0.885 | 142.8 | 242.5 | -43.7 [-49.0,-38.9] |
| Houston | 0.914 | 249.3 | 332.9 | 0.867 | 165.8 | 297.3 | -35.7 [-41.6,-30.3] |
| Minneapolis | 0.909 | 214.3 | 295.6 | 0.889 | 150.8 | 251.7 | -43.9 [-48.2,-39.3] |
| Design | Size | SMSCP median sec. | HopCPT median train+infer sec. | HopCPT train-time IQR |
|---|---|---|---|---|
| HomoSin | S | 1.42 | 20.51 | 30.11 |
| HomoSin | L | 3.49 | 26.17 | 13.34 |
| DualFreq | S | 1.41 | 20.64 | 1.46 |
| DualFreq | L | 4.64 | 29.03 | 2.94 |
| HeteroSin | S | 1.06 | 18.55 | 4.42 |
| HeteroSin | L | 4.72 | 28.54 | 1.63 |
Conformal Prediction
Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.
See all 352 papers