Selective Next-Location Prediction with MetaModelSelect: A Lightweight Post-Hoc Confidence Metamodel
Organizations: Systems and Computer Engineering, Carleton University, Ottawa, Ontario, Canada
Abstract
Next-location prediction can support proactive mobile-network decisions, but acting on unreliable predictions can waste resources. We apply MetaModelSelect, a lightweight post-hoc confidence metamodel, to selective cellular next-location prediction. The mobility-specific model combines a learned embedding of the predicted location with the four largest softmax probabilities and Softmax Concentration. It learns whether a frozen recurrent neural network's top-1 prediction is correct, using only its outputs and without retraining the backbone. On path-balanced Changchun cellular mobility trajectories, we evaluate backbones trained with cross-entropy (CE) and self-adaptive training plus entropy minimization (SAT+EM), comparing against eight scalar confidence scores, two location-aware lookup methods, and a raw-logit multilayer perceptron. Relative to maximum softmax probability, MetaModelSelect achieves mean per-run reductions in area under the risk--coverage curve (AURC), excess AURC (E-AURC), and area under the generalized risk--coverage curve (AUGRC) of 2.6%, 4.2%, and 3.0% for CE, and 2.0%, 3.3%, and 1.9% for SAT+EM. It also achieves the lowest reported mean risk at every evaluated partial coverage. The results show that MetaModelSelect improves confidence ranking in this mobility setting while leaving the underlying location predictor unchanged.
Figures & tables
| Input (dimension) | Information represented |
|---|---|
| Predicted-location embedding ( ) | Location-dependent reliability. |
| Top-4 probabilities (4) | Confidence and ambiguity among leading alternatives. |
| Softmax Concentration (1) | Concentration of the full output distribution. |
| Setting | Value |
|---|---|
| Window length / embedding | 5 / 10 |
| Recurrent architecture | 2 layers, 533 hidden units |
| Normalization | Layer normalization |
| Epochs / batch size | 150 / 2500 |
| Optimizer / learning rate | Adam / |
| Gradient clipping norm | 1.0 |
| Selected settings: | CE / SAT+EM |
|---|---|
| Embedding dimension | 43 / 24 |
| MetaModelSelect batch size | 63 / 59 |
| Fixed settings: | |
| Numerical inputs | Top-4 probabilities and SC (5) |
| Hidden architecture | 2 layers, 200 units each |
| Normalization / activation | Batch normalization / ReLU |
| Score | Definition |
|---|---|
| MSP (Softmax) | |
| Entropy | |
| Energy | |
| Maximum logit | |
| Logit margin | |
| Softmax Concentration |
| Metric | MSP | Entropy Conf. | Energy Conf. | Max Logit | Logit Margin | Softmax Conc. | Lorenz-Gini | pNorm ( ) |
|---|---|---|---|---|---|---|---|---|
| Backbone: Cross-Entropy (CE) | ||||||||
| AUGRC | .0001) | .0001) | .0004) | .0004) | .0002) | .0001) | .0001) | .0007) |
| E-AURC | .0001) | .0001) | .0012) | .0010) | .0001) | .0001) | .0001) | .0011) |
| AURC | .0002) | .0001) | .0013) | .0011) | .0002) | .0002) | .0001) | .0011) |
| 100 | .06) | .06) | .06) | .06) | .06) | .06) | .06) | .06) |
| 95 | .06) | .06) | .07) | .08) | .05) | .06) | .06) | .12) |
| Metric | MSP | Softmax Conc. | Location- Reliability | Location- Calibrated MSP | RawLogit-MLP | MetaModelSelect |
| Backbone: Cross-Entropy (CE) | ||||||
| AUGRC | .00011) | .00011) | .00038) | .00010) | .00029) | .00016) |
| E-AURC | .00004) | .00004) | .00042) | .00023) | .00034) | .00004) |
| AURC | .00015) | .00015) | .00052) | .00015) | .00044) | .00015) |
| 100 | .06) | .06) | .06) | .06) | .06) | .06) |
| 95 | .06) | .06) | .09) | .07) | .07) | .07) |
| Metric | CE MMS | CE SATEM | SATEM MMS |
|---|---|---|---|
| AUGRC | .14) | .71) | .16) |
| E-AURC | .04) | .99) | .59) |
| AURC | .02) | .82) | .37) |
| Variant | AURC | E-AURC | AUGRC |
|---|---|---|---|
| Full MetaModelSelect | 0.04099(15) | 0.02478(4) | 0.03267(16) |
| No embedding | 0.04197(24) | 0.02576(16) | 0.03362(15) |
| No Top Softmax | 0.04137(15) | 0.02516(9) | 0.03305(15) |
| No Top Softmax | 0.04131(17) | 0.02510(11) | 0.03299(14) |
| No Top Softmax | 0.04122(15) | 0.02501(13) | 0.03288(15) |
| No Top Softmax | 0.04111(18) | 0.02490(12) | 0.03278(16) |
| Method | Parameters | CPU (ms) | GPU (ms) |
|---|---|---|---|
| Base RNN | 2,020,635 | 0.451(6) | 0.579(6) |
| Maximum Softmax Probability | 0 | 0.015(1) | 0.029(1) |
| Softmax Concentration | 0 | 0.014(1) | 0.034(1) |
| Loc.-Rel. | 2,135 | 0.019(1) | 0.027(1) |
| Loc.-MSP | 42,719 | 0.022(1) | 0.082(1) |
| RawLogit-MLP | 468,401 | 0.075(2) | 0.276(1) |
Selective Prediction
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
See all 281 papers