Selective Next-Location Prediction with MetaModelSelect: A Lightweight Post-Hoc Confidence Metamodel

Calvin Jary; Nafiseh Kahani

cs.LGOct 14, 2026

Selective Next-Location Prediction with MetaModelSelect: A Lightweight Post-Hoc Confidence Metamodel

Authors: Calvin Jary, Nafiseh Kahani

Organizations: Systems and Computer Engineering, Carleton University, Ottawa, Ontario, Canada

Abstract

Next-location prediction can support proactive mobile-network decisions, but acting on unreliable predictions can waste resources. We apply MetaModelSelect, a lightweight post-hoc confidence metamodel, to selective cellular next-location prediction. The mobility-specific model combines a learned embedding of the predicted location with the four largest softmax probabilities and Softmax Concentration. It learns whether a frozen recurrent neural network's top-1 prediction is correct, using only its outputs and without retraining the backbone. On path-balanced Changchun cellular mobility trajectories, we evaluate backbones trained with cross-entropy (CE) and self-adaptive training plus entropy minimization (SAT+EM), comparing against eight scalar confidence scores, two location-aware lookup methods, and a raw-logit multilayer perceptron. Relative to maximum softmax probability, MetaModelSelect achieves mean per-run reductions in area under the risk--coverage curve (AURC), excess AURC (E-AURC), and area under the generalized risk--coverage curve (AUGRC) of 2.6%, 4.2%, and 3.0% for CE, and 2.0%, 3.3%, and 1.9% for SAT+EM. It also achieves the lowest reported mean risk at every evaluated partial coverage. The results show that MetaModelSelect improves confidence ranking in this mobility setting while leaving the underlying location predictor unchanged.

Figures & tables

Input (dimension)	Information represented
Predicted-location embedding ( $d_{e}$ )	Location-dependent reliability.
Top-4 probabilities (4)	Confidence and ambiguity among leading alternatives.
Softmax Concentration (1)	Concentration of the full output distribution.

TABLE I: MetaModelSelect inputs. The total dimension is

d_{e}+5

: 48 for CE and 29 for SAT+EM.

Fig. 1: MetaModelSelect training. The frozen base model supplies correctness as the ground truth, to the metamodel, as a supervision target for training. The base model also outputs logits, which supply the predicted-location embedding and the five numerical features.

Fig. 2: MetaModelSelect inference. The frozen base model supplies the predicted class and output logits. The metamodel combines the class embedding, top-four probabilities, and softmax concentration, then thresholds its correctness score to accept or abstain.

Setting	Value
Window length / embedding	5 / 10
Recurrent architecture	2 layers, 533 hidden units
Normalization	Layer normalization
Epochs / batch size	150 / 2500
Optimizer / learning rate	Adam / $2.4\times 10^{-4}$
Gradient clipping norm	1.0

TABLE II: Base-RNN architecture and training settings. Shared settings are identical for CE and SAT+EM.

Selected settings:	CE / SAT+EM
Embedding dimension $d_{e}$	43 / 24
MetaModelSelect batch size	63 / 59
Fixed settings:
Numerical inputs	Top-4 probabilities and SC (5)
Hidden architecture	2 layers, 200 units each
Normalization / activation	Batch normalization / ReLU

TABLE III: Selected and fixed MetaModelSelect settings. Selection uses five-fold cross-validation within the training split.

Score	Definition
MSP (Softmax)	$\max_{k}p_{k}$
Entropy	$\sum_{k}p_{k}\log p_{k}$
Energy	$T\log\sum_{k}\exp(z_{k}/T)$
Maximum logit	$\max_{k}z_{k}$
Logit margin	$z_{(1)}-z_{(2)}$
Softmax Concentration	$\sum_{k}p_{k}^{2}$

TABLE IV: Scalar confidence scores; larger is more confident. Only energy temperature

T

and pNorm order

p

are tuned.

Metric	MSP	Entropy Conf.	Energy Conf.	Max Logit	Logit Margin	Softmax Conc.	Lorenz-Gini	pNorm ( $p=9$ )
Backbone: Cross-Entropy (CE)
AUGRC	$0.0337(0)$ .0001)	$0.0346(0)$ .0001)	$0.0729(0)$ .0004)	$0.0701(0)$ .0004)	$0.0343(0)$ .0002)	$0.0337(0)$ .0001)	$0.0357(0)$ .0001)	$0.0532(0)$ .0007)
E-AURC	$0.0259(0)$ .0001)	$0.0271(0)$ .0001)	$0.1134(0)$ .0012)	$0.1064(0)$ .0010)	$0.0265(0)$ .0001)	$0.0259(0)$ .0001)	$0.0288(0)$ .0001)	$0.0591(0)$ .0011)
AURC	$0.0421(0)$ .0002)	$0.0433(0)$ .0001)	$0.1296(0)$ .0013)	$0.1226(0)$ .0011)	$0.0427(0)$ .0002)	$0.0421(0)$ .0002)	$0.0450(0)$ .0001)	$0.0753(0)$ .0011)
100	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)
95	$14.86(0)$ .06)	$14.98(0)$ .06)	$16.83(0)$ .07)	$16.63(0)$ .08)	$15.24(0)$ .05)	$14.87(0)$ .06)	$15.15(0)$ .06)	$16.61(0)$ .12)

TABLE V: Scalar post-hoc baselines. Values are mean (standard deviation) over three backbone seeds; lower is better. Cross-validation selected

p=9

for pNorm under both objectives.

Metric	MSP	Softmax Conc.	Location- Reliability	Location- Calibrated MSP	RawLogit-MLP	MetaModelSelect
Backbone: Cross-Entropy (CE)
AUGRC	$0.033\,69(0)$ .00011)	$0.033\,72(0)$ .00011)	$0.054\,73(0)$ .00038)	$0.033\,56(0)$ .00010)	$0.045\,45(0)$ .00029)	$0.032\,67(0)$ .00016)
E-AURC	$0.025\,86(0)$ .00004)	$0.025\,89(0)$ .00004)	$0.074\,29(0)$ .00042)	$0.027\,72(0)$ .00023)	$0.056\,51(0)$ .00034)	$0.024\,78(0)$ .00004)
AURC	$0.042\,07(0)$ .00015)	$0.042\,10(0)$ .00015)	$0.090\,50(0)$ .00052)	$0.043\,93(0)$ .00015)	$0.072\,72(0)$ .00044)	$0.040\,99(0)$ .00015)
100	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)	$17.46(0)$ .06)
95	$14.86(0)$ .06)	$14.87(0)$ .06)	$15.42(0)$ .09)	$14.70(0)$ .07)	$14.83(0)$ .07)	$14.68(0)$ .07)

TABLE VI: Main selective next-location prediction results. Values are mean (standard deviation) over three backbone seeds; lower is better. 100 is the backbone error and is identical across selectors.

Metric	CE $\to$ MMS	CE $\to$ SATEM	SATEM $\to$ MMS
AUGRC	$3.03(0)$ .14)	$7.70(0)$ .71)	$1.87(0)$ .16)
E-AURC	$4.16(0)$ .04)	$6.93(0)$ .99)	$3.26(0)$ .59)
AURC	$2.55(0)$ .02)	$8.00(0)$ .82)	$2.03(0)$ .37)

TABLE VII: Headline relative improvements (% reduction; higher is better). Entries are mean(std) across three runs in compact uncertainty notation. CE = cross-entropy, SAT = SAT+EM. MMS = MetaModelSelect.

Fig. 3: Classwise mean softmax confidence among correct CE predictions, sorted by decreasing mean. This large variation from class-to-class motivates our class-specific metamodel embedding.

Fig. 4: Top-1 Softmax versus Softmax Concentration. There is a correlation between the two features, however the two features offer complementary information to the metamodel.

Fig. 5: Largest versus second-largest softmax probability. A greater Top-2 Softmax increases the likelihood of an incorrect prediction.

Variant	AURC	E-AURC	AUGRC
Full MetaModelSelect	0.04099(15)	0.02478(4)	0.03267(16)
No embedding	0.04197(24)	0.02576(16)	0.03362(15)
No Top $p_{(1)}$ Softmax	0.04137(15)	0.02516(9)	0.03305(15)
No Top $p_{(2)}$ Softmax	0.04131(17)	0.02510(11)	0.03299(14)
No Top $p_{(3)}$ Softmax	0.04122(15)	0.02501(13)	0.03288(15)
No Top $p_{(4)}$ Softmax	0.04111(18)	0.02490(12)	0.03278(16)

TABLE VIII: CE-backbone feature ablations. Mean (sample standard deviation) over three seeds, with parentheses applying to the last displayed digits. Lower is better.

Method	Parameters	CPU (ms)	GPU (ms)
Base RNN	2,020,635	0.451(6)	0.579(6)
Maximum Softmax Probability	0	0.015(1)	0.029(1)
Softmax Concentration	0	0.014(1)	0.034(1)
Loc.-Rel.	2,135	0.019(1)	0.027(1)
Loc.-MSP	42,719	0.022(1)	0.082(1)
RawLogit-MLP	468,401	0.075(2)	0.276(1)

TABLE IX: Model complexity and per-example latency. Metamodel parameter counts are calculated from the stated architecture. Parentheses give timing SDs in the last displayed digits.

Selective Prediction

Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.

See all 281 papers

Towards Efficient and Evidence-grounded Mobility Prediction with LLM-Driven Agent
Jun 3, 2026Linyao Chen, Qinlao Zhao, Zechen Li +7LLM Agents Human Behavior Prediction
The University of Tokyo · Huazhong University of Science and Technology · University of New South Wales, Sydney +3
Metacognitive Selective Ensemble for Mobile Systems
Sep 25, 2026Sungmin Lee, Kichang Lee, Joonhee Lee +3Ensemble Learning Deep Ensembles
Yonsei University · Dankook University
Enhancing Human Mobility Prediction with Spatially Aware LLM-based Multi-Agent Systems
Sep 13, 2026Shangyu Lou, Ziqi CuiTrajectory Prediction Spatial Reasoning
University of California & Santa Barbara & San Diego State University California, USA · Politecnico di Milano Milan, Italy

Jun 3, 2026J/K move · Enter open · S save

Towards Efficient and Evidence-grounded Mobility Prediction with LLM-Driven Agent

Linyao Chen, Qinlao Zhao, Zechen Li, Mingming Li+6

The University of Tokyo · Huazhong University of Science and Technology · University of New South Wales, Sydney+3

LLM Agents Human Behavior Prediction Trajectory Prediction

Open paper AskPDF arXiv

Individual-level mobility prediction is central to urban simulation, transportation planning, and policy analysis. Supervised sequence models achieve strong accuracy but require task-specific training and offer limited decision-level transparency. Recent LLM-based methods improve interpretability, yet mostly rely on static prompts and single-pass inference, limiting their ability to seek additional evidence when mobility signals are weak or conflicting. We propose \method{}, a training-free LLM-driven agent framework that formulates next-location prediction as adaptive evidence-controlled decision making. \method{} resolves routine cases through a fast path based on historical regularity, while ambiguous cases trigger iterative tool use over recent trajectories, historical behavior, stay-move likelihood, and geographical evidence. Across three mobility datasets, AgentMob achieves the strongest overall performance among training-free LLM-based methods, with GPT-5.4 reaching 71.42% Acc@1 on BW, 33.14% on YJMob100K, and 33.50% on Shanghai ISP. On BW non-fast-path cases, the LLM controller improves Acc@1 from 30.65% to 48.62% over a same-tool statistical baseline, showing that its main benefit lies in resolving ambiguous predictions through adaptive evidence gathering. Our code is available at https://github.com/Unknown-zoo/AgentMob.

Abstract

Selective Prediction

Explore similar work

Towards Efficient and Evidence-grounded Mobility Prediction with LLM-Driven Agent

Metacognitive Selective Ensemble for Mobile Systems

Enhancing Human Mobility Prediction with Spatially Aware LLM-based Multi-Agent Systems