Which Objects Stay? Learning Hierarchical Persistence Beliefs for Lifelong Open-Vocabulary Object Maps
Authors: Thanh Nguyen Canh, Haolan Zhang, Nak Young Chong
Organizations: School of Information Science, Japan Advanced Institute of Science and Technology, Ishikawa 923-1211, Japan · School of Information Science, Japan Advanced Institute of Science and Technology, Ishikawa 923-1211, Japan, and also with the Department of Robotics, Hanyang University, Gyeonggi-do 15588, South Korea
Lifelong object maps must estimate whether an object remains at its previously observed location after it leaves view. Existing persistence filters rely on class-based survival priors, making them difficult to apply reliably in open-vocabulary settings. We introduce HiPer, a hierarchical persistence filter that derives persistence beliefs directly from data rather than semantic categories. For a new object, HiPer retrieves visually and geometrically similar training instances and estimates an initial survival rate from their observed histories, without requiring class labels. This estimate initializes a conjugate Beta belief over the object's survival probability, which is updated in closed form using detections and missed observations under Rosen's persistence-filter framework. When an object is re-identified at a new location, its persistence belief is transferred with the object, enabling the map to learn object-specific mobility over time. Experiments on 120 held-out simulated sequences show that HiPer consistently achieves the best performance among competing methods. Retrieval-based priors outperform CLIP-derived label priors, and object-specific online learning provides further gains. Results on 47 held-out 3RScan environments demonstrate that the proposed retrieval-based prior also outperforms state-of-the-art baselines in real-world settings.
Figures & tables
Fig. 1: Which objects stay? Top: four sessions of HiPer’s best held-out THOR-Lifelong sequence. Bottom: predicted probability that an object stays in place until the next session for HiPer (solid) and PF( λ ) [ 9 ] (dashed); dots: ground truth (open: gone). Right: negative log-likelihood (NLL) of all 120 held-out sequences; HiPer attains the lower NLL in 116 of them.
Fig. 2: Overview of HiPer on a held-out sequence. Association turns each session into one piece of evidence per map object (hit, miss or moved); ray casting decides which unmatched map objects are in view and thus missed. (1) The survival prior of a new object is retrieved from the K nearest training objects (2-D projection of their DINOv2 embeddings, colored by observed lifetime; stars: the garbage can and the bread). (2) Each object keeps a Beta belief over its own survival, updated in closed form (the bread: prior, after three hits and after two relocations). (3) At a relocation, the belief moves with the object. Objects whose predicted presence falls below τπ are pruned. Top right: predicted presence of all map objects at the next session.
Fig. 3: Probability of remaining in place for one more day, for two mugs with the same prior (lifetime of 2 days) observed daily ( κ0=32 , PM=0.2 , PF=0.001 ); left: the mug stays, right: it moves every second day. HiPer’s prediction rises for the mug that stays and falls for the one that moves. PF( λ ) and fuse-then-decay predict the same for both, and ELite and POCD approach certainty for the mug that stays.
Method
Prediction
Calibrated
Map maintenance
NLL ↓
Brier ↓
ECE ↓
AUC ↑
NLL ↓
ECE ↓
Prem. ↓
Stale ↓
Never forget [ 23 ]
2.470 ± 0.772
0.179 ± 0.056
0.179 ± 0.056
0.500 ± 0.000
0.344 ± 0.076
0.038 ± 0.025
–
1.000 ± 0.000
Fuse-then-decay [ 22 ]
0.879 ± 0.299
0.255 ± 0.061
0.328 ± 0.075
0.769 ± 0.081
0.325 ± 0.079
0.058 ± 0.024
0.146 ± 0.206
0.838 ± 0.182
ELite scalar [ 6 ]
0.365 ± 0.099
0.104 ± 0.029
0.064 ± 0.026
0.773 ± 0.073
0.341 ± 0.079
0.051 ± 0.023
0.118 ± 0.146
0.011 ± 0.041
POCD [ 3 ]
0.402 ± 0.092
0.127 ± 0.034
0.076 ± 0.036
0.784 ± 0.073
0.326 ± 0.077
0.059 ± 0.021
–
1.000 ± 0.000
PF( λ ) [ 9 ]
0.341 ± 0.113
0.097 ± 0.028
0.045 ± 0.025
0.786 ± 0.062
0.318 ± 0.076
0.041 ± 0.022
0.089 ± 0.133
0.008 ± 0.033
TABLE II: Results on the 120 held-out THOR-Lifelong sequences (mean ± standard deviation over sequences). Calibrated: with a calibration map fitted on validation. Bold: best; underlined: second best.
Fig. 4: Qualitative results on THOR-Lifelong sequence. Color: predicted probability of remaining in place until the next session; black ring: the object leaves; red cross: confident error. Each panel reports the NLL of the session (bold: best) and the number of confident errors; the row labels give the NLL of the sequence.
Fig. 5: Reliability diagram on the held-out sequences (15 bins); the diagonal corresponds to perfect calibration. The map that never forgets predicts 1 for every object (single point).
Fig. 6: Predicted presence for objects that leave before the next session (left) and that stay (right), all 42,374 held-out predictions.
Fig. 7: NLL per session on the held-out sequences, raw (left) and after calibration (right). Band: 95% interval of HiPer; shaded: gain over PF( λ ). Raw fuse-then-decay (0.835 to 0.912) and never forget (1.967 to 2.904) lie above the range.
Method
pickupable ( n =16669)
moveable ( n =9866)
fixed ( n =15839)
NLL ↓
AUC ↑
NLL ↓
AUC ↑
NLL ↓
ELite scalar [ 6 ]
0.621 ± 0.111
0.728 ± 0.075
0.278 ± 0.176
0.502 ± 0.224
0.125 ± 0.129
POCD [ 3 ]
0.748 ± 0.112
0.676 ± 0.078
0.279 ± 0.204
0.504 ± 0.222
0.079 ± 0.013
PF( λ ) [ 9 ]
0.563 ± 0.110
0.816 ± 0.057
0.243 ± 0.155
0.722 ± 0.160
0.149 ± 0.186
HiPer
0.481 ± 0.079
0.828 ± 0.052
0.237 ± 0.182
0.716 ± 0.168
0.026 ± 0.046
TABLE III: Results on the held-out THOR-Lifelong sequences per object group.
Method
Prediction
Calibrated
Map maintenance
NLL ↓
Brier ↓
ECE ↓
AUC ↑
NLL ↓
ECE ↓
Prem. ↓
Stale ↓
Never forget [ 23 ]
1.754 ± 1.857
0.127 ± 0.134
0.127 ± 0.134
0.500 ± 0.000
0.371 ± 0.277
0.099 ± 0.090
–
1.000 ± 0.000
Fuse-then-decay [ 22 ]
0.551 ± 0.076
0.181 ± 0.036
0.284 ± 0.092
0.622 ± 0.163
0.372 ± 0.280
0.108 ± 0.088
0.779 ± 0.373
0.787 ± 0.237
ELite scalar [ 6 ]
0.382 ± 0.249
0.110 ± 0.095
0.117 ± 0.077
0.551 ± 0.124
0.371 ± 0.276
0.100 ± 0.086
0.667 ± 0.471
0.958 ± 0.102
POCD [ 3 ]
0.375 ± 0.305
0.108 ± 0.106
0.102 ± 0.095
0.650 ± 0.158
0.364 ± 0.270
0.104 ± 0.084
0.476 ± 0.387
0.771 ± 0.406
PF( λ ) [ 9 ]
0.377 ± 0.298
0.109 ± 0.106
0.098 ± 0.095
0.522 ± 0.127
0.369 ± 0.277
0.097 ± 0.086
0.750 ± 0.354
0.958 ± 0.102
TABLE IV: Results on the 47 3RScan test environments (mean ± standard deviation over environments). Calibrated: with calibration maps fitted on the tuning environments.
Fig. 8: Qualitative results on the 3RScan environment (largest NLL margin over the baselines among environments with at least 50 labeled predictions). Each column shows the scan of one session from above (gray: unobserved) for the three sessions with the most departures.
Leave-one-room-out
Calibrated
Method
NLL ↓
ECE ↓
AUC ↑
NLL ↓
ECE ↓
Never forget [ 23 ]
11.597 ± 2.350
0.839 ± 0.170
0.500 ± 0.000
0.445 ± 0.266
0.156 ± 0.092
Fuse-then-decay [ 22 ]
0.594 ± 0.140
0.298 ± 0.058
0.594 ± 0.098
0.457 ± 0.278
0.159 ± 0.093
ELite scalar [ 6 ]
0.545 ± 0.082
0.265 ± 0.096
0.619 ± 0.067
0.446 ± 0.267
0.160 ± 0.086
POCD [ 3 ]
0.774 ± 0.054
0.400 ± 0.154
0.601 ± 0.160
0.464 ± 0.272
0.161 ± 0.093
PF( λ ) [ 9 ]
0.427 ± 0.318
0.108 ± 0.128
0.544 ± 0.100
0.446 ± 0.266
0.156 ± 0.093
TABLE V: Results on ObChange with leave-one-room-out tuning and cross-fitted priors, raw and after calibration (mean ± standard deviation over rooms). Bold: best; underlined: second best.
Fig. 9: Qualitative results on the ObChange room. Objects on tables and shelves seen from above, colored by the predicted presence at the next visit; black ring: the object leaves. Each panel reports the Brier score.
Variant
THOR held-out
3RScan test
NLL ↓
AUC ↑
NLL ↓
AUC ↑
HiPer (full)
0.265 ± 0.078
0.911 ± 0.040
0.310 ± 0.228
0.790 ± 0.174
Prior
w/o retrieval: CLIP class rates
0.296 ± 0.118
0.887 ± 0.040
0.322 ± 0.227
0.776 ± 0.187
w/o retrieval: hand-set table
0.545 ± 0.218
0.831 ± 0.062
0.521 ± 0.499
0.668 ± 0.159
w/o CLIP: DINOv2-only key
0.270 ± 0.080
0.908 ± 0.040
0.328 ± 0.238
0.733 ± 0.219
TABLE VI: Ablation study: HiPer with one component removed.All other rows keep HiPer’s parameters.
Param.
Value
Prediction
Map maint.
NLL ↓
Brier ↓
ECE ↓
AUC ↑
Prem. ↓
Stale ↓
κ0
8
0.285
0.086
0.054
0.898
0.096
0.019
16
0.276
0.084
0.049
0.904
0.096
0.019
32
0.274
0.083
0.050
0.906
0.085
0.019
64
0.275
0.084
0.052
0.906
0.080
0.013
K
10
0.277
0.084
0.051
0.905
0.080
0.013
TABLE VII: Parameter sensitivity of HiPer on THOR-Lifelong validation (means over sequences): one parameter of the hierarchy or the prune threshold τπ is changed, and the others keep their selected values (bold). τπ changes only pruning; ν=0.1 and τπ=1 lie outside the tuning grids.