Cross-Visibility-Gap Identity Persistence for 4D Scene Graphs
Authors: Jiahui Hu, Li Gao, Yu Liu, Pengzhan Zhou
Organizations: Department of Computer Science, The University of Hong Kong · Chongqing University · Department of Computing, The Hong Kong Polytechnic University
4D scene graphs require persistent object identity when an object disappears inside an opaque carrier and later reappears as a fresh detection. When appearance and target-centric motion are unavailable, identity recovery must rely on relational history. We formulate cross-visibility-gap identity persistence as historical-node recovery through a remembered target--carrier binding. We introduce CVG-SGM, a perception-frontend-compatible transfer--maintain--read memory. Geometry-weighted cross-node interaction transfers target-specific evidence into carrier states, a shared per-node GRU maintains this information during transport, and a two-time readout combines stored carrier--identity compatibility with current detection--carrier geometry to recover the historical identity. In the controlled one-identity-per-carrier setting, CVG-SGM achieves 1.000 re-link accuracy, compared with 0.570 for both no-memory and parameter-count-matched no-write controls. It also generalizes to scene-disjoint, unseen longer visibility gaps. On real RGB-D/SAM2 feature-level episodes, it correctly re-links 36/36 held-out detections, compared with 28/36 for the strongest fixed carrier--identity control. Counterfactual carrier-memory swaps redirect predictions according to the swapped bindings, supporting the role of stored relational information in identity recovery. The current approach remains limited by shared-carrier identity ambiguity and does not provide end-to-end raw-video re-identification.
Figures & tables
Figure 1: Cross-visibility-gap identity persistence through remembered carrier relations. (a) Controlled 2t2c diagnostic: two identical targets bind to carriers cA and cB , are occluded while the carriers swap, and reappear near their current carriers at locations that are uninformative with respect to their own pre-gap positions. (b) The bindings persist as memory through the gap, re-linking each fresh detection to its identity; on unseen long gaps, FULL improves re-link accuracy from 0.576 (NOMEM) to 1.000 . (c) Complementary human-operated cup-shuffle videos provide offline carrier-recovery evidence, while full real RGB-D/SAM2 feature-level 2t2c re-linking is reported in Table 4 .
Figure 2: Overview of CVG-SGM. (1) Identical targets disappear with separately trackable carriers and reappear without continuous target tracks. (2) Nodes from GT, SAM2, or 4DSG pass through a track-indexed encoder and masked intra-frame attention. (3) Geometry-weighted cross-node transfer lets target-specific evidence enter a carrier’s recurrent state, updated by a shared GRU that maintains it through the gap; an auxiliary relation head reads these states under relation supervision. (4) At reappearance, routing weights rkc from current detection–carrier geometry combine with identity compatibility Bci from end-of-carry memory as ski=∑c∈CrkcBci to recover the historical identity.
Protocol
NOMEM
FULL
Oracle
Gain
Clean-GT, gap 10
0.570±0.000
1.000±0.000
1.000
+0.430
SAM2 geometry
0.550±0.087
1.000±0.000
1.000
+0.450
OOD 5,10→15,20
0.576±0.000
1.000±0.000
1.000
+0.424
Table 1: Re-link accuracy over three seeds. SAM2 uses derived pre-gap geometry with a ground-truth reappearance proxy; OOD trains on gaps 5 and 10 and evaluates scene-disjoint gaps 15 and 20.
Condition
NOMEM
FULL
Gain
Clean
0.530±0.000
1.000±0.000
+0.470
Read jitter ( σ=.02 )
0.529±0.001
0.999±0.001
+0.470
Maint. dropout ( q=.20 )
0.516±0.001
0.887±0.044
+0.371
Table 2: Re-link accuracy on 200 independently generated gap-10 scenes. Corruption seeds are averaged within each training seed before reporting mean ± sample SD over three frozen training seeds.
Variant
Masked attn.
Explicit msg.
GRU
Re-link acc.
NOMEM
Yes
No
No
0.570±0.000
NOWRITE
No
No
Yes
0.570±0.000
NORELWRITE
Yes
No
Yes
0.716±0.246
FULL
Yes
Yes
Yes
1.000±0.000
Table 3: Mechanism ablations on controlled 2t2c re-linking. “Explicit msg.” denotes the general geometry-weighted cross-node message in Eq. ( 5 ).
Method
All (desc.)
Held-out test
Side-only prior
0.508
0.556
Carrier prior
0.483
0.222
Fixed carrier–identity
0.517
0.778
Neural FULL
0.983
1.000
Table 4: Real-data full re-linking over 120 RGB-D/SAM2 episodes. “All” descriptively combines 240 train/validation/test decisions; the held-out test contains 36 decisions. Controls are deterministic, while Neural FULL is identical across five training seeds.
Setting
NOMEM
Sym. Range
FULL
4 tgt / 4 car
0.217±0.056
0.792 – 0.875
0.917±0.007
4 tgt / 3 car
0.271±0.062
0.479 – 0.742
0.738±0.013
Table 5: Instance-level roster scaling and carrier collision. “Sym. Range” spans Hard-mean, Hard-max, and Hard-last; entries report mean ± sample SD over seeds 0–2.
Multi-view tracking extends visual coverage beyond a single camera, providing complementary observations when targets are occluded or leave the view. This raises a question: how can observations from different views be matched to the correct identities without offline scene-specific training or camera calibration? Progress toward this goal remains limited because many offline formulations assume such deployment-specific knowledge, while online adaptation can amplify association errors. We study this problem as online multi-view identity maintenance (OMIM): targets are initialized once per view, with no further identity labels provided after each initialization. To this end, we introduce OmniGlimpse, an asymmetric propose-verify framework that separates adaptive association from stable identity consolidation. At the representation level, short-term local and cross-view evidence is mapped into target-specific adaptive spaces to propose associations, whereas long-term identity distributions are maintained for verification in a canonical space defined by a frozen encoder. At the update level, short-term evidence may guide current associations, whereas, after bootstrapping, only verified observations may update persistent memory or drive adaptation. Under a unified target-specified evaluation protocol, OmniGlimpse improves over the best evaluated online baseline for each metric by 9.1 IDF1 and 22.1 CVIDF1 points on M3Track. The corresponding gains on MvMHAT are 20.3 and 26.2 points. Despite using no offline scene-specific training, it outperforms offline-trained references in IDF1 and remains competitive in cross-view association. These results support an approach to online identity maintenance that adapts to changing appearances while consolidating reliable evidence into stable identity anchors.
Long-term language-guided referring in fixed-view videos is challenging: the referent may be occluded or leave the scene for long intervals and later re-enter, while framewise referring pipelines drift as re-identification (ReID) becomes unreliable. AR2-4FV leverages background stability for long-term referring. An offline Anchor Bank is distilled from static background structures; at inference, the text query is aligned with this bank to produce an Anchor Map that serves as persistent semantic memory when the referent is absent. An anchor-based re-entry prior accelerates re-capture upon return, and a lightweight ReID-Gating mechanism maintains identity continuity using displacement cues in the anchor frame. The system predicts per-frame bounding boxes without assuming the target is visible in the first frame or explicitly modeling appearance variations. AR2-4FV achieves +10.3% Re-Capture Rate (RCR) improvement and -24.2% Re-Capture Latency (RCL) reduction over the best baseline, and ablation studies further confirm the benefits of the Anchor Map, re-entry prior, and ReID-Gating.
Teng Yan, Yihan Liu, Jiongxu Chen +3
The Hong Kong University of Science and Technology (Guangzhou)
City-scale person re-identification across distributed cameras must handle severe appearance changes from viewpoint, occlusion, and domain shift while complying with data protection rules that prevent sharing raw imagery. We introduce CityGuard, a topology-aware transformer for privacy-preserving identity retrieval in decentralized surveillance. The framework integrates three components. A dispersion-adaptive metric learner adjusts instance-level margins according to feature spread, increasing intra-class compactness. Spatially conditioned attention injects coarse geometry, such as GPS or deployment floor plans, into graph-based self-attention to enable projectively consistent cross-view alignment using only coarse geometric priors without requiring survey-grade calibration. Differentially private embedding maps are coupled with compact approximate indexes to support secure and cost-efficient deployment. Together these designs produce descriptors robust to viewpoint variation, occlusion, and domain shifts, and they enable a tunable balance between privacy and utility under rigorous differential-privacy accounting. Experiments on Market-1501 and additional public benchmarks, complemented by database-scale retrieval studies, show consistent gains in retrieval precision and query throughput over strong baselines, confirming the practicality of the framework for privacy-critical urban identity matching.