Cross-Visibility-Gap Identity Persistence for 4D Scene Graphs

Jiahui Hu; Li Gao; Yu Liu; Pengzhan Zhou

cs.CVOct 20, 2026

Cross-Visibility-Gap Identity Persistence for 4D Scene Graphs

Authors: Jiahui Hu, Li Gao, Yu Liu, Pengzhan Zhou

Organizations: Department of Computer Science, The University of Hong Kong · Chongqing University · Department of Computing, The Hong Kong Polytechnic University

Abstract

4D scene graphs require persistent object identity when an object disappears inside an opaque carrier and later reappears as a fresh detection. When appearance and target-centric motion are unavailable, identity recovery must rely on relational history. We formulate cross-visibility-gap identity persistence as historical-node recovery through a remembered target--carrier binding. We introduce CVG-SGM, a perception-frontend-compatible transfer--maintain--read memory. Geometry-weighted cross-node interaction transfers target-specific evidence into carrier states, a shared per-node GRU maintains this information during transport, and a two-time readout combines stored carrier--identity compatibility with current detection--carrier geometry to recover the historical identity. In the controlled one-identity-per-carrier setting, CVG-SGM achieves 1.000 re-link accuracy, compared with 0.570 for both no-memory and parameter-count-matched no-write controls. It also generalizes to scene-disjoint, unseen longer visibility gaps. On real RGB-D/SAM2 feature-level episodes, it correctly re-links 36/36 held-out detections, compared with 28/36 for the strongest fixed carrier--identity control. Counterfactual carrier-memory swaps redirect predictions according to the swapped bindings, supporting the role of stored relational information in identity recovery. The current approach remains limited by shared-carrier identity ambiguity and does not provide end-to-end raw-video re-identification.

Figures & tables

Figure 1: Cross-visibility-gap identity persistence through remembered carrier relations. (a) Controlled 2t2c diagnostic: two identical targets bind to carriers $c_{A}$ and $c_{B}$ , are occluded while the carriers swap, and reappear near their current carriers at locations that are uninformative with respect to their own pre-gap positions. (b) The bindings persist as memory through the gap, re-linking each fresh detection to its identity; on unseen long gaps, FULL improves re-link accuracy from $0.576$ (NOMEM) to $1.000$ . (c) Complementary human-operated cup-shuffle videos provide offline carrier-recovery evidence, while full real RGB-D/SAM2 feature-level 2t2c re-linking is reported in Table 4 .

Figure 2: Overview of CVG-SGM. (1) Identical targets disappear with separately trackable carriers and reappear without continuous target tracks. (2) Nodes from GT, SAM2, or 4DSG pass through a track-indexed encoder and masked intra-frame attention. (3) Geometry-weighted cross-node transfer lets target-specific evidence enter a carrier’s recurrent state, updated by a shared GRU that maintains it through the gap; an auxiliary relation head reads these states under relation supervision. (4) At reappearance, routing weights $r_{kc}$ from current detection–carrier geometry combine with identity compatibility $B_{ci}$ from end-of-carry memory as $s_{ki}=\sum_{c\in\mathcal{C}}r_{kc}B_{ci}$ to recover the historical identity.

Protocol	NOMEM	FULL	Oracle	Gain
Clean-GT, gap 10	$0.570\pm 0.000$	$\mathbf{1.000\pm 0.000}$	$1.000$	$+0.430$
SAM2 geometry	$0.550\pm 0.087$	$\mathbf{1.000\pm 0.000}$	$1.000$	$+0.450$
OOD $5,10\!\rightarrow\!15,20$	$0.576\pm 0.000$	$\mathbf{1.000\pm 0.000}$	$1.000$	$+0.424$

Table 1: Re-link accuracy over three seeds. SAM2 uses derived pre-gap geometry with a ground-truth reappearance proxy; OOD trains on gaps 5 and 10 and evaluates scene-disjoint gaps 15 and 20.

Condition	NOMEM	FULL	Gain
Clean	$0.530\pm 0.000$	$\mathbf{1.000\pm 0.000}$	$+0.470$
Read jitter ( $\sigma=.02$ )	$0.529\pm 0.001$	$\mathbf{0.999\pm 0.001}$	$+0.470$
Maint. dropout ( $q=.20$ )	$0.516\pm 0.001$	$\mathbf{0.887\pm 0.044}$	$+0.371$

Table 2: Re-link accuracy on 200 independently generated gap-10 scenes. Corruption seeds are averaged within each training seed before reporting mean

\pm

sample SD over three frozen training seeds.

Variant	Masked attn.	Explicit msg.	GRU	Re-link acc.
NOMEM	Yes	No	No	$0.570\pm 0.000$
NOWRITE	No	No	Yes	$0.570\pm 0.000$
NORELWRITE	Yes	No	Yes	$0.716\pm 0.246$
FULL	Yes	Yes	Yes	$\mathbf{1.000\pm 0.000}$

Table 3: Mechanism ablations on controlled 2t2c re-linking. “Explicit msg.” denotes the general geometry-weighted cross-node message in Eq. ( 5 ).

Method	All (desc.)	Held-out test
Side-only prior	$0.508$	$0.556$
Carrier prior	$0.483$	$0.222$
Fixed carrier–identity	$0.517$	$0.778$
Neural FULL	$\mathbf{0.983}$	$\mathbf{1.000}$

Table 4: Real-data full re-linking over 120 RGB-D/SAM2 episodes. “All” descriptively combines 240 train/validation/test decisions; the held-out test contains 36 decisions. Controls are deterministic, while Neural FULL is identical across five training seeds.

Setting	NOMEM	Sym. Range	FULL
4 tgt / 4 car	$0.217\pm 0.056$	$0.792$ – $0.875$	$0.917\pm 0.007$
4 tgt / 3 car	$0.271\pm 0.062$	$0.479$ – $0.742$	$0.738\pm 0.013$

Table 5: Instance-level roster scaling and carrier collision. “Sym. Range” spans Hard-mean, Hard-max, and Hard-last; entries report mean

\pm

sample SD over seeds 0–2.

4D Scene Understanding

4D scene understanding reconstructs and reasons about how objects, geometry, and actions change across space and time.

See all 57 papers

OmniGlimpse: Asymmetric Identity Memory for Online Multi-View Tracking
Oct 27, 2026Rangya Zhang, Jiaping Xiao, Yujing Shang +3Multi-Object Tracking Test-Time Adaptation
1Nanyang Technological University
AR2-4FV: Anchored Referring and Re-identification for Long-Term Grounding in Fixed-View Videos
Mar 8, 2026Teng Yan, Yihan Liu, Jiongxu Chen +3Query Re-Identification Vision-Language Grounding
The Hong Kong University of Science and Technology (Guangzhou)
CityGuard: Graph-Aware Private Descriptors for Bias-Resilient Identity Search Across Urban Cameras
Feb 20, 2026Rong Fu, Yibo Meng, Rui Lu +2Person Re-Identification Metric Learning
Independent Researcher

Oct 27, 2026J/K move · Enter open · S save

OmniGlimpse: Asymmetric Identity Memory for Online Multi-View Tracking

Rangya Zhang, Jiaping Xiao, Yujing Shang, Yuhang Zhang+2

1Nanyang Technological University

Multi-Object Tracking Test-Time Adaptation

Open paper AskPDF arXiv

Multi-view tracking extends visual coverage beyond a single camera, providing complementary observations when targets are occluded or leave the view. This raises a question: how can observations from different views be matched to the correct identities without offline scene-specific training or camera calibration? Progress toward this goal remains limited because many offline formulations assume such deployment-specific knowledge, while online adaptation can amplify association errors. We study this problem as online multi-view identity maintenance (OMIM): targets are initialized once per view, with no further identity labels provided after each initialization. To this end, we introduce OmniGlimpse, an asymmetric propose-verify framework that separates adaptive association from stable identity consolidation. At the representation level, short-term local and cross-view evidence is mapped into target-specific adaptive spaces to propose associations, whereas long-term identity distributions are maintained for verification in a canonical space defined by a frozen encoder. At the update level, short-term evidence may guide current associations, whereas, after bootstrapping, only verified observations may update persistent memory or drive adaptation. Under a unified target-specified evaluation protocol, OmniGlimpse improves over the best evaluated online baseline for each metric by 9.1 IDF1 and 22.1 CVIDF1 points on M3Track. The corresponding gains on MvMHAT are 20.3 and 26.2 points. Despite using no offline scene-specific training, it outperforms offline-trained references in IDF1 and remains competitive in cross-view association. These results support an approach to online identity maintenance that adapts to changing appearances while consolidating reliable evidence into stable identity anchors.

Abstract

4D Scene Understanding

Explore similar work

OmniGlimpse: Asymmetric Identity Memory for Online Multi-View Tracking

AR2-4FV: Anchored Referring and Re-identification for Long-Term Grounding in Fixed-View Videos

CityGuard: Graph-Aware Private Descriptors for Bias-Resilient Identity Search Across Urban Cameras