Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video

Yuting Tan; Xilong Cheng; Yunxiao Qin; Zhengnan Li; Jingjing Zhang

cs.CVMar 14, 2026

Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video

Authors: Yuting Tan, Xilong Cheng, Yunxiao Qin, Zhengnan Li, Jingjing Zhang

Organizations: Communication University of China, Beijing, China · State Key Laboratory of Media Convergence and Communication, Beijing, China · The Chinese University of Hong Kong, Shenzhen

Abstract

Humans develop visual intelligence through perceiving and interacting with their environment - a self-supervised learning process grounded in egocentric experience. Inspired by this, we ask how can artificial systems learn stable object representations from continuous, uncurated first-person videos without relying on manual annotations. This setting poses challenges of separating, recognizing, and persistently tracking objects amid clutter, occlusion, and ego-motion. We propose EgoViT, a unified vision Transformer framework designed to learn stable object representations from unlabeled egocentric video. EgoViT bootstraps this learning process by jointly discovering and stabilizing "proto-objects" through three synergistic mechanisms: (1) Proto-object Learning, which uses intra-frame distillation to form discriminative representations; (2) Depth Regularization, which grounds these representations in geometric structure; and (3) Teacher-Filtered Temporal Consistency, which enforces identity over time. This creates a virtuous cycle where initial object hypotheses are progressively refined into stable, persistent representations. The framework is trained end-to-end on unlabeled first-person videos and exhibits robustness to geometric priors of varied origin and quality. On standard benchmarks, EgoViT achieves +8.0% CorLoc improvement in unsupervised object discovery and +4.8% mIoU improvement in semantic segmentation, demonstrating its potential to lay a foundation for robust visual abstraction in embodied intelligence.

Figures & tables

Figure 1: Visual data complexity comparison. (a) ImageNet [ 19 ] and (b) Kinetics-400 [ 36 ] feature predominantly object-centric scenes with clear backgrounds or structured interactions. (c) Unconstrained egocentric videos [ 62 ] present substantially greater complexity, featuring dense object interactions, severe occlusions, and continuous ego-motion, which together pose unique challenges for learning persistent object representations.

Figure 2: EgoViT adopts a Teacher-Student architecture, processing input frames

\{X^{t}\}_{t=1}^{T}

and

\{P_{n}^{t}\}^{T,N}_{t=1,n=1}

. Student

g_{\theta}

learns from three mechanism: (1) depth-regularization

\mathcal{L}_{\text{depth}}

; (2) proto-object learning

\mathcal{L}_{\text{proto}}

; (3) teacher-filtered temporal consistency

\mathcal{L}_{\text{temp}}

. The teacher network

g_{\theta^{\prime}}

is updated using EMA.

Figure 3: Proto-object Delineation via Teacher Attention.

Figure 4: (a) Depth Regularization: An auxiliary task,

\mathcal{L}_{\text{depth}}

, provides geometric constraint. (b) Proto-Object Learning: A distillation loss,

\mathcal{L}_{\text{proto}}

, aligns student and teacher features in proto-level. Here,

H(y^{\prime},y)

denotes the cross-entropy between the softmax outputs of a teacher target

y^{\prime}

and a student prediction

y

. (c) Teacher-Filtered Temporal Consistency: A contrastive loss

\mathcal{L}_{\text{temp}}

is applied on reliable pairs filtered by the teacher to enforce temporal identity.

METHOD	Semantic SEG.		Object DET.	Instance SEG.	Video Object Segmentation			Object DIS.	Classification
METHOD	mIoU	Acc m	mAP	mAP	$(\mathcal{J}\&\mathcal{F})_{m}$	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$	CORLOC	LP	$k$ -NN
SimCLR [ 14 ]	22.5	32.8	22.2	20.1	52.3	51.5	53.1	35.5	28.2	33.1
AttMask [ 34 ]	25.1	35.4	25.9	23.9	52.2	52.5	51.8	37.8	25.4	35.9
MoCo-v3 [ 15 ]	17.9	26.0	19.0	17.3	52.5	50.9	54.0	43.1	22.5	31.2
MAE [ 28 ]	23.0	33.2	24.6	22.1	51.3	50.3	52.2	35.9	13.0	17.8
iBOT [ 75 ]	23.9	34.5	22.1	19.6	53.9	53.5	54.3	36.3	27.5	33.6

Table 1: Comparison of EgoViT with prior methods on downstream tasks. Our default model, EgoViT

{}_{\text{Zurich}}

is trained on a single 65-minute video, while EgoViT

{}_{\text{WT-all}}

is trained on the full Walking Tours dataset to evaluate scalability. Bold indicates the best result in each column. Numbers in parentheses ( green ) denote absolute gains of EgoViT

{}_{\text{Zurich}}

over DINO. All metrics are reported in %, and higher is better.

D	P	T	$k$ -NN	CORLOC
$\times$	$\times$	$\times$	21.8	27.5
$\checkmark$	$\times$	$\times$	22.2	34.6
$\times$	$\checkmark$	$\times$	22.0	35.6
$\times$	$\times$	$\checkmark$	22.2	35.2
$\checkmark$	$\checkmark$	$\times$	22.5	36.2
$\checkmark$	$\times$	$\checkmark$	22.9	37.9

Table 2: Component ablation study. The results highlight the individual contributions of D, P, and T. Notably, combining P and T without D (P+T) leads to performance degradation, underscoring the importance of geometric prior.

Method	Strategy	Frames	$k$ -NN	CORLOC
EgoViT (D+T)	Frame	3	21.5 $\downarrow$	34.2 $\downarrow$
EgoViT (D+T ${}_{\text{w/o filter}}$ )	Proto	3	21.5	31.4 $\downarrow$
EgoViT (D+T ${}_{\text{w/o filter}}$ )	Proto	4	21.2	33.2 $\downarrow$
EgoViT (D+T)	Proto	3	22.7	35.9 $\uparrow$
EgoViT (D+T)	Proto	4	22.9	37.9 $\uparrow$
EgoViT (D+T)	Proto	5	22.0	35.4 $\uparrow$

Table 3: Ablation on temporal modeling strategies. Our full proto-level approach (with the teacher-filtered) is the only strategy that significantly outperforms the baseline, highlighting the filter’s essential role.

Method	$k$ -NN	CORLOC
w/o Depth (no D)	$22.4$	$35.9$
Gaussian blur ( $\sigma_{0}=0.15$ )	$22.9$	$37.7$
Gaussian blur ( $\sigma_{0}=0.3$ )	$22.7$	$37.2$
Gaussian blur ( $\sigma_{0}=0.6$ )	$22.6$	$36.4$
MiDaS (other estimator)	$22.6$	$36.7$
Depth-Pro (other estimator)	$\mathbf{23.5}$	$\mathbf{38.6}$

Table 4: Robustness to depth quality and source. We retrain EgoViT with (a) Gaussian blurred depth (

\sigma=\sigma_{0}\times W/224

, scaled to image size) and (b) alternative monocular estimators (MiDaS, Depth-Pro).

Figure 5: EgoViT achieves superior temporal attention stability. Compared to baselines (DINO, DoRA) that exhibit significant attention drift over time, our method maintains a coherent focus on the target object across the sequence, even through severe occlusion (see red circles for failure cases).

Backbone	AUC	P	$P_{\text{norm}}$
DINO	60.5	64.7	70.1
DoRA	61.7	66.4	72.3
EgoViT	64.7	70.6	74.1

Table 5: Dynamic Tracking on LaSOT. Evaluated under long sequences with occlusion and target disappearance–reappearance using standard metrics (AUC,

P

P_{\text{norm}}

Figure 6: EgoViT demonstrates robust generalization across diverse visual environments.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Shape / Dim.	Source
$X^{t}$	RGB input frame	$H\times W\times 3$	Input
$P^{t}_{n}$	Masked RGB input for proto-object $n$	$H\times W\times 3$	Input
Student Network Outputs
$\mathbf{m}^{t}$	Middle-layer representation	$\mathbb{R}^{D_{1}}$	Student
$\mathbf{z}^{t}$	Penultimate-layer feature	$\mathbb{R}^{D_{2}}$	Student
$\mathbf{f}^{t}$	Final-layer feature (unmasked input)	$\mathbb{R}^{D_{3}}$	Student

Appendix

Table 6: Notation used throughout the paper. All variables are indexed by frame time

t

. For shapes,

S

denotes the number of spatial patch tokens.

D_{1},D_{2},D_{3}

are layer-specific feature dimensions for the backbone.

D,D_{q},D_{k}

are the feature dimensions for patch embeddings and attention components.

Video Name	Duration (min)
WT-Zurich	$\mathbf{\sim 65}$
WT-Istanbul	$\sim$ 73
WT-Stockholm	$\sim$ 68
WT-Chiang Mai	$\sim$ 70
WT-Kuala Lumpur	$\sim$ 66
WT-Venice	$\sim$ 110

Appendix

Table 7: Details of egocentric videos used for pre-training. All videos are 4K at 60 FPS. The primary video is marked in bold; all others are used for augmentation.

Parameter	Symbol	Value
Student Temperature	$\alpha_{\text{student}}$	0.1
Teacher Temperature (Final)	$\alpha_{\text{teacher}}$	0.04
Teacher Similarity Threshold	$\lambda$	0.8
Sampled Proto-objects ( $K$ )	$K$	3
Temporal Window Size	$W$	4
Teacher momentum coefficient	$m$	0.996

Appendix

Table 8: Hyperparameters for proto-object learning framework.

METHOD	Depth	$k$ -NN	CORLOC
DINO	$\times$	33.7	28.6
EgoViT-D	3	34.5	40.6
EgoViT-D	4	35.0	39.8
EgoViT-D	5	34.3	35.2
EgoViT-D	6	34.9	37.2
EgoViT-D	8	34.8	39.0

Appendix

Table 9: Feature extraction depth evaluation on

k

-NN and CORLOC metrics. Shallower layers (depth 3–4) better preserve spatial information while maintaining comparable semantic features.

Threshold and Window Size Effect
$W$	$\lambda$	$k$ -NN	CORLOC
3	0.5	20.8	32.2
3	0.7	22.6	36.2
3	0.8	22.7	35.9
3	0.9	22.1	37.5
4	0.7	22.2	37.0

Appendix

Table 10: Ablation on temporal modeling with different thresholds (

\lambda

) and temporal window sizes (

W

). CORLOC benefits from stricter correspondence filtering.

Figure 7: Ablation visualization of EgoViT.

METHOD	Semantic SEG.		Object DET.	Instance SEG.	Video Object Segmentation			Object DIS.	Classification
METHOD	mIoU	Acc m	mAP	mAP	$(\mathcal{J}\&\mathcal{F})_{m}$	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$	CORLOC	LP	$k$ -NN
SimCLR	22.5	32.8	22.2	20.1	52.3	51.5	53.1	35.5	28.2	33.1
AttMask	25.1	35.4	25.9	23.9	52.2	52.5	51.8	37.8	25.4	35.9
MoCo-v3	17.9	26.0	19.0	17.3	52.5	50.9	54.0	43.1	22.5	31.2
MAE	23.0	33.2	24.6	22.1	51.3	50.3	52.2	35.9	13.0	17.8
iBOT	23.9	34.5	22.1	19.6	53.9	53.5	54.3	36.3	27.5	33.6

Appendix

Table 11: Performance comparison of EgoViT against state-of-the-art methods across a range of downstream tasks. Our main model, EgoViT

{}_{\text{Zurich}}

, is trained on a single 65-minute video, while EgoViT

{}_{\text{WT-Sub5}}

is an additional model trained on five videos to demonstrate scalability. Models without an explicit subscript are trained on the default Zurich video unless otherwise noted. Except for PooDLe, which uses a ResNet-50 backbone (weights obtained from the official release), all other baselines and our models use a ViT-S architecture for fair comparison.

TrackingNet
Metric	DINO	DoRA	EgoViT
AUC	76.2	77.7	78.9
P_Norm	81.2	82.5	83.5
P	73.7	74.7	77.6
GOT-10k (zero-shot)
Metric	DINO	DoRA	EgoViT

Appendix

Table 12: DINO, DoRA, and EgoViT (Zurich-pretrained) backbones are evaluated under the OSTrack framework on TrackingNet and GOT-10k.

Method	Pretrain	VOS Fine-Tune	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$(\mathcal{J}\&\mathcal{F})_{unseen}$
Baseline	–	$\checkmark$	55.5	53.9	57.1	49.5
DNIO	WT ${}_{\text{Zurich}}$	$\checkmark$	62.4	61.5	63.3	56.2
DORA	WT ${}_{\text{Zurich}}$	$\checkmark$	63.3	62.2	64.3	57.4
EgoViT	WT ${}_{\text{Zurich}}$	$\checkmark$	69.4	67.8	70.9	64.3

Appendix

Table 13: EPIC-KITCHENS VISOR VOS results. All methods use a ViT-S/16 backbone.

Figure 8: EgoViT maintains stable and distinct masks for the moving tram (red) and the pedestrian (green), effectively distinguishing them despite rapid camera panning and a cluttered background. When the pedestrian leaves the frame, the model seamlessly transitions to tracking the approaching car (also green). Additionally, EgoViT potentially exhibits partial capability in attending to small, distant objects, such as the signage (blue).

Figure 9: EgoViT maintains identity-consistent masks for the walking person (green) and the trash bin (blue), even as illumination, viewpoint, and crowd density vary. The text on the signage (red) is also consistently segmented across frames.

Figure 10: Even when the camera turns and the target becomes partially occluded, EgoViT tends to preserve relatively distinct proto-objects, such as the store sign (red), the shoulder bag (blue), and the suitcase (green). This may indicate that the model is robust to moderate viewpoint changes and occlusions.

Dimension	DoRA	EgoViT (ours)
Core paradigm	Multi-view spatial consistency on global image embeddings.	Proto-consistency on object-centric prototypes distilled from attention heads.
Learning unit	Whole-scene representation; no explicit object-level carrier.	Proto-object prototype as the basic unit for representation and supervision.
Temporal modeling	Similar patches across time used only to create augmented views; the loss is purely spatial, without explicit temporal consistency.	Dedicated temporal loss that penalizes prototype drift and enforces identity persistence across frames.
Structure awareness	RGB-only, without geometric priors.	Auxiliary depth regularization encourages geometry-aware, structure-sensitive proto-object representations.

Appendix

Table 14: Conceptual comparison between DoRA and EgoViT.

Figure 11: Example frames from the proposed Ego4D-mini benchmark. Row (a) shows cluttered kitchen scenes with strong hand–object interactions. Row (b) shows in-car driving sequences under low light and motion blur, with diverse dashboard and road objects. Row (c) shows domestic walking scenes. These clips are used only for evaluation; both DINO and EgoViT are pre-trained on a separate Zurich city-walk video.

Figure 12: Qualitative comparison on the Kitchen clip. Each column corresponds to a different time step. From top to bottom: SAM2 pseudo boxes (red), DINO predictions (green), and EgoViT predictions (blue). EgoViT typically suppresses background clutter and localizes manipulated tools and food items near the hands, whereas DINO often locks onto larger static structures such as countertops or the sink area.

Figure 13: Qualitative comparison on the Driving clip. The layout is the same as in Fig. 12 . Under low light and strong camera motion, DINO frequently drifts to dashboard edges or windshield borders, while EgoViT more reliably tracks salient traffic participants and in-cabin control elements over time.

Figure 14: Qualitative comparison on the HomeWalk clip. The layout is the same as in Fig. 12 . Due to frequent egocentric self-occlusions by the wearer’s body and carried items, DINO often falls back to static background regions (walls, floors, furniture), whereas EgoViT maintains focus on object across occlusion and reappearance.

Split	EgoViT (%)	DINO (%)
Kitchen	10.51	11.63
Driving	16.61	12.11
HomeWalk	25.93	14.89

Appendix

Table 15: CorLoc (%) on Ego4D-mini benchmark with SAM2 pseudo GT.

Unsupervised Visual Concept Discovery

See all 13 papers

Jul 8, 2026

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew +1

School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.

Video Object Segmentation Self-Supervised Learning Video Representation Learning Self-Supervised Visual Representation Learning

Ask PDF

Aug 19, 2026

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.

Mohammad Zamani, Fatemeh Ziaeetabar

Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran, Tehran, Iran

Hand-Object Interaction Detection Egocentric Video Understanding Embodied AI Temporal Video Understanding Human-Object Interaction VLMs for Robotics

Ask PDF

Sep 3, 2026

Object Concepts Emerge from Motion

Object-centric visual representations are important for physical-world perception, but existing visual pretraining methods often capture semantic categories without preserving the identity and coherence of individual instances. We present a biologically inspired framework that learns object-centric representations for single images from raw videos. Our approach uses motion boundaries as a source of object-level grouping: off-the-shelf optical flow and clustering produce pseudo-instance masks, which supervise a single-image encoder with pixel-level pairwise metric learning. The framework requires neither human annotations nor camera calibration. We first obtain 195 million pseudo-labeled frames from 7,163 hours of driving and web videos, then expand the supervision to 421 million frames with Motion-Verified Self-Training, which combines model proposals with motion evidence. We train encoders up to Swin-H and distill the learned representations into a family of Swin backbones. Across monocular depth estimation, 3D object detection, 3D occupancy prediction, and end-to-end planning, the resulting models achieve competitive or superior performance relative to supervised and self-supervised pretraining baselines, with particularly strong transfer on geometry- and instance-sensitive tasks. These results show that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining.

Boshi Li, Xiaohui Wang, Xiaoyang Wu +3

Beijing University of Posts and Telecommunications · The University of Hong Kong, Hong Kong · Department of Computer Science, The University of Hong Kong, Hong Kong

Object-Centric Representation Learning Self-Supervised Visual Representation Learning Video Object-Centric Learning

Ask PDF

Abstract

Unsupervised Visual Concept Discovery

Explore similar work

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI

Object Concepts Emerge from Motion