Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

Hui Wei; Licai Sun; Guoying Zhao

cs.CVAug 21, 2026

Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

Authors: Hui Wei, Licai Sun, Guoying Zhao

Organizations: ELLIS Institute Finland, Finland · Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland

Abstract

Machines that understand humans should perceive the present and anticipate the future. Existing human-centric vision model are pretrained on human images, set the state of the art in static dense perception, so motion and anticipation are out of reach. Here we present Human-JEPA, a human-centric vision model trained on video by anchored forecasting: dense targets are pinned to a frozen copy of the initialization, preventing a silent collapse of dense perception, and block masks are replaced by a pure past-to-future split, avoiding a five-point action tax and a seventeen-point re-identification collapse. Under frozen probes, Human-JEPA leads the pixel-anchored specialists on pose and person re-identification at 2.7 times fewer parameters, conceding high-resolution dense parsing, and its released predictor head is the first that does not degrade anticipation. A single safely adapted model thus serves both halves of understanding humans.

Figures & tables

Figure 1: What human specialization buys, and what it costs. Seven frozen-probe tracks under one protocol. Human-JEPA (blue, outlined) leads the human specialists on pose, identity, and anticipates.

Figure 2: One frozen checkpoint covers the human perception tasks and anticipates past what it has seen. All outputs are from this paper’s probes on the frozen seed-1 encoder: parsing and surface-part maps, COCO keypoints, and the top Market-1501 match (green frames carry the query identity). For anticipation the released predictor forecasts the latents of the unseen second half of held-out clips; the curve is cosine agreement with the true continuation over 20 clips, above the static reference at every horizon out to two seconds.

		ATR	DensePose	Pose	NTU	Mutual	ReID
Model	Params	mIoU $\uparrow$	mIoU $\uparrow$	AP $\uparrow$	top-1 $\uparrow$	top-1 $\uparrow$	mAP $\uparrow$
Human-centric image encoders
Sapiens2-0.8B	0.8B	.767	.666	.591	n/a	n/a	.4522
DINOv3-L	0.3B	.766	.661	.564	n/a	n/a	.5148
HAP ViT-B	0.09B	.685	.492	.490	n/a	n/a	n/a
Video JEPA

Table 1: Human-JEPA leads the human specialists on pose and identity while conceding dense parsing and paying on both action columns. Frozen probes measured by us under one protocol; bold best, underline second best;

\pm

is the two-seed spread (ReID is seed 1); n/a marks image models, which have no video-action probe.

Model	mAP $\uparrow$	Rank-1 $\uparrow$	Rank-5 $\uparrow$
DINOv3-L	.5148	.7586	.9074
Sapiens2-0.8B	.4522	.7055	.8744
V-JEPA 2.1-L (base)	.4370	.6974	.8702
Human-JEPA (ours)	.4635	.7043	.8798
block-mask variant (A1)	.2666	.5235	.7271

Table 2: The mask family decides the identity cost: forecasting gains 2.7 mAP over the frozen base and passes the human specialist, while block masks lose 17 mAP from the same base, data, and budget. Frozen BNNeck probe on Market-1501, identical head and budget per backbone; bold best, underline second best.

Model	Encoder only	Encoder $+$ shipped head	Effect of the head
	top-1 $\uparrow$	top-1 $\uparrow$	$\Delta$ top-1
V-JEPA 2.1-L (base)	78.91	75.93	$-2.98$
Human-JEPA (ours)	79.93	79.99	$+0.06$

Table 3: The shipped head is not neutral: the base’s released head costs it 2.98 points, while ours is the only one that does no harm. NTU-120 early action with the probe observing the first half of each clip.

Model	NTU top-1 $\uparrow$	mutual $\uparrow$	non-mutual $\uparrow$
A0: V-JEPA 2.1-L (frozen)	70.8	79.6	68.1
A1: anchored block-mask adaptation	65.6	74.7	62.8
whole-person deletion (200 ep)	45.5	57.0	42.0
Human-JEPA	70.4	79.0	67.7

Table 4: Forecasting returns action to the base level, block masks pay a uniform five-point tax, and whole-person deletion collapses at scale. Frozen NTU-120 probe at full scale; mutual is the 26 two-person classes.

Masking	Dense mIoU $\uparrow$	Pose AP $\uparrow$	Mutual top-1 $\uparrow$
random multiblock (default)	$0.7423\pm 0.0010$	n/a	56.5
random multiblock, blended corpus	0.7418	0.6185	55.8
part tubes (HAP-style control)	0.7357	n/a	n/a
whole person, $\|M\|{=}1$	0.7357	0.6132	56.4
whole person $+$ trajectory prior	0.7363	0.6105	52.9
whole person $+$ traj. prior, $\|M\|{=}2$	0.7354	0.6105	54.4

Table 5: Making people the unit of prediction never pays: every person-level variant sits at or below the random-mask band on dense parsing and at or below its matched control on interaction, and only scene-level forecasting lifts interaction. Matched-recipe pilots; rows compare within a horizon block; bold best mutual; n/a marks unprobed cells.

Figure 3: The collapse is silent, and the two preservers play different roles. Top: frozen ATR linear canaries at matched epochs; the full recipe (green) rises to the frozen baseline, the naive control (orange) collapses after 90 epochs, and the anchor-only run (gray) never collapses but plateaus. Bottom: the training losses, indistinguishable and anomaly-free.

Arm	Params	ATR-lin $\uparrow$	ATR-DPT $\uparrow$	DensePose $\uparrow$	Pose AP $\uparrow$	NTU $\uparrow$
A0: V-JEPA 2.1-L (frozen)	0.3B	.6230	.7479	.6490	.6142	70.8
C: naive continued PT (uncurated)	0.3B	.2947	.5907	.3534	.1105	66.5
C img : C $+$ image branch	0.3B	.6223	.7415	.6367	.6262	65.4
anchor only (no image branch)	0.3B	n/a	.709	n/a	.554	61.9
A1: anchored block-mask adaptation	0.3B	.6235	.7452	.6384	.6279	65.6
Human-JEPA	0.3B	.6210	.7433	.6333	.6233	70.4

Table 6: Naive continued pretraining collapses dense perception while its training loss improves, and each ingredient recovers a specific part of it. Full-scale arms under identical frozen probes; gray marks collapsed cells, n/a unprobed metrics.

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 134 papers

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
Mar 15, 2026Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar +6Self-Supervised Visual Representation Learning Visual Representation Learning
FAIR at Meta · Universidad de Zaragoza
ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Mar 23, 2026Haichao Zhang, Yijiang Li, Shwai He +5Latent World Models Trajectory Prediction
Northeastern University · University of California San Diego · University of Maryland +2
Frozen in a Frame: The Velocity Blind Spot in JEPA World Models
Oct 3, 2026Tinghe Zhang, Chunyu Liu, Yu Leon Liu +6World Model Learning Joint-Embedding Predictive Architecture
Tsinghua University · Institute of Automation, Chinese Academy of Sciences · Northeastern University +2

Mar 15, 2026J/K move · Enter open · S save

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran+5

FAIR at Meta · Universidad de Zaragoza

Self-Supervised Visual Representation Learning Visual Representation Learning Video Representation Learning Joint-Embedding Predictive Architecture

Open paper AskPDF arXiv

We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.

Abstract

Video Representation Learning

Explore similar work

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

Frozen in a Frame: The Velocity Blind Spot in JEPA World Models