EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling

Yanlin Jin; Bowen Yang; Cheng Zhang; Yifan Yang; Zhe Li; Jianfei Cai; Kai Zhu

cs.CVNov 5, 2026

EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling

Authors: Yanlin Jin, Bowen Yang, Cheng Zhang, Yifan Yang, Zhe Li, Jianfei Cai, Kai Zhu

Organizations: Monash University · Ant Digital Technologies, Ant Group · Institute of Automation, Chinese Academy of Sciences

Abstract

Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-conditioned egocentric video generation, pose estimation, and joint action-video generation. Our role-adaptive action renderer provides video-native representations for multidirectional action-video learning, making metric motion usable as both a spatial control signal and a recoverable generation target. Cross-stream action-video coupling establishes close correspondence between metric motion and visual dynamics through lightweight adaptation while preserving the pretrained visual prior. Together, these designs establish a generalist framework spanning five action-video tasks. Despite adaptation only on controlled laboratory recordings, EgoGeneralist generalizes to diverse unseen scenes and interactions with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.

Figures & tables

Figure 2: Overview of EgoGeneralist. Role-adaptive rendering ( Sec. 3.2 ) tailors the visual representation of metric 3D action to its task role: an ego-view map with a fixed 3D anchor for conditioning, or a multiview Posegrid for generation and metric readout. A shared frozen video VAE encodes rendered action and video into two latent streams. Cross-stream coupling ( Sec. 3.3 ) uses joint self-attention and attention LoRAs activated only for generation targets, retaining the backbone weights and per-stream input structure. The bottom panel illustrates condition–target assignments for the three prediction directions defined in Sec. 3.1 .

Figure 3: Action-conditioned image-to-video generation. Comparisons on EgoDex (left) and Xperience-10M (right, out-of-distribution). Adapted only on laboratory recordings, EgoGeneralist generalizes to in-the-wild scenes. The generated videos follow the prescribed motion with physically coherent interactions, correctly removing the purple block and tilting the bucket.

	EgoDex (111 clips)						Xperience-10M (50 clips)
Method	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	PCK@20 $\uparrow$	IoU $\uparrow$	Failure $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	PCK@20 $\uparrow$	IoU $\uparrow$	Failure $\downarrow$
EgoSim O [ 12 ]	21.44	0.8572	333.51	65.11	63.47	29.26	16.81	0.6501	1035.55	47.91	50.96	56.38
LOME [ 9 ]	22.33	0.8446	350.61	60.09	64.87	37.77	11.99	0.5413	2114.01	50.85	57.50	87.24
EgoSim Q [ 12 ]	22.48	0.8654	264.87	71.83	66.62	26.99	17.02	0.6535	869.67	51.92	52.41	55.15
Zhang et al. [ 56 ]	23.18	0.8429	253.16	62.78	64.33	29.14	14.99	0.5897	667.10	72.33	64.05	43.21
RynnWorld-Teleop [ 59 ]	25.79	0.8796	258.75	69.28	71.14	38.43	16.05	0.6510	763.77	50.65	54.30	69.57

Table 1: Action-conditioned image-to-video generation on EgoDex and Xperience-10M. EgoGeneralist achieves strong visual fidelity and action control in both datasets. PCK, IoU, and Failure are in percentages; best and second-best are bold and underlined. EgoSim O /EgoSim Q use dataset-provided/Qwen-generated prompts.

Figure 4: Action-conditioned text-to-video generation. EgoGeneralist generates videos that share the same prescribed 3D motion while varying scenes and object appearances according to different text prompts. Actions are drawn from EgoDex (top) and Xperience-10M (bottom).

	EgoDex				Xperience-10M
Method	PCK $\uparrow$	IoU $\uparrow$	Fail $\downarrow$	EgoV-S $\uparrow$	PCK $\uparrow$	IoU $\uparrow$	Fail $\downarrow$	EgoV-S $\uparrow$
Fun-Control	35.41	48.11	51.81	0.3583	47.91	52.64	63.17	0.4213
VACE-Fun	37.09	48.94	53.74	0.3440	49.04	49.61	63.33	0.4095
EgoGeneralist	67.91	66.36	40.15	0.3773	50.59	50.52	61.99	0.4126

Table 2: Action-conditioned text-to-video generation on 100 action–prompt instances per dataset. PCK, hand IoU, and Fail assess action control, while EgoV-S measures video–text semantic alignment. Our method maintains competitive action adherence across text-specified visual contexts.

		Method properties			EgoDex (111 clips)							Xperience-10M (50 clips)
	Method	Camera	Pose output	Data	Abs	PA	W	WA	Fail	RPE-T	RPE-R	Abs	PA	W	WA	Fail	RPE-T	RPE-R
(a) Hand + camera	ReViV [ 27 ]	Predicted	Hand + body	Incl. wild	334.04	21.21	126.84	43.96	0.00	1.08	0.135	192.79	15.93	114.81	44.23	0.00	4.32	0.431
	MINT [ 61 ]	Predicted	Hand only	Incl. wild	186.62	10.50	30.94	18.22	0.34	0.74	0.087	131.97	7.90	61.55	26.56	0.85	3.09	0.220
	HaMeR [ 33 ] †	SLAM †	Hand only	Incl. wild	119.47	12.12	77.05	32.31	14.19	—	—	124.92	6.95	82.80	34.20	6.79	—	—
	EgoForce [ 28 ] †	SLAM †	Hand + forearm	Lab only	114.71	11.79	61.42	31.73	15.17	—	—	105.50	8.27	81.13	39.16	14.24	—	—
	HaWoR [ 58 ]	SLAM	Hand only	Lab only	111.98	11.75	46.44	25.09	9.65	5.17	0.176	110.43	8.38	77.56	32.71	2.54	4.78	0.134

Table 3: Hand, upper-limb, and camera reconstruction on EgoDex and Xperience-10M. EgoGeneralist remains competitive with specialist estimators while jointly recovering hand, upper-limb, and camera motion. HaMeR † and EgoForce † use HaWoR’s camera estimates with camera scores reported in the HaWoR row. ReViV keeps its native two-second input context and all methods are evaluated on 41 frames.

Figure 5: World-action policy on HOT3D. Top: EgoGeneralist’s generated video and 3D action in ego and side views. Bottom: action predictions from all three methods in a shared top view. GT object meshes and scene points are used only for visualization. Additional examples are provided in supplementary Fig. S7 .

Figure 6: Text-only joint action–video generation. Each example shows jointly generated video and 3D action, with the action projected onto video frames. Insets show the generated 3D upper-body skeleton and camera pose. Additional examples appear in Figs. S8 and S9 .

	MinDist (cm) $\downarrow$		Reach (%) $\uparrow$		Progress $\uparrow$
Method	Mean	Med.	@2cm	@5cm	(%)
Initial state	15.98	16.20	0.00	0.00	0.00
Being-H0 †	11.64	11.90	16.16	25.25	32.64
VITRA	6.32	4.59	31.31	52.53	61.96
EgoGeneralist	3.81	1.07	58.59	77.78	77.42

Table 4: World-action policy evaluation on HOT3D. Our method outperforms specialized policy baselines in target reaching on unseen HOT3D scenes while jointly generating video. All methods receive an initial observation and instruction. VITRA and EgoGeneralist also receive initial action states. † Being-H0 uses a single wrist-anchored rigid alignment to the GT initial hand.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Hands and camera
Rendering	Abs $\downarrow$	PA $\downarrow$	W $\downarrow$	WA $\downarrow$	RPE-T $\downarrow$	RPE-R $\downarrow$
Ego-view	38.70	12.61	36.28	22.07	2.13	0.219
Posegrid	21.67	12.01	22.91	15.74	1.59	0.186

Appendix

Table 5: Hand, upper-limb, and camera readout on EgoDex. Multiview Posegrid gives lower reconstruction errors, indicating that its VAE latents better preserve metric action information for readout.

		EgoDex (111 clips)						Xperience-10M (50 clips)
Ablation component	Configuration	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	PCK@20 $\uparrow$	IoU $\uparrow$	Failure $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	PCK@20 $\uparrow$	IoU $\uparrow$	Failure $\downarrow$
Action–video coupling	Action embedding	22.14	0.834	369.06	31.94	43.16	48.91	14.72	0.589	790.48	25.65	34.19	64.80
Action–video coupling	Latent channel concatenation	24.02	0.854	270.98	43.23	52.87	34.57	15.06	0.605	823.82	27.94	37.76	57.29
Action rendering	Multiview Posegrid	24.95	0.865	254.77	50.53	58.21	33.56	15.01	0.604	749.33	36.73	43.36	58.19
Action rendering	Ego-view w/o 3D anchor	26.07	0.877	205.15	67.09	69.37	24.71	15.65	0.626	667.88	58.03	57.67	38.45
Full model	Ego-view + 3D anchor (ours)	28.42	0.916	170.96	68.43	71.46	24.57	19.06	0.707	506.83	58.77	56.79	41.18

Appendix

Table 6: Ablations of two design components for action-conditioned video generation. We evaluate how different action–video coupling strategies and action renderings affect video controllability under matched data and training budgets. The full configuration performs best on most metrics, supporting our cross-stream coupling and role-adaptive rendering design. Best and second-best scores are bold and underlined.

Figure S1 : Decoder learning curves. Multiview Posegrid enables faster loss reduction, particularly for body and hand positions, and reaches lower final validation losses across all supervised components, making it better suited for metric action readout.

Figure S2 : Additional action-conditioned image-to-video generation comparisons on EgoDex and Xperience-10M. EgoGeneralist maintains effective action control and coherent object interactions. Columns show 0%, 30%, 60%, and 90% progress through each 41-frame clip.

Figure S3 : Action-conditioned image-to-video generation across EgoDex and Xperience-10M. Adapted only on EgoDex, EgoGeneralist generates motion-consistent interactions across laboratory scenes and diverse in-the-wild environments. Each example shows reference frames, the ego-view action condition, and our generated video. Columns show 0%, 30%, 60%, and 90% progress through each 41-frame clip.

Figure S4 : Additional action-conditioned text-to-video generation. Action sequences from EgoDex (top) and Xperience-10M (bottom) are each paired with three text prompts. EgoGeneralist varies scenes and objects according to text while maintaining the prescribed motion. GT video is reference only.

Figure S5 : Action-conditioned text-to-video generation on EgoDex (left) and Xperience-10M (right), compared with Wan2.2 Fun-Control and VACE-Fun. Our 3D anchor enables explicit camera-motion control, allowing EgoGeneralist to reproduce prescribed viewpoint changes more faithfully than the baselines. Methods share source actions and prompts but use their native condition representations. The displayed ego-view condition is ours. Columns show frames 0, 20, and 40 of the shared 41-frame evaluation window.

Figure S6 : Egocentric pose estimation. Ground-truth (GT) and predicted hand and arm poses are projected onto the same input frames on EgoDex (top two examples) and Xperience-10M (bottom two examples). EgoGeneralist’s projected hand and arm predictions align closely with the visible hands and ground-truth overlays. Columns indicate relative progress through each displayed window.

Figure S7 : Additional HOT3D examples of joint action–video generation and hand-action comparisons. EgoGeneralist generates smooth hand trajectories that accurately approach the specified objects, whereas the baseline trajectories miss the targets in these examples. Columns show normalized prediction progress, with visualization conventions following Fig. 5 .

Figure S8 : Additional text-only joint action–video generation. Across diverse scenes, generated actions remain well aligned with the overall hand-motion trends in the generated videos. Each example pairs generated video with decoded action overlays at frames 0, 10, 20, 30, and 40.

Figure S9 : Further examples of text-only joint action–video generation.

Figure S10 : Long-sequence generation. EgoGeneralist maintains physically coherent interactions over multiple autoregressive chunks. Four chunks are generated by conditioning each continuation on the preceding generated final frame. Alternating rows compare ground truth and generated video over the first and second halves of the sequence.

Figure S11 : Appearance and background variation. EgoGeneralist generalizes to changes in appearance and environment, opening up further applications of action-conditioned video generation. (a) Two human-to-robot appearance-transfer examples. (b) Ground truth and generated videos in the original scene and edited indoor and outdoor settings.

Action-Conditioned Video Generation

Action-conditioned video generation creates videos whose motion and scene changes are guided by controls, actions, or interaction signals.

See all 117 papers

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
Jul 30, 2026Zexuan Yan, Yuzhou Wu, Yue Ma +9Action-Conditioned Video Generation Memory-Augmented Video Generation
1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd. +4
E $^3$ C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
May 25, 2026Qiao Gu, Lingni Ma, Adam W Harley +3Video Diffusion Models Egocentric Vision
Meta Reality Labs, Redmond, WA 98052, USA · University of Toronto, Toronto, ON M5S 1A1, Canada
Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos
Dec 1, 2025Xavier Thomas, Youngsun Lim, Ananya Srinivasan +2Physical Consistency in Video Generation Video Generation Evaluation
Boston University · Belmont High School · Canyon Crest Academy +1

Jul 30, 2026J/K move · Enter open · S save

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE

Zexuan Yan, Yuzhou Wu, Yue Ma, Zonghang He+8

1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd.+4

Action-Conditioned Video Generation Memory-Augmented Video Generation Synthetic Data Generation for Robotics Egocentric Vision Embodied World Models

Open paper AskPDF arXiv

Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77% to 84% on single-arm tasks and from 53% to 70% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.

Abstract

Action-Conditioned Video Generation

Explore similar work

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE

E3^33C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control

Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

E $^3$ C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control