Frame-Level Temporal Alignment for Human-to-Robot Visual Adaptation

Xizhe Zhang; Jingfeng Zhang; Zirun Zhou; Hong Jia

cs.ROOct 3, 2026

Frame-Level Temporal Alignment for Human-to-Robot Visual Adaptation

Authors: Xizhe Zhang, Jingfeng Zhang, Zirun Zhou, Hong Jia

Organizations: The University of Auckland · Fudan University

Abstract

Transferring visual representations pretrained on human videos to robot manipulation requires learning reliable correspondences between human and robot demonstrations. However, paired demonstrations can differ in execution rate and in the proportion of non-key frames that do not directly reflect task progress. Frames at the same relative timestamp may therefore represent different task stages, which can cause correspondence learning to fail. To address these issues, we propose Frame-Level Temporal Alignment (FLTA), a framework that uses two temporal priors to adapt visual encoders pretrained on human videos for robot manipulation. It learns shared task-progress representations without frame-level correspondence annotations, allowing frames at different relative temporal positions to match. A global progress prior combines normalized temporal positions with visual similarity to construct a soft correspondence target. A local temporal order prior penalizes backward transitions while allowing stays and varying forward rates to accommodate execution-rate differences. With ResNet-50 and ViT encoders, our method achieves relative improvements of 46.93% and 65.96%, respectively, over the best baselines in average simulation success rates and achieves higher task success rates on real-world manipulation tasks. These results also suggest that effective human-robot adaptation depends less on the number of parameters updated than on which parameters are selected. Our project page is available at https://rtx5090ultra.github.io/FLTA-Project-Page/.

Figures & tables

Figure 1: From video-level to frame-level alignment. Video-level alignment brings human and robot video representations closer without explicitly establishing frame-level correspondences. Differences in execution rates and the proportion of non-key frames mean that frames sampled at the same relative time may show different task stages. Our method addresses these temporal discrepancies by learning cross-domain frame correspondences without requiring frame-level annotations.

Figure 2: Overview of the proposed temporal alignment framework. Here, only the normalization-affine parameters of the visual encoder are updated. The auxiliary frame discrimination loss learns discriminative frame representations within each domain. The global progress prior $D$ and visual similarity jointly construct soft correspondences. The local temporal order prior $Q$ penalizes backward transitions while allowing stays and forward transitions of varying magnitudes. Numbered stages are shown only for illustration.

Figure 3: Simulated and real-world evaluation environments. Top: Level 1 and Level 2 unseen manipulation tasks from the AGNOSTOS benchmark in RLBench. Bottom: Four real-world tasks used on the Trossen WidowX AI Arm.

Encoder	Method	L1 $\uparrow$	L2 $\uparrow$	Avg. $\uparrow$	Encoder	Method	L1 $\uparrow$	L2 $\uparrow$	Avg. $\uparrow$
D4R-ViT	Frozen	8.31 (0.50)	11.60 (0.33)	9.74 (0.25)	DNCE-CLIP	Frozen	10.05 (1.24)	9.73 (1.32)	9.91 (1.14)
	+ HRP	10.56 (1.67)	12.13 (0.38)	11.25 (1.03)		+ HRAlign ∗	12.62 (2.06)	9.47 (1.05)	11.25 (1.58)
	+ Ours	18.05 (0.52)	19.47 (0.50)	18.67 (0.49)		+ Ours	15.38 (0.44)	16.13 (1.00)	15.71 (0.22)
		(+70.93%)	(+60.51%)	(+65.96%)			(+21.87%)	(+70.33%)	(+39.64%)
R3M-RN50	Frozen	12.62 (0.25)	11.73 (0.50)	12.23 (0.36)	AcTOL-CLIP	Frozen	11.28 (1.53)	11.47 (0.19)	11.36 (0.82)
	+ HRAlign	11.59 (1.62)	12.80 (0.33)	12.12 (0.78)		+ HRAlign ∗	11.90 (1.47)	12.27 (0.19)	12.06 (0.78)

Table 1: Zero-shot manipulation success rates (%) on the 23 unseen AGNOSTOS tasks, with standard deviations in parentheses. Avg. denotes the average over all 23 tasks. Bold indicates the best result within each initialization group. Green indicates relative gains over HRP for D4R-ViT and HRAlign for the other initializations. DNCE denotes DecisionNCE. ∗ denotes reproduced.

Method	T1	T2	T3	T4	Avg. $\uparrow$
Frozen	0.35	0.25	0.55	0.00	0.29
+ HRP	0.40	0.50	0.35	0.05	0.33
+ Ours	0.65	0.60	0.85	0.15	0.56

Table 2: Real-world success rates using D4R-ViT initialization. T1–T4 denote pick–place, press, push, and open, respectively.

	In-distribution (RH20T)		Zero-shot (self-collected)
Setting	D4R-ViT	R3M-RN50	D4R-ViT	R3M-RN50
Frozen	15/100	14/100	42/100	36/100
+ HRP / HRAlign	17/100	22/100	47/100	43/100
+ Ours	39/100	43/100	56/100	61/100

Table 3: Human-to-robot task-progress retrieval, reported as correct retrievals out of 100 queries per evaluation setting. Higher is better.

Figure 4: Human-to-robot task-progress retrieval examples from RH20T (top) and the self-collected zero-shot data (bottom). Each row shows a human query and the nearest robot frames retrieved by Frozen R3M, HRAlign, and our method. Green checks indicate correct retrievals.

Variant	$D$	$Q$	$\mathcal{L}_{\mathrm{InfoNCE}}$	D4R-ViT			R3M-RN50
Variant	$D$	$Q$	$\mathcal{L}_{\mathrm{InfoNCE}}$	L1 $\uparrow$	L2 $\uparrow$	Avg. $\uparrow$	L1 $\uparrow$	L2 $\uparrow$	Avg. $\uparrow$
Full	✓	✓	✓	18.05 (0.52)	19.47 (0.50)	18.67 (0.49)	16.82 (1.24)	19.47 (0.19)	17.97 (0.67)
w/o $\mathcal{L}_{\mathrm{InfoNCE}}$	✓	✓	–	16.15 (0.66)	17.27 (1.15)	16.64 (0.54)	16.31 (0.50)	17.73 (0.19)	16.93 (0.22)
w/o $D$	–	✓	✓	14.56 (0.77)	18.53 (0.38)	16.29 (0.43)	15.69 (1.26)	17.07 (0.82)	16.29 (0.91)
w/o $Q$	✓	–	✓	13.74 (0.77)	17.87 (0.50)	15.54 (0.46)	13.13 (0.38)	17.47 (0.68)	15.01 (0.36)
w/o $D$ & $Q$	–	–	✓	11.18 (0.29)	15.20 (0.65)	12.93 (0.33)	11.79 (0.52)	15.47 (0.38)	13.39 (0.38)

Table 4: Ablations and alternatives on the simulation. In the TCC and Soft-DTW variants,

\mathcal{L}_{\mathrm{TA}}

is replaced by the corresponding alignment loss.

Figure 5: Data efficiency and adaptation strategies with R3M-RN50. (a) Overall average success rate across L1 and L2 versus the fraction of paired human–robot pre-training data. (b) Late-adapter (Ad.) versus BN-affine (BN) fine-tuning. Params.: trainable parameters; HRA: HRAlign. All success rates are in percent.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Temporal differences in paired human–robot videos. Left: median Robot-to-Human execution-time ratio, with Human time normalized to 1. Right: mean proportion of non-key frames before action execution.

Symbol	Meaning
$\mathcal{H},\mathcal{R}$	Paired human and robot videos.
$f_{\theta}$	Visual encoder with parameters $\theta$ ; only normalization-affine parameters are updated during adaptation.
$\mathcal{A}_{1},\mathcal{A}_{2}$	Augmentation operators for the two branches.
$q_{n}^{X},k_{n}^{X}$	Representations of the same frame index $n$ under the two augmentations, for domain $X\in\{\mathcal{H},\mathcal{R}\}$ .
$\mathcal{T},\ \|\mathcal{T}\|$	Sampled temporal index set and number of sampled frames per video.
$n,N$	Frame-instance index and total number of sampled frames per domain.

Appendix

Table 5: Core notation used in the proposed method.

Configuration	ViT	RN50
Training iterations	40,000	40,000
Optimizer	AdamW	AdamW
Learning rate	$7.5\times 10^{-5}$	$1.0\times 10^{-4}$
Frames per domain	160	160
Sampled frames	8	8
Weight decay	0	0

Appendix

Table 6: Pretraining configurations for ViT and RN50.

Configuration	DecisionNCE-CLIP	AcTOL-CLIP
Training iterations	40,000	40,000
Optimizer	AdamW	AdamW
Learning rate	$1.0\times 10^{-4}$	$1.0\times 10^{-4}$
Frames per domain	160	160
Sampled frames	8	8
Weight decay	0	0

Appendix

Table 7: Pretraining configurations for DecisionNCE-CLIP and AcTOL-CLIP.

Figure 7: Effect of the number of sampled frames per video on the overall average success rate across L1 and L2 simulation tasks using R3M-RN50. Full denotes the default eight-frame configuration.

Configuration	ViT	RN50
Policy architecture	Modified RVT2
Stage 1 attention layers	8
Stage 2 attention layers	1
Attention dimension	512
Optimizer	LAMB
Base learning rate	$5\times 10^{-5}$

Appendix

Table 8: Downstream policy training configurations.

Encoder	Method	Knife	Fridge	Micro.	Laptop	Seat	Lamp Off	Lamp On	Book	Umb.	Grill	Bin	Avg.	Std.
D4R-ViT	Frozen	0	16.0	18.7	14.7	56.0	0	0	0	0	2.7	0	8.31	0.50
	+ HRP	1.3	22.7	10.7	32.0	45.3	2.67	2.67	0	1.3	18.7	0	10.56	1.67
	+ Ours	0	64.00	54.67	28.00	73.33	0	0	0	1.33	12.00	1.33	18.05	0.52
R3M-RN50	Frozen	1.3	29.3	32.0	13.3	73.3	1.3	1.3	1.3	0	10.7	0	12.62	0.25
	+ HRAlign	0	37.3	14.7	24.0	65.3	6.7	1.3	0	0	1.3	0	11.59	1.62
	+ Ours	6.67	64.00	33.33	46.67	61.33	1.33	0	0	0	5.33	0	16.82	1.24

Appendix

Table 9: Per-task success rates (%) on the 23 unseen AGNOSTOS tasks, averaged over three evaluation runs. Avg. reports the average success rate for each level, and Std. its standard deviation across runs. DNCE denotes DecisionNCE. ∗ denotes reproduced.

Figure 8: Human-to-robot task-progress retrieval examples from RH20T. Each row shows a human query and the nearest robot frames retrieved by Frozen R3M, HRAlign, and our method. Green checks indicate correct retrievals.

Figure 9: Human-to-robot task-progress retrieval examples from the self-collected zero-shot data. Each row shows a human query and the nearest robot frames retrieved by Frozen R3M, HRAlign, and our method. Green checks indicate correct retrievals. “None” denotes frames outside the annotated progress segments.

task_0010: S1: approach; S2: grasp and lift; S3: move handset away; S4: retreat
Task	Domain	Outside	Annotated	S1	S2	S3	S4
task_0010	Human	0–41; 69–85	42–68	42–46	46–53	53–64	64–68
task_0010	Robot	0–519	520–938	520–600	600–680	680–880	880–938
task_0015: S1: approach; S2: lift and aim; S3: striking; S4: return stick and release
task_0015	Human	0–43	44–69	44–47	47–55	55–64	64–69
task_0015	Robot	0–199	200–315	200–220	220–247	247–280	280–315

Appendix

Table 10: Frame annotations for the ten selected RH20T human–robot pairs. Each task has one human-robot pair. Stage names are task-specific.

carrot: S1: approach; S2: lift and carry to pot; S3: release and retreat
Task	Demo	Domain	Outside	Annotated	S1	S2	S3
carrot	001	Human	0–78; 155–164; 169–203	79–154; 165–168	79–119	119–154	165–168
carrot	001	Robot	247–299	0–246	0–155	155–226	226–246
carrot	002	Human	0–86; 169–203	87–168	87–113	113–156	156–168
carrot	002	Robot	253–299	0–252	0–142	142–232	232–252
carrot	003	Human	0–81; 174–218	82–173	82–108	108–157	157–173

Appendix

Table 11: Frame annotations for the four self-collected zero-shot tasks, with five human and five robot videos per task. Stage names are task-specific.

Split	# Tasks	Task categories
Training	18	Close jar; insert onto square peg; light bulb in; meat off grill; open drawer; place cups; place shape in shape sorter; place wine at rack location; push buttons; put groceries in cupboard; put item in drawer; put money in safe; reach and drag; slide block to color target; stack blocks; stack cups; sweep to dustpan of size; and turn tap.
Testing Level-1	13	Put toilet roll on stand; put knife on chopping board; close fridge; close microwave; close laptop lid; phone on base; toilet seat down; lamp off; lamp on; put books on bookshelf; put umbrella in umbrella stand; open grill; and put rubbish in bin.
Testing Level-2	10	Take USB out of computer; take lid off saucepan; take plate off colored dish rack; basketball in hoop; scoop with spatula; straighten rope; turn oven on; beat the buzz; water plants; and unplug charger.

Appendix

Table 12: Seen training and unseen testing task categories in the AGNOSTOS benchmark. The 23 testing categories are disjoint from the 18 training categories.

Figure 10: Real-world manipulation tasks evaluated using the Trossen WidowX AI arm.

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 323 papers

HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model
May 29, 2026Xiang Zhu, Puzhen Yuan, Yichen Liu +1Cross-Embodiment Robot Learning Robotic Manipulation
Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China
EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
Oct 14, 2026Meng Cao, Can Zhao, Jinjie Wei +12Robotic Manipulation Cross-Embodiment Robot Learning
Mohamed bin Zayed University of Artificial Intelligence · Shanghai Jiao Tong University · X Square Robot +2
TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation
Aug 27, 2026Jiarui Yang, Yehao Lu, Yuning Su +9Robot Learning Long-Horizon Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Zhejiang University, Hangzhou, China. · Simon Fraser University, Burnaby, BC, Canada. +1

May 29, 2026J/K move · Enter open · S save

HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model

Xiang Zhu, Puzhen Yuan, Yichen Liu, Jianyu Chen

Institute for Interdisciplinary Information Sciences, Tsinghua University, China · Shanghai Qi Zhi Institute, China

Cross-Embodiment Robot Learning Robotic Manipulation Vision-Language-Action Models Vision-Language Alignment

Open paper AskPDF arXiv

Learning generalizable vision-language-action (VLA) models from large-scale human videos is promising but challenging due to cross-embodiment discrepancies in both visual observations and executable actions. While latent action models reduce the action execution gap by learning action abstractions, they still rely on visual features. Thus, misaligned human and robot visual representations can lead to inconsistencies in policy inputs and induce domain-dependent latent actions, hindering effective co-training with human videos.To address this, we propose HARP, a human-robot aligned representation learning framework for more effective VLA pretraining from human videos. Specifically, HARP uses paired human-robot demonstrations as cross-embodiment bridges and abundant unpaired human and robot videos as a scalable dynamics supervision data source. It trains a robot-adapted visual encoder and a latent action model with manipulation-centric auxiliary cues and a source-relative pair-discriminative alignment loss, which adapts robot representations toward human semantics while preserving pair-level discrimination. The learned aligned vision encoder and latent action model provide a unified vision and action representation for VLA-style policy learning, where human and robot videos provide vision-language-to-latent-action supervision and a lightweight robot action head grounds latent actions into executable commands. Experiments on feature visualization, simulation, and real-world manipulation show improved human-robot alignment and downstream policy performance, achieving 4.481 average length on CALVIN ABC

\rightarrow

D and a 7.1 percentage-point improvement in real-world success rate over the strongest evaluated baseline. Code, videos, and demos are available at https://puzhenyuan.github.io/HARP-VLA-website/.

Abstract

Robot Learning

Explore similar work

HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation