A4A: Cross-Embodiment Transfer of Action-Oriented 4D Affordances from Human Demonstrations
Authors: Yifan Han, Litao Liu, Yuqi Gu, Ye Lu, Hanqing Wang, Sidney Wai, Ishaan Myrie, Qi Zhang, +2 more
Organizations: Shanghai Jiao Tong University · Rutgers University–New Brunswick · The Hong Kong University of Science and Technology (GZ) · Shanghai AI Laboratory · Nanyang Technological University
Human demonstrations contain rich manipulation knowledge, but it remains unclear what information can be transferred effectively to robot control. Existing affordance representations are typically formulated as 2D masks, 3D regions, contact points, or actionability scores, and therefore primarily identify where interaction may occur. However, effective manipulation also requires modeling how interaction-relevant geometry evolves during task execution. To bridge this gap, we introduce action-oriented 4D affordances, which represent the language-conditioned future trajectories of interaction-relevant 3D points. These trajectories capture task-conditioned geometric evolution rather than embodiment-specific actions, enabling transferable interaction priors across humans and robots. Based on this representation, we construct a large-scale action-oriented 4D affordance dataset from existing human--object interaction video data and complementary RGB-D demonstrations, and introduce A4A, an affordance-to-action framework that uses 4D affordance trajectory prediction to pretrain robot policies before manipulation fine-tuning. Experiments in both simulation and the real world validate the effectiveness of A4A, showing that pretraining with action-oriented 4D affordance data consistently improves the manipulation performance of diverse VLA policies. These results establish action-oriented 4D affordances as an effective cross-embodiment representation for transferring manipulation knowledge from human demonstrations to robot control.
Figures & tables
Figure 2: Affordance-to-action representation transfer. Guided by the short-horizon geometric alignment between interaction-relevant 4D point motion and robot end-effector motion, A4A pretrains a vision–language policy to predict future 3D point trajectories and transfers the learned representation to robot control through the policy’s native state and action interfaces.
Figure 3: Real-world manipulation. Representative A4A rollouts on placing a cup on a mug, opening a microwave, cutting a peach, and cooking rice.
Octo
OpenVLA
OpenVLA-OFT
π0
π0.5
Object
Base
A4A
Δ
Base
A4A
Δ
Base
A4A
Δ
Base
A4A
Δ
Base
A4A
Δ
Alphabet soup
36
64
+28
76
88
+12
100
100
0
76
88
+12
100
100
0
Cream cheese
20
56
+36
68
78
+10
98
98
0
86
92
+6
90
94
+4
Salad dressing
30
68
+38
84
92
+8
98
98
0
86
94
+8
100
98
-2
BBQ sauce
16
36
+20
42
54
+12
98
100
+2
56
78
+22
84
92
+8
Ketchup
12
48
+36
74
86
+12
100
98
-2
88
92
+4
100
100
0
Table 1: Per-task success rate (%) on LIBERO-Object. Each row picks up the named object and places it in the basket. Base denotes the corresponding downstream VLA policy, A4A denotes its A4A-instantiated variant, and Δ is the performance difference between them.
Method
Meat
Sweep
Turn Tap
Slide
Avg.
Image-BC (ViT)
0.0
0.0
16.0
0.0
4.0
C2FARM-BC
20.0
0.0
68.0
16.0
26.0
ManiGaussian
60.0
64.0
56.0
24.0
51.0
ARM4R
68.0
48.0
28.0
20.0
41.0
Ours w/o pretrain
56.0
44.0
44.0
12.0
39.0
Ours w/ pretrain
88.0
60.0
68.0
28.0
61.0
Table 2: Success rate (%) on RLBench. Meat, Sweep, Turn Tap, and Slide denote meat off grill, sweep to dustpan, turn tap, and slide block to target. Avg. is the mean success rate over the four tasks. Δ is the gain from 4D affordance pretraining, and comparison methods are quoted from prior work.
Cook Rice
Method
Microwave
Cup
Peach
Pot
Rice
Water
Avg.
Octo ( Team et al. 2024 )
7/10
3/10
3/10
2/10
4/10
3/10
36.7%
+ A4A
8/10
5/10
6/10
5/10
6/10
6/10
60.0%
OpenVLA ( Kim et al. 2024 )
6/10
6/10
4/10
3/10
6/10
4/10
48.3%
+ A4A
9/10
6/10
7/10
5/10
7/10
7/10
68.3%
OpenVLA-OFT ( Kim et al. 2025 )
9/10
7/10
7/10
5/10
8/10
7/10
71.7%
Table 3: Success on the physical Piper arm. Cells are successful trials out of 10. Microwave, Cup, and Peach denote open the microwave, place the cup on the mug, and cut the peach. Cook rice is reported as three subtasks, Pot (open the pot), Rice (pour rice), and Water (pour water). Avg. is the mean success rate over the six tasks. +A4A denotes the corresponding policy trained through our affordance-to-action framework.
Figure 4: Architecture-controlled comparison. Both variants use the same ARM4R architecture, robot demonstrations, finetuning recipe, and evaluation protocol. Gray uses the released initialization learned from generic scene-wide 4D point tracks; green uses the same architecture pretrained on our action-oriented 4D affordance dataset.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Visualization of actionable 4D affordances. The colored point flows represent the displacement of interaction-relevant 3D points from the current timestep to the next, with the color gradient indicating the temporal evolution of the motion.
Pour, cut, hang, sweep, lid removal, and additional articulated, tool-mediated, and object-to-object interactions
∼ 30K clips
Combined corpus
More than ten affordance categories with diverse translational and rotational interaction patterns
> 80K clips
Appendix
Table 4: Composition of the actionable 4D affordance corpus. Existing human-interaction datasets provide natural visual diversity, while the in-house RGB-D subset expands the affordance taxonomy and supplies geometrically consistent supervision for underrepresented manipulation motions.
Base policy
Native paradigm
Transferred module
4D pretraining objective
Robot-action objective
OpenVLA
Discrete autoregressive
Llama-2-7B decoder and output head
Point-token cross-entropy
Action-token cross-entropy
OpenVLA-OFT
Continuous regression
Llama-2-7B decoder
L1 /MSE point regression
L1 /MSE action regression
Octo
DDPM diffusion
Block-causal Transformer
Point-residual denoising
Action denoising
π0
End-to-end flow matching
Low-level action expert
Flow matching on point residuals
Flow matching on actions
π0.5
Hierarchical flow matching
Low-level action expert
Flow matching on point residuals
Flow matching on actions
Appendix
Table 5: Instantiation of A4A across five VLA action paradigms. Each policy retains its native prediction pathway and objective family while replacing the stage-specific state and output interfaces.
Policy
Image input
Camera views
Action formulation
Execution horizon
Inference steps
OpenVLA
224 px, center crop
Third-person
Autoregressive tokens
1
—
OpenVLA-OFT
224 px, center crop
Third-person + wrist
Continuous regression
8
—
Octo
256/128 px
Third-person + wrist
DDPM diffusion
4
Native
π0
256→224 px, padded
Third-person + wrist
Flow matching
50
10
π0.5
256→224 px, padded
Third-person + wrist
Flow matching
50
10
Appendix
Table 6: Policy-specific inference settings on LIBERO-Object. Execution horizon is the number of actions executed per policy query; inference steps apply only to iterative diffusion or flow-matching policies.
Figure 6: Qualitative results of stage-one actionable 4D affordance prediction. We compare the predicted and ground-truth point-motion targets across four representative interactions: closing an oven, cutting fruit, dragging a drawer, and pouring from one cup into another.
Institute of Automation, Chinese Academy of Sciences. · School of Artificial Intelligence, Shanghai Jiao Tong University. · 3South China University of Technology.
School of Artificial Intelligence, Donders Centre for Cognition, Department of Human-Centred Intelligent Systems, Radboud University, Nijmegen, The Netherlands.