PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Rikhat Akizhanov; Yangsong Zhang; Nikolai Kaliazin; Peter Wolf; Yoshihiko Nakamura; Pascal Fua; Fabio Pizzati; Ivan Laptev

cs.CVSep 30, 2026

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Authors: Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin, Peter Wolf, Yoshihiko Nakamura, Pascal Fua, Fabio Pizzati, Ivan Laptev

Organizations: MBZUAI · ETH Zürich · EPFL

Abstract

Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular video. Our approach augments a human reconstruction foundation model with learnable contact-force tokens and a temporal transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world videos. We also introduce a real-world climbing benchmark ForceWall with climbing videos and corresponding ground-truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from video.

Figures & tables

Figure 1: End-to-end joint estimation of human pose, contacts, and forces with PACT. Predictions from monocular videos show generalization across diverse activities and support configurations. White meshes show estimated poses; yellow arrows show predicted forces at estimated contacts.

Figure 2: PACT architecture. We add contact-force tokens to the frozen SAM 3D Body to account for additional outputs, that we process with a temporal transformer, alongside pose tokens enriched with world-related information. We use dedicated heads for contacts and forces, and predict pose offsets to smooth human motion learning from videos.

	ClimbingVideos			ForceWall			RICH
Method	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
BSTRO ( Huang et al., 2022 )	0.472	0.518	0.494	0.840	0.454	0.590	0.800	0.937	0.863
DECO ( Tripathi et al., 2023a )	0.435	0.594	0.502	0.821	0.548	0.658	0.736	0.751	0.744
InteractVLM ( Dwivedi et al., 2025 ) , “scene”	0.419	0.503	0.457	0.830	0.604	0.699	0.833	0.855	0.844
InteractVLM ( Dwivedi et al., 2025 ) , “wall”	0.738	0.849	0.789	0.820	0.676	0.741	0.785	0.669	0.722
GVHMR ( Shen et al., 2024 )	0.559	0.441	0.493	0.892	0.680	0.772	0.814	0.746	0.778

Table 1: Contacts benchmarks. We compare on ClimbingVideos , the sensor contacts of ForceWall , and the contact annotations of RICH. We are best on climbing datasets and highly competitive on RICH against methods that have seen its training set. For PACT , RICH is completely out-of-distribution. Best is bold , second is underlined .

	Method	MPJPE [mm] $\downarrow$	Force MAE [N] $\downarrow$					Force agreement			Residual $\downarrow$
	Method	MPJPE [mm] $\downarrow$	L. foot	R. foot	L. hand	R. hand	Avg.	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw]	Moment [bw m]
ForceWall	SAM 3D Body	42.8	–	–	–	–	–	–	–	–	–	–
	Li et al. (2019) ++	137.4	166.2	141.5	128.8	129.1	141.4	21.2	0.243	18.5	0.384	0.113
	PhysPT	86.7	193.8	147.4	170.3	159.8	167.8	21.4	0.169	28.0	0.785	0.080
	Inverse dynamics	42.7	200.7	162.2	166.0	192.4	180.3	28.8	0.007	21.9	0.024	0.040
	PACT	49.1	98.4	90.4	72.9	80.4	85.5	14.4	0.725	8.7	0.167	0.038

Table 2: Force estimation benchmarks. We outperform other methods in force MAE and correlation, reach the lowest moment residual and improve the poses of SAM 3D Body on Parkour.

Method	Input	Type	OC-Walking						OC-All
			Estimated mass			True mass			Estimated mass			True mass
			Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML
OpenCap ( Uhlrich et al., 2023 )	two views	simulation	–	–	–	12.2	3.1	1.2	–	–	–	11.4	3.5	3.8
OpenCap-mono ( Gilon et al., 2026 )	one view	learn + optim.	–	–	–	9.7	4.4	1.7	–	–	–	–	–	–
PhysPT ( Zhang et al., 2024 )	one view	learned	66.3	8.0	2.5	64.8	8.0	2.5	46.0	4.1	3.9	45.9	4.2	4.0
Li et al. (2019) ++	one view	optimization	33.9	9.0	4.1	35.0	8.6	4.0	37.0	6.1	4.3	36.5	5.8	4.3

Table 3: Evaluation on OpenCap data. We outperform all methods in all metrics except two-view OpenCap, where we are monocular. We report results with each method’s estimated mass and with the measured mass, and published numbers for OpenCap and OpenCap-mono.

Figure 3: Qualitative comparison on in-the-wild videos. Using downloaded videos from YouTube, we generate plausible forces (yellow), while our physics-based data annotation and optimization (red) fails. This shows that PACT generalizes beyond the tool used to annotate ClimbingVideos .

Architecture ( ClimbingVideos training)
Setup	Pose	Contacts	Forces				Residual
Setup	MPJPE [mm] $\downarrow$	F1 $\uparrow$	MAE [N] $\downarrow$	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw] $\downarrow$	Moment [bw m] $\downarrow$
Per-frame tokens and heads	72.9	0.895	120.1	22.1	0.472	12.2	0.471	0.096
+ temporal transformer	50.5	0.914	102.2	17.3	0.649	10.2	0.422	0.075
+ world-informed poses	61.2	0.914	117.2	21.2	0.504	11.6	0.291	0.075
+ iterative refinement	57.7	0.917	112.0	19.9	0.578	11.0	0.286	0.073

Table 4: Ablations on ForceWall . Architecture additions are cumulative (Appendix E.1 ), with the physics loss completing the full model. The training-data ablation uses the full model. The full model reaches the best force accuracy.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Annotation pipeline. Foundation models reconstruct the scene, the cameras and an initial body, and a physics-based optimization labels the contacts, the motion and the forces.

Figure 5: Contact points. The $K=35$ contact points on the SMPL-X body, front and back, with the hand and foot points enlarged. Each point carries a frame whose normal $\mathbf{n}_{t,k}$ defines its friction cone.

Term	Weight	Root	Pass 1	Pass 2	Dynamics
Reprojection to the initial joints	$w_{\text{init}}$	30	30	30	30
Reprojection to the Sapiens2 keypoints	$w_{\text{2D}}$	8	8	8	8
Chamfer distance to the human depth cloud	$w_{\text{ch}}$	10	10	0	0
Pose prior	$w_{\text{pose}}$	300	300	300	300
Root orientation prior	$w_{\text{orient}}$	50	50	50	50
Smoothness of the root pose	$w^{\text{r}}_{n}$	24 / 300 / 300	400 / 300 / 300	24 / 300 / 300	24 / 300 / 300

Appendix

Table 5: Auto-labeling weights. Terms and weights of Equation 14 to Equation 18 and of Equation 8 and Equation 20 in the root pass, the two kinematic passes and the dynamics pass. Grouped cells list the weights of the second, third and fourth differences for smoothness, of the first to third differences for the forces and of the first and second for the torques.

	hip, knee, spine	ankle, foot	neck, head	collar, shoulder	elbow	wrist	fingers
Max. torque [N m]	200	118	33	80	61	12	8
$\mu_{j}$	1.0	1.7	6.0	2.5	3.3	17	25

Appendix

Table 6: Torque multipliers. The multipliers

\mu_{j}

per joint group and the maximum voluntary torque they encode.

Loss	Definition	Weight	Conf.
Per-frame body $\mathcal{L}_{\text{body}}$ (SMPL-X and camera heads)
2D joints	Huber on the full-frame projection, crop units, $\delta=0.05$	10
3D joints	Huber, pelvis-relative, $\delta=0.1$ m	5
Root, body and finger rotations	squared error on 6D	1, 1, 0.5
Shape	squared error	0.1
Camera	Huber on the weak-perspective parameters, $\delta=0.05$	1

Appendix

Table 7: Training losses. Terms and weights. Confidence-weighted terms are marked in the last column, and the contact, force, trajectory and gravity terms also supervise the intermediate blocks at weight 0.5. Huber losses are given with their scale.

	Sources	Scenes	Frames	Labels
ClimbingVideos train	149	1,419	460k	pseudo
ClimbingVideos test	42	109	31k	pseudo, manual contacts
BEDLAM2 ( Tesch et al., 2025 )	59 jobs	18,076	5.5M	true pose, pseudo contacts and forces
ForceWall	14	68	29.6k	sensors, multi-view pose
RICH ( Huang et al., 2022 )	8 subjects	164	70.2k	contact labels
Parkour ( Li et al., 2019 )	–	28	2.3k	motion capture, plates and bar

Appendix

Table 8: Datasets. Data used in this paper.

Stage	Count
Videos returned by keyword search (“bouldering gym”, “moonboard climbing” and the like)	782
Duration between 10 s and 20 min	591
Thumbnail and metadata VLM gate	584
Downloaded at up to 1080p (48 h of video)	581
Scenes after shot detection, between 100 and 500 frames	16,152
Full-body pose gate: in half of the sampled frames, a person with 60% of the 17 COCO keypoints visible, including an ankle, a shoulder and a hip, spanning 40% of the frame height	4,612

Appendix

Table 9: Curation funnel. Curation of ClimbingVideos . Counts are YouTube videos until the scene split and scenes afterwards.

Figure 6: ClimbingVideos samples. One frame from each of 30 scenes, spanning gyms and competition walls, slabs to steep overhangs, static and moving cameras, and varied lighting and clothing.

Figure 7: Contact annotation tool. The interface used to annotate contacts on the ClimbingVideos test split.

Panel of Figure 8 — Figure 8: ForceWall hardware. (a) The four instrumented panels with their holds. (b) The tilted frame that carries the panels, with a zoom on one of the three-axis load cells between a panel and the frame.

Window	Pose	Contacts	Forces				Residual
Window	MPJPE [mm] $\downarrow$	F1 $\uparrow$	MAE [N] $\downarrow$	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw] $\downarrow$	Moment [bw m] $\downarrow$
No mask (240-frame window)	58.1	0.940	86.3	15.5	0.756	8.2	0.153	0.040
$\pm 0.25\,\mathrm{s}$ (default)	56.6	0.936	87.5	15.0	0.729	8.6	0.155	0.037
$\pm 0.125\,\mathrm{s}$	56.4	0.933	88.2	15.1	0.721	8.7	0.157	0.038

Appendix

Table 10: Temporal attention window. On ForceWall , with the model trained on ClimbingVideos and the protocol of Table 4 . The window is the receptive field of the three blocks together.

	ClimbingVideos			ForceWall
Method	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
Optimization	0.938	0.850	0.892	0.902	0.931	0.916
PACT	0.948	0.895	0.921	0.896	0.980	0.936

Appendix

Table 11: Optimization vs. PACT : contacts. On ClimbingVideos and ForceWall , with the protocol of Table 1 .

	Method	MPJPE [mm] $\downarrow$	Force MAE [N] $\downarrow$					Force agreement			Residual $\downarrow$
	Method	MPJPE [mm] $\downarrow$	L. foot	R. foot	L. hand	R. hand	Avg.	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw]	Moment [bw m]
ForceWall	Optimization	57.5	101.8	101.8	89.2	84.9	94.5	14.7	0.654	10.8	0.087	0.019
ForceWall	PACT	49.1	98.4	90.4	72.9	80.4	85.5	14.4	0.725	8.7	0.167	0.038
Parkour	Optimization	65.7	127.3	124.8	91.5	65.7	102.3	20.7	0.666	15.8	0.089	0.019
Parkour	PACT	66.3	117.1	148.4	77.9	66.3	102.4	16.7	0.712	14.6	0.283	0.044

Appendix

Table 12: Optimization vs. PACT : forces. On ForceWall and on the parkour sequences of Li et al. (2019) , with the protocol of Table 2 . The residual of the optimization is that of its own solver, which minimizes it.

Method	OC-Walking						OC-All
	Estimated mass			True mass			Estimated mass			True mass
	Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML
Optimization	19.2	5.4	2.5	15.2	5.6	2.6	13.6	3.9	3.8	11.5	4.2	3.8
PACT	11.8	3.0	1.5	7.8	3.3	1.6	11.6	3.4	2.3	9.7	3.7	2.4

Appendix

Table 13: Optimization vs. PACT : OpenCap. Ground reaction forces with the protocol of Table 3 : mean absolute error per foot in percent of body weight along the vertical, anteroposterior (AP) and mediolateral (ML) axes.

Figure 9: Per-limb forces on ForceWall . One clip: instrumented panels (black), the optimization (orange) and the learned model (blue), in newtons with the reconstructed mass. The dotted line marks the frame shown on the left.

Figure 10: Ground reaction forces on OpenCap. Both feet on one walking trial: force plates (black), the optimization (orange) and the learned model (blue), in percent of body weight with the measured mass.

Figure 11: Qualitative results in 3D. Input frame and two rendered views of the estimated body with the predicted contact forces.

Figure 12: Qualitative results over time. Consecutive frames of one clip per row with the estimated poses and the predicted contact forces.

Explore similar work

Jun 4, 2026cs.CV

From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video

Joint contact forces govern implant longevity, cartilage health, and rehabilitation outcomes, shaping who develops osteoarthritis, who recovers well from joint replacement, and who benefits from biomechanical interventions. Yet they remain measurable only invasively, in a few dozen patients with instrumented implants. I present a physics-free pipeline to predict instantaneous 3D hip and knee contact forces from an uncalibrated monocular video: no markers, force plates, electromyography, subject-specific imaging, or musculoskeletal model. Parametric body meshes are recovered per frame, encoded as kinematic features, and decoded into forces by a transformer whose pose stream is adaptively modulated at every layer by body shape, joint, side, activity text, and self-supervised video tokens (V-JEPA 2), unifying hip and knee in a single model. Under leave-one-subject-out cross-validation across 26 patients and 25 activity categories from the in vivo OrthoLoad database, the pipeline matches the accuracy of subject-specific musculoskeletal simulations (

0.32 \pm 0.08

BW RMSE for hip;

0.23 \pm 0.03

BW for knee) and resolves peak force changes smaller than those reported for gait retraining and osteoarthritis progression. Applied zero-shot to an independent instrumented cohort, it rivals or outperforms prior published methods. Even without curated activity labels, video features alone preserve accuracy and enable end-to-end inference on raw footage. Driven by the predictor, a generative motion prior produces biomechanically plausible variants with reduced peak loading, rediscovering strategies from the predictive simulation literature. This pipeline establishes uncalibrated monocular video as a viable modality for estimating joint loading, opening a path toward retrospective analysis of archived clinical recordings, primary-care screening, and at-home rehabilitation tracking.

Jessy Lauer

Harvard

Biomechanics Force Motion Capture Monocular Video Newton

Ask PDF

Apr 21, 2026cs.CV

UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video

We introduce UniCon3R, a unified feed-forward framework for online human-scene 4D reconstruction from monocular video. Current feed-forward human-scene reconstruction methods suffer from artifacts, where bodies float above the ground or penetrate parts of the scene. A key reason is the lack of effective interaction modelling between the human and the environment. Our goal is to exploit contact between the human and the scene during inference to actively improve the human mesh reconstruction. To that end, we explicitly model interaction by inferring 4D contact from the human pose and scene geometry and use the contact as a corrective cue for generating the pose. This enables UniCon3R to jointly recover scene geometry and spatially aligned 4D humans within the scene. Experiments on standard human-centric video benchmarks show that UniCon3R outperforms state-of-the-art baselines on physical plausibility and global human motion estimation while preserving fast, feed-forward inference speeds. The results validate our central claim: contact serves as a powerful internal prior, thus establishing a new paradigm for physically grounded joint human-scene reconstruction. Project page is available at https://surtantheta.github.io/UniCon3R .

Tanuj Sur, Shashank Tripathi, Nikos Athanasiou +4

1National University of Singapore · 2Max Planck Institute for Intelligent Systems, Tübingen, Germany

Scene Reconstruction Monocular Video

Ask PDF

Jun 3, 2026cs.CV

Recovering Physically Plausible Human-Object Interactions from Monocular Videos

In this paper, we propose RePHO, a method to reconstruct physically plausible human-object interactions (HOI) from monocular videos. While existing kinematic-based approaches produce visually plausible motion, they often result in physically implausible artifacts such as interpenetration and object floating. To overcome these issues, we introduce a physics-guided reconstruction framework. We begin with a kinematic estimate and then refine it by training a policy with reinforcement learning (RL). This policy is optimized to reproduce the interaction in a physics simulator. Because kinematic estimates are typically noisy, naive RL training can fail. Therefore, we propose an adaptive sampling strategy with a dual self-updating mechanism that can identify the frames with the most informative and reliable kinematic reconstruction. Our process progressively improves reconstruction quality and yields physically consistent HOI sequences. We demonstrate our approach on two standard HOI benchmarks and achieve clear improvements in physical plausibility metrics over state-of-the-art methods. Project Page: https://dingbang777.github.io/RePHO/

Dingbang Huang, Etienne Vouga, Qixing Huang +1

University of Texas at Austin

Monocular Video Motion

Ask PDF