PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Rikhat Akizhanov; Yangsong Zhang; Nikolai Kaliazin; Peter Wolf; Yoshihiko Nakamura; Pascal Fua; Fabio Pizzati; Ivan Laptev

cs.CVSep 30, 2026

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Authors: Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin, Peter Wolf, Yoshihiko Nakamura, Pascal Fua, Fabio Pizzati, Ivan Laptev

Organizations: MBZUAI · ETH Zürich · EPFL

Abstract

Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular video. Our approach augments a human reconstruction foundation model with learnable contact-force tokens and a temporal transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world videos. We also introduce a real-world climbing benchmark ForceWall with climbing videos and corresponding ground-truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from video.

Figures & tables

Figure 1: End-to-end joint estimation of human pose, contacts, and forces with PACT. Predictions from monocular videos show generalization across diverse activities and support configurations. White meshes show estimated poses; yellow arrows show predicted forces at estimated contacts.

Figure 2: PACT architecture. We add contact-force tokens to the frozen SAM 3D Body to account for additional outputs, that we process with a temporal transformer, alongside pose tokens enriched with world-related information. We use dedicated heads for contacts and forces, and predict pose offsets to smooth human motion learning from videos.

	ClimbingVideos			ForceWall			RICH
Method	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
BSTRO ( Huang et al., 2022 )	0.472	0.518	0.494	0.840	0.454	0.590	0.800	0.937	0.863
DECO ( Tripathi et al., 2023a )	0.435	0.594	0.502	0.821	0.548	0.658	0.736	0.751	0.744
InteractVLM ( Dwivedi et al., 2025 ) , “scene”	0.419	0.503	0.457	0.830	0.604	0.699	0.833	0.855	0.844
InteractVLM ( Dwivedi et al., 2025 ) , “wall”	0.738	0.849	0.789	0.820	0.676	0.741	0.785	0.669	0.722
GVHMR ( Shen et al., 2024 )	0.559	0.441	0.493	0.892	0.680	0.772	0.814	0.746	0.778

Table 1: Contacts benchmarks. We compare on ClimbingVideos , the sensor contacts of ForceWall , and the contact annotations of RICH. We are best on climbing datasets and highly competitive on RICH against methods that have seen its training set. For PACT , RICH is completely out-of-distribution. Best is bold , second is underlined .

	Method	MPJPE [mm] $\downarrow$	Force MAE [N] $\downarrow$					Force agreement			Residual $\downarrow$
	Method	MPJPE [mm] $\downarrow$	L. foot	R. foot	L. hand	R. hand	Avg.	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw]	Moment [bw m]
ForceWall	SAM 3D Body	42.8	–	–	–	–	–	–	–	–	–	–
	Li et al. (2019) ++	137.4	166.2	141.5	128.8	129.1	141.4	21.2	0.243	18.5	0.384	0.113
	PhysPT	86.7	193.8	147.4	170.3	159.8	167.8	21.4	0.169	28.0	0.785	0.080
	Inverse dynamics	42.7	200.7	162.2	166.0	192.4	180.3	28.8	0.007	21.9	0.024	0.040
	PACT	49.1	98.4	90.4	72.9	80.4	85.5	14.4	0.725	8.7	0.167	0.038

Table 2: Force estimation benchmarks. We outperform other methods in force MAE and correlation, reach the lowest moment residual and improve the poses of SAM 3D Body on Parkour.

Method	Input	Type	OC-Walking						OC-All
			Estimated mass			True mass			Estimated mass			True mass
			Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML
OpenCap ( Uhlrich et al., 2023 )	two views	simulation	–	–	–	12.2	3.1	1.2	–	–	–	11.4	3.5	3.8
OpenCap-mono ( Gilon et al., 2026 )	one view	learn + optim.	–	–	–	9.7	4.4	1.7	–	–	–	–	–	–
PhysPT ( Zhang et al., 2024 )	one view	learned	66.3	8.0	2.5	64.8	8.0	2.5	46.0	4.1	3.9	45.9	4.2	4.0
Li et al. (2019) ++	one view	optimization	33.9	9.0	4.1	35.0	8.6	4.0	37.0	6.1	4.3	36.5	5.8	4.3

Table 3: Evaluation on OpenCap data. We outperform all methods in all metrics except two-view OpenCap, where we are monocular. We report results with each method’s estimated mass and with the measured mass, and published numbers for OpenCap and OpenCap-mono.

Figure 3: Qualitative comparison on in-the-wild videos. Using downloaded videos from YouTube, we generate plausible forces (yellow), while our physics-based data annotation and optimization (red) fails. This shows that PACT generalizes beyond the tool used to annotate ClimbingVideos .

Architecture ( ClimbingVideos training)
Setup	Pose	Contacts	Forces				Residual
Setup	MPJPE [mm] $\downarrow$	F1 $\uparrow$	MAE [N] $\downarrow$	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw] $\downarrow$	Moment [bw m] $\downarrow$
Per-frame tokens and heads	72.9	0.895	120.1	22.1	0.472	12.2	0.471	0.096
+ temporal transformer	50.5	0.914	102.2	17.3	0.649	10.2	0.422	0.075
+ world-informed poses	61.2	0.914	117.2	21.2	0.504	11.6	0.291	0.075
+ iterative refinement	57.7	0.917	112.0	19.9	0.578	11.0	0.286	0.073

Table 4: Ablations on ForceWall . Architecture additions are cumulative (Appendix E.1 ), with the physics loss completing the full model. The training-data ablation uses the full model. The full model reaches the best force accuracy.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Annotation pipeline. Foundation models reconstruct the scene, the cameras and an initial body, and a physics-based optimization labels the contacts, the motion and the forces.

Figure 5: Contact points. The $K=35$ contact points on the SMPL-X body, front and back, with the hand and foot points enlarged. Each point carries a frame whose normal $\mathbf{n}_{t,k}$ defines its friction cone.

Term	Weight	Root	Pass 1	Pass 2	Dynamics
Reprojection to the initial joints	$w_{\text{init}}$	30	30	30	30
Reprojection to the Sapiens2 keypoints	$w_{\text{2D}}$	8	8	8	8
Chamfer distance to the human depth cloud	$w_{\text{ch}}$	10	10	0	0
Pose prior	$w_{\text{pose}}$	300	300	300	300
Root orientation prior	$w_{\text{orient}}$	50	50	50	50
Smoothness of the root pose	$w^{\text{r}}_{n}$	24 / 300 / 300	400 / 300 / 300	24 / 300 / 300	24 / 300 / 300

Appendix

Table 5: Auto-labeling weights. Terms and weights of Equation 14 to Equation 18 and of Equation 8 and Equation 20 in the root pass, the two kinematic passes and the dynamics pass. Grouped cells list the weights of the second, third and fourth differences for smoothness, of the first to third differences for the forces and of the first and second for the torques.

	hip, knee, spine	ankle, foot	neck, head	collar, shoulder	elbow	wrist	fingers
Max. torque [N m]	200	118	33	80	61	12	8
$\mu_{j}$	1.0	1.7	6.0	2.5	3.3	17	25

Appendix

Table 6: Torque multipliers. The multipliers

\mu_{j}

per joint group and the maximum voluntary torque they encode.

Loss	Definition	Weight	Conf.
Per-frame body $\mathcal{L}_{\text{body}}$ (SMPL-X and camera heads)
2D joints	Huber on the full-frame projection, crop units, $\delta=0.05$	10
3D joints	Huber, pelvis-relative, $\delta=0.1$ m	5
Root, body and finger rotations	squared error on 6D	1, 1, 0.5
Shape	squared error	0.1
Camera	Huber on the weak-perspective parameters, $\delta=0.05$	1

Appendix

Table 7: Training losses. Terms and weights. Confidence-weighted terms are marked in the last column, and the contact, force, trajectory and gravity terms also supervise the intermediate blocks at weight 0.5. Huber losses are given with their scale.

	Sources	Scenes	Frames	Labels
ClimbingVideos train	149	1,419	460k	pseudo
ClimbingVideos test	42	109	31k	pseudo, manual contacts
BEDLAM2 ( Tesch et al., 2025 )	59 jobs	18,076	5.5M	true pose, pseudo contacts and forces
ForceWall	14	68	29.6k	sensors, multi-view pose
RICH ( Huang et al., 2022 )	8 subjects	164	70.2k	contact labels
Parkour ( Li et al., 2019 )	–	28	2.3k	motion capture, plates and bar

Appendix

Table 8: Datasets. Data used in this paper.

Stage	Count
Videos returned by keyword search (“bouldering gym”, “moonboard climbing” and the like)	782
Duration between 10 s and 20 min	591
Thumbnail and metadata VLM gate	584
Downloaded at up to 1080p (48 h of video)	581
Scenes after shot detection, between 100 and 500 frames	16,152
Full-body pose gate: in half of the sampled frames, a person with 60% of the 17 COCO keypoints visible, including an ankle, a shoulder and a hip, spanning 40% of the frame height	4,612

Appendix

Table 9: Curation funnel. Curation of ClimbingVideos . Counts are YouTube videos until the scene split and scenes afterwards.

Figure 6: ClimbingVideos samples. One frame from each of 30 scenes, spanning gyms and competition walls, slabs to steep overhangs, static and moving cameras, and varied lighting and clothing.

Figure 7: Contact annotation tool. The interface used to annotate contacts on the ClimbingVideos test split.

Panel of Figure 8 — Figure 8: ForceWall hardware. (a) The four instrumented panels with their holds. (b) The tilted frame that carries the panels, with a zoom on one of the three-axis load cells between a panel and the frame.

Window	Pose	Contacts	Forces				Residual
Window	MPJPE [mm] $\downarrow$	F1 $\uparrow$	MAE [N] $\downarrow$	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw] $\downarrow$	Moment [bw m] $\downarrow$
No mask (240-frame window)	58.1	0.940	86.3	15.5	0.756	8.2	0.153	0.040
$\pm 0.25\,\mathrm{s}$ (default)	56.6	0.936	87.5	15.0	0.729	8.6	0.155	0.037
$\pm 0.125\,\mathrm{s}$	56.4	0.933	88.2	15.1	0.721	8.7	0.157	0.038

Appendix

Table 10: Temporal attention window. On ForceWall , with the model trained on ClimbingVideos and the protocol of Table 4 . The window is the receptive field of the three blocks together.

	ClimbingVideos			ForceWall
Method	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
Optimization	0.938	0.850	0.892	0.902	0.931	0.916
PACT	0.948	0.895	0.921	0.896	0.980	0.936

Appendix

Table 11: Optimization vs. PACT : contacts. On ClimbingVideos and ForceWall , with the protocol of Table 1 .

	Method	MPJPE [mm] $\downarrow$	Force MAE [N] $\downarrow$					Force agreement			Residual $\downarrow$
	Method	MPJPE [mm] $\downarrow$	L. foot	R. foot	L. hand	R. hand	Avg.	Angle [ ∘ ] $\downarrow$	Corr $\uparrow$	Share [pp] $\downarrow$	Force [bw]	Moment [bw m]
ForceWall	Optimization	57.5	101.8	101.8	89.2	84.9	94.5	14.7	0.654	10.8	0.087	0.019
ForceWall	PACT	49.1	98.4	90.4	72.9	80.4	85.5	14.4	0.725	8.7	0.167	0.038
Parkour	Optimization	65.7	127.3	124.8	91.5	65.7	102.3	20.7	0.666	15.8	0.089	0.019
Parkour	PACT	66.3	117.1	148.4	77.9	66.3	102.4	16.7	0.712	14.6	0.283	0.044

Appendix

Table 12: Optimization vs. PACT : forces. On ForceWall and on the parkour sequences of Li et al. (2019) , with the protocol of Table 2 . The residual of the optimization is that of its own solver, which minimizes it.

Method	OC-Walking						OC-All
	Estimated mass			True mass			Estimated mass			True mass
	Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML	Vert.	AP	ML
Optimization	19.2	5.4	2.5	15.2	5.6	2.6	13.6	3.9	3.8	11.5	4.2	3.8
PACT	11.8	3.0	1.5	7.8	3.3	1.6	11.6	3.4	2.3	9.7	3.7	2.4

Appendix

Table 13: Optimization vs. PACT : OpenCap. Ground reaction forces with the protocol of Table 3 : mean absolute error per foot in percent of body weight along the vertical, anteroposterior (AP) and mediolateral (ML) axes.

Figure 9: Per-limb forces on ForceWall . One clip: instrumented panels (black), the optimization (orange) and the learned model (blue), in newtons with the reconstructed mass. The dotted line marks the frame shown on the left.

Figure 10: Ground reaction forces on OpenCap. Both feet on one walking trial: force plates (black), the optimization (orange) and the learned model (blue), in percent of body weight with the measured mass.

Figure 11: Qualitative results in 3D. Input frame and two rendered views of the estimated body with the predicted contact forces.

Figure 12: Qualitative results over time. Consecutive frames of one clip per row with the estimated poses and the predicted contact forces.

Explore similar work

From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video
Jun 4, 2026Jessy LauerBiomechanics Force
Harvard
UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video
Apr 21, 2026Tanuj Sur, Shashank Tripathi, Nikos Athanasiou +4Scene Reconstruction Monocular Video
1National University of Singapore · 2Max Planck Institute for Intelligent Systems, Tübingen, Germany
Recovering Physically Plausible Human-Object Interactions from Monocular Videos
Jun 3, 2026Dingbang Huang, Etienne Vouga, Qixing Huang +1Monocular Video Motion
University of Texas at Austin

Jun 4, 2026, cs.CVJ/K move · Enter open · S save

From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video

Jessy Lauer

Harvard

Biomechanics Force Motion Capture Monocular Video Newton

Open paper AskPDF arXiv

Joint contact forces govern implant longevity, cartilage health, and rehabilitation outcomes, shaping who develops osteoarthritis, who recovers well from joint replacement, and who benefits from biomechanical interventions. Yet they remain measurable only invasively, in a few dozen patients with instrumented implants. I present a physics-free pipeline to predict instantaneous 3D hip and knee contact forces from an uncalibrated monocular video: no markers, force plates, electromyography, subject-specific imaging, or musculoskeletal model. Parametric body meshes are recovered per frame, encoded as kinematic features, and decoded into forces by a transformer whose pose stream is adaptively modulated at every layer by body shape, joint, side, activity text, and self-supervised video tokens (V-JEPA 2), unifying hip and knee in a single model. Under leave-one-subject-out cross-validation across 26 patients and 25 activity categories from the in vivo OrthoLoad database, the pipeline matches the accuracy of subject-specific musculoskeletal simulations (

0.32 \pm 0.08

BW RMSE for hip;

0.23 \pm 0.03

BW for knee) and resolves peak force changes smaller than those reported for gait retraining and osteoarthritis progression. Applied zero-shot to an independent instrumented cohort, it rivals or outperforms prior published methods. Even without curated activity labels, video features alone preserve accuracy and enable end-to-end inference on raw footage. Driven by the predictor, a generative motion prior produces biomechanically plausible variants with reduced peak loading, rediscovering strategies from the predictive simulation literature. This pipeline establishes uncalibrated monocular video as a viable modality for estimating joint loading, opening a path toward retrospective analysis of archived clinical recordings, primary-care screening, and at-home rehabilitation tracking.

Abstract

Explore similar work

From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video

UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video

Recovering Physically Plausible Human-Object Interactions from Monocular Videos