cs.CVMar 26, 2026
SaveTHFM: A Unified Video Foundation Model for 4D Human Perception and Beyond
Organizations: Google DeepMind
Abstract
We present THFM, a unified video foundation model for human-centric perception that jointly addresses dense tasks (depth, normals, segmentation, dense pose) and sparse tasks (2d/3d keypoint estimation) within a single architecture. THFM is derived from a pretrained text-to-video diffusion model, repurposed as a single-forward-pass perception model and augmented with learnable tokens for sparse predictions. Modulated by the text prompt, our single unified model is capable of performing various perception tasks. Crucially, our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks despite being trained exclusively on synthetic data (i.e.~without training on real-world or benchmark specific data). We further highlight intriguing emergent properties of our model, which we attribute to the underlying diffusion-based video representation. For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals -- a capability that hasn't been demonstrated in the past.
Figures & tables
Figure 1: THFM is a single, unified video perception model with SOTA performance on a multitude of output modalities . From left to right we show: a frame from the input video, estimated surface normals, depth, segmentation, dense human semantics [ 68 , 57 ] , 2d and 3d keypoints. Our approach has been trained on videos of people generated synthetically, yet it generalizes to real videos both for people as well as other categories such as animals and anthropomorphic characters.
| Method | Number of tasks | One model for all tasks | Sparse tasks | Task by Text Prompt | Temporal smooth. | 1-Step Inference |
|---|---|---|---|---|---|---|
| Diffusion-based models (single-frame) | ||||||
| Marigold [ 24 ] | 1 | ✗ | ✗ | ✗ | ✗ | ✗ |
| GeoWizard [ 14 ] | 1 | ✗ | ✗ | ✗ | ✗ | ✗ |
| Marigold [ 25 ] | 3 | ✗ | ✗ | ✗ | ✗ | ✗ |
| Martin Garcia et al. [46] | 1 | ✗ | ✗ | ✗ | ✗ | ✔ |
| GenPercept [ 73 ] | 4 | ✗ | ✗ | ✗ | ✗ | ✔ |
Table 1: Systematic comparison of THFM to related work across key design dimensions. We compare methods along the number of supported tasks, whether a single unified model handles all tasks, support for sparse prediction tasks (e.g. articulated 2d/3d pose estimation, which require continuous predictions outside the pixel space), whether task selection is driven by text prompts, temporal smoothness of predictions across video frames, and single-step (non-iterative) inference capability. To the best of our knowledge, THFM is the first model that unifies multiple sparse and dense video perception tasks within a single architecture, while simultaneously leveraging text-based task conditioning, producing temporally smooth outputs, and operating in a single forward pass.
Figure 2: Method overview of THFM, a simple yet powerful architecture adapted from text-to-video diffusion models. Given an input video and a text prompting specifying the desired output, our unified model, trained only on synthetic data, is capable of performing a wide range of dense and sparse perception tasks, with a single forward-pass of the model. The dense vision tasks are unified in the RGB ambient space where supervision can be applied in both latent space and RGB ambient space, and the sparse vision tasks are realized by adding learnable tokens as additional inputs to the diffusion transformer (DiT).
Figure 3: 3D Pose Estimation . Example 3D pose estimation results on challenging skiing (top) and snowboarding (bottom) video. We show every 10-th frame for a subsequence of the input video. Compared to the existing state-of-the-art 3D human pose estimation methods THFM model takes a full video frame as input and does not require pre-processing steps such as person detection or 2d keypoint estimation.
| Method | Hi4D | ||
|---|---|---|---|
| Angular Error (°) | % Within | ||
| Mean | Median | 11.25° / 22.5° / 30° | |
| PIFuHD [ 58 ] | 22.39 | 19.26 | 23.0 / 60.1 / 77.0 |
| HDNet [ 21 ] | 28.60 | 26.85 | 19.1 / 57.9 / 70.1 |
| ICON [ 71 ] | 20.18 | 17.52 | 26.8 / 66.3 / 82.7 |
| ECON [ 72 ] | 18.46 | 16.47 | 29.3 / 68.1 / 84.9 |
Table 2: Surface normal estimation results on the Hi4D [ 79 ] dataset. We follow the evaluation protocol of Sapiens [ 27 ] and DAVID [ 59 ] . We report mean and median angular error (lower is better) alongside the percentage of pixels within angular thresholds (higher is better). Baseline results are taken from DAVID [ 59 , 27 ] . The bottom rows present ablations of THFM across key design choices: decoder training space (latent vs. ambient), input timestep selection, number of jointly trained tasks, and token masking.
| Method | Goliath-Face | Goliath-UpperBody | Goliath-FullBody | ||||
|---|---|---|---|---|---|---|---|
| RMSE | AbsRel | RMSE | AbsRel | RMSE | AbsRel | ||
| Results from DAVID | |||||||
| MiDaS-DPT_L [ 53 ] | 0.224 | 0.016 | 0.553 | 0.015 | 0.973 | 0.027 | |
| DepthAnythingV2-L [ 76 ] | 0.229 | 0.017 | 0.492 | 0.014 | 1.039 | 0.029 | |
| Depth-Pro [ 6 ] | 0.295 | 0.020 | 0.442 | 0.010 | 0.723 | 0.016 | |
| Sapiens-0.3B [ 27 ] | 0.179 | 0.012 | 0.368 | 0.010 | 0.690 | 0.019 | |
Table 3: Depth estimation results on the Goliath [ 47 ] dataset, evaluated across three body-crop settings (Face, UpperBody, FullBody) using RMSE and AbsRel metrics (lower is better). We report two groups of results: “Results from DAVID” reproduces numbers as reported in DAVID [ 59 ] ; “Our Eval” provides our re-evaluation of publicly available models using the same data splits (12 cameras, 16 frames, 4 subjects) to ensure a consistent comparison protocol. The bottom rows show ablations of THFM variants.
| Method | VideoMatte | |||
|---|---|---|---|---|
| MAD | MSE | Grad | dtSSD | |
| MODNet + FGF [ 26 ] | 11.04 | 5.42 | 15.80 | 3.10 |
| RVM [ 42 ] | 5.64 | 1.07 | 9.80 | 1.84 |
| Decoder | 4 Dense | 2 Sparse | Token | THFM Variant | |||
| Training | Task | Task | Mask | ||||
| ✗ ✔ ✗ ✗ | 6.46 | 2.14 | 0.945 | 3.31 |
Table 4: Evaluation of our approach in soft foreground segmentation on PhotoMatte85 [ 41 ] , PPM100 [ 26 ] , and VideoMatte [ 41 ] dataset (static split). PhotoMatte85 and Conn in PPM-100 taken from DAVID [ 59 ] and MODNet [ 26 ] . VideoMatte results taken from RVM [ 42 ] .
| RICH (24) | EMDB (24) | ||||||
| Models | PA-MPJPE | MPJPE | Accel | PA-MPJPE | MPJPE | Accel | |
| per-frame | CLIFF [ 39 ] | 56.6 | 102.6 | 22.4 | 68.1 | 103.3 | 24.5 |
| HybrIK [ 36 ] | 56.4 | 96.8 | – | 65.6 | 122.2 | – | |
| HMR2.0 [ 15 ] | 48.1 | 96.0 | 18.8 | 60.6 | 98.0 | 19.8 | |
| ReFit [ 69 ] | 47.9 | 80.7 | 17.1 | 58.6 | 88.0 | 20.7 | |
| temporal | VIBE [ 32 ] | 68.4 | 120.5 | 21.8 | 81.4 | 125.9 | 26.6 |
Table 5: We evaluate 3D keypoints reconstruction on RICH and EMDB-1 (left) and H3.6M (right) against SOTA methods . Parenthesis denotes body joints used. All errors are in , except accel ( ). For H3.6M testing, please note that 1) modern HPE methods use 2D keypoints as input (CPN [ 10 ] predictions in this case), with no existing pure video methods (that we are aware of) which can obtain competitive results and 2) that our method was the only one not fine-tuned on any real data (including the H3.6M training set).
Figure 4: Emerging behavior: sim-to-real generalization to OOD classes. Our approach has been trained only on synthetic videos of single class (humans), but generalizes to real-world videos with a variety of other classes of articulated objects.
Figure 5: Emerging behavior: generalization to multiple instances. Trained purely on synthetic data with one single human within each video, our method generalizes in zero-shot to real videos with multiple humans.