HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies
Authors: Youngrae Kim, Yejin Jang, Eunho Kim, Youjin Sung, Kun-Woo Song, Sangpil Kim, Sang Ho Yoon
Organizations: Graduate School of Metaverse, KAIST, South Korea · Department of AX, KAIST College of AI, South Korea · Graduate School of Culture Technology, KAIST, South Korea · Department of Artificial Intelligence, Korea University, South Korea
Physical reasoning about humans in videos has largely focused on everyday interactions, where contact, pressure, pose, and motion support embodied perception and contact-aware applications. Still, short and forceful properties like impact events remain relatively underexplored, despite their relevance to broader understanding of how abrupt physical events affect the human body. To fill this gap, we introduce human-centric impact localization, a new task that localizes target physical impacts temporally within a video and spatially on the human body. We further present HAPACT, a HumAn-centric imPACT dataset and benchmark built from movie clips, containing over 2.8K video-query pairs with more than 20K fine-grained human-annotated frames. We built HAPACT around viewer-grounded annotation, text-conditioned localization, and supervision at frame-level temporal and SMPL-X-based spatial granularity. To demonstrate the effectiveness of our benchmark, we conduct extensive evaluations of existing methods, employing video moment retrieval models for temporal localization and human-scene contact models for spatial localization. For the spatial task, we propose a text-conditioned baseline by reformulating localization as query-guided body-region selection. Benchmark results show that physical impact localization remains challenging for existing methods, motivating HAPACT as a dedicated benchmark to support physical impact reasoning in videos.
Figures & tables
Figure 1: Human-Centric Impact Localization. Given a video and a text query describing a target impact event, the task requires localizing when the queried impact affects the person and which body regions are affected in each frame.
Figure 2: Overview of the HAPACT Construction Pipeline. Videos from the Movieclips YouTube channel are filtered by genre priors, processed with Gemini 2.5 Pro [ 13 ] to identify impacts and generate captions. Then, we refined videos through human-targeted filtering and manual revision. The clips are labeled in two phases, including temporal localization and spatial on-body regions.
Dataset
Domain
Scale
#Text Queries
Avg. #M / Query
Avg. M Length
Physical Cue
Body Rep.
Image/video body-level physical datasets
PROX [ 24 ]
Indoor scenes
100,000 frames
✗
✗
✗
Scene contact
SMPL-X
BEHAVE [ 5 ]
Natural scenes
15,200 frames
✗
✗
✗
Object contact
SMPL-H
RICH [ 27 ]
Natural scenes
577,000 frames
✗
✗
✗
Scene contact
SMPL-X
HOT [ 11 ]
In-the-wild
15,082 images
✗
✗
✗
Object/scene contact
2D regions
DAMON [ 64 ]
In-the-wild
5,522 images
✗
✗
✗
Object/scene contact
SMPL
Table 1: Comparison with Representative Body-Centric Physical Understanding and Video Moment Retrieval Datasets. #Text Queries, Avg. #M / Query, and Avg. M Length denote the number of queries, average moments per query, and average moment duration, respectively. HAPACT contains shorter moments than existing video grounding datasets, with an average duration of 0.20 s.
Figure 3: Data Distribution Overview. (a) Distribution of impact occurrences on body mesh, where 1 indicates high and 0 indicates low occurrence probability; (b) Distribution of start-end positions of impact moments, normalized from 0 to 1 relative to the total video length; (c) Distribution of movie genres; (d) Distribution of moment lengths by frame count; (e) Distribution of single- and multi-moment queries; (f) Word cloud of verbs and nouns in the description of HAPACT. It shows that verbs such as “strike” and nouns such as “man” appear frequently, emphasizing that the dataset focuses on physical interactions. More detailed statistical information can be found in Section A.5 .
Figure 4: MLLM Baseline for Spatial Impact Localization with Query-Guided Region Selection.
Table 2: Performance Comparison on Temporal Impact Localization of Retrieval-based Moment Retrieval Models. ZS denotes zero-shot evaluation using QVHighlights [ 37 ] checkpoints, and FT denotes fine-tuning on the HAPACT training split. Random evaluates randomly sampled intervals within each clip. Bold indicates the best result and underline indicates the second-best result.
Table 3: Zero-shot Performance Comparison on Temporal Impact Localization of Generation-based Moment Retrieval Models. Closed-source MLLM results are reported as mean (standard deviation) over five runs. Detailed evaluation settings are provided in Section A.6.2 . Orange denotes the best result, while purple indicates the second best.
Method
Setting
F1 ↑
Precision ↑
Recall ↑
geo. (cm) ↓
F1 Scores per Body-part ↑
Head
Arms
Torso
Hips
Legs
Hands
Feet
BSTRO [ 27 ]
ZS
6.13
10.69
5.68
81.53
0.00
2.75
0.00
0.35
0.01
14.82
16.61
FT
5.72
11.20
5.93
51.35
1.65
3.72
7.44
13.40
4.30
7.18
3.66
DECO [ 64 ]
ZS
8.28
9.87
11.84
83.84
0.00
0.25
1.45
4.21
1.26
12.87
28.86
FT
5.41
68.16
6.18
50.34
1.46
4.78
8.79
10.80
0.85
6.77
0.20
GraphiContact [ 41 ]
ZS
7.88
8.85
10.23
87.42
0.09
0.28
1.34
8.52
1.84
0.63
29.34
Table 4: Performance Comparison on Spatial On-body Impact Localization. ZS , FT , and RS denote zero-shot, fine-tuning on HAPACT, and fine-tuning with query-guided region selection, respectively. Per-region F1 is reported over seven region groups. Bold indicates the best result.
Figure 9
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Prompt for Query-Clip Candidate Generation
Identify segments where a person experiences a physical impact. Consider only sudden and salient physical forces affecting the subject. Exclude light contact, minor bumps, distant events, off-screen events, or events that are not physically relevant to the affected subject.
Answer in the format “[MM:SS - MM:SS] Description of the impact”. If there are multiple events, return multiple lines. If none, answer “None”.
Appendix
Table 10
Prompt for Human-Targeted Impact Filtering
Determine whether the described impact physically involves a human. Count as True if a human is the recipient of the impact, causes the impact through direct physical action, or is directly affected by the resulting physical force. Do not count cases where a human only observes, hears, or is merely near the impact without being physically affected.
Answer with only “True” or “False”.
Appendix
Table 11
Instructions for Temporal Impact Annotation
You are given a movie clip and an impact description. Your task is to mark the start and end frames of the physical impact described in the video. Review the impact description and identify when the impact is applied to the person. Mark only the impact timing itself, excluding the approach before the impact and the aftermath after the impact. If the described impact occurs multiple times, mark each occurrence separately. If the queried impact is not visible or cannot be reliably inferred, leave the comment and skip.
Appendix
Table 12
Instructions for Spatial On-body Impact Annotation
You are given a movie clip, an impact description, and the temporal interval(s) for the impact. Carefully review the clip, the impact description, and the provided interval(s). For every frame within the interval(s), identify which part of the human body is affected by the impact and mark the affected region using the web-based annotation tool. All frames inside the provided interval(s) must be annotated. For human-human impacts, use the grammatical subject of the impact description as the target person, and annotate only the body region of that subject. If the impact is not perceived on the subject in a frame, or if the affected body region cannot be inferred, briefly explain the reason in the comment and remove that frame from the interval.
Appendix
Table 13
Figure 7: Spatial annotation interface. Annotators mark affected body regions on an SMPL-X mesh using draw and erase modes, with controls for mesh navigation, brush size, and shortcut views.
Figure 8: Distribution of nouns (left) and verbs (right) in the descriptions.
Total
Train
Test
Val
Video Clip Statistics
Number of Movies
212
148
32
32
Number of Clips
2,884
2,019
433
432
Mean Clip Duration
6.49s
6.49s
6.44s
6.55s
Query Statistics
Total Word Count
32,039
22,395
4,920
4,724
Appendix
Table 5: Statistics on splits of HAPACT.
Moments per Query
Total
Train
Test
Val
1
1,900
1,355
272
273
2
753
507
119
127
3+
231
157
42
32
Total
2,884
2,019
433
432
Appendix
Table 6: Distribution of the number of moments per query for each split.
Figure 9: Distribution of movie genres for each split. This shows that action and adventure comprise a similar proportion of each split.
Figure 10: Distribution of start-end positions for each split. Start and end positions of impact moments, normalized from 0 to 1 relative to the total video length. The distribution of moment positions is comparable across all splits.
Figure 11: Distribution of impact occurrences on the body mesh for each split. 1 indicates a high probability of contact and 0 indicates a low probability. This indicates that the affected body vertices were distributed similarly across the splits.
Figure 12: Distribution of affected body parts weighted by face-level intensity across the SMPL-X mesh. Each bar represents the face-level intensity score (%) for each body part.
Model
Prompt
VTimeLLM
This is a video with {num_frames} frames: <video> During which frames can we see the impact event “{query}” happening in the video? If the event occurs multiple separate times, return all intervals, not just the most salient one. Localize only the applied moment of impact, not the approach or aftermath. Return all intervals in chronological order. Answer only in the format: From s1 to e1; From s2 to e2.
TRACE
<video> Localize a series of moments in the video where the impact on a person’s body described by “{query}” is physically applied, and output the start and end timestamps for each matching moment. The query may match multiple separate impact moments in the same video. Return all matching intervals, not just the most salient one. Localize only the applied moment of impact, not the approach or aftermath.
Grounded-Video-LLM
<video> <grounding> Give you a textual query: “{query}”. The described content may occur multiple times in the video, where a physical impact is applied to a person’s body. When do all the described impact moments occur in the video? Localize only the moment when the impact is physically applied to the person, not the approach or aftermath. Please return all start and end timestamps in chronological order.
UniTime
This is a sequence interleaved with timestamps and frames. Your task is to identify the temporal window(s) (start and end timestamps) when the physical impact on a person’s body described by the given query is applied. The query may match multiple separate impact moments in the video. Localize only the applied moment of impact, not the approach or aftermath. Query: {query} Answer:
TimeLens
You are given a video with multiple frames. The numbers before each video frame indicate its sampling timestamp (in seconds). Please find the physical impact on a person’s body described by the sentence “{query}”, determining its starting and ending times. The query may match multiple separate impact moments in the same video. Localize only the moment of impact, not the approach or aftermath. The format should be: “The event happens in <start time 1> - <end time 1> seconds; <start time 2> - <end time 2> seconds; …”.
Qwen3-VL / InternVL-3.5
Identify one or more time intervals in this video where the impact on a person’s body described by “{query}” is physically applied. The query may match multiple separate impact moments in the same video. Return all matching intervals, not just the most salient one. Localize only the moment when the impact is applied to the person, not the approach or aftermath. Return only the list of relevant intervals in seconds as [[start_1, end_1], ..., [start_n, end_n]] , for example [[1.040, 1.375], [4.833, 5.125], [7.208, 7.417]] . If there is only one interval, return it as a nested list [[start, end]] . Do not return any explanation or extra text.
Appendix
Table 7: Prompts for Generation-based Temporal Localization Baselines. Each prompt follows the model’s native input convention while preserving the same impact-localization constraints.
Figure 13: The 18 body-part labels for query-guided region selection. Left: front; right: back.
Prompt for Query-Guided Region Selection
You are observing a movie frame. The impact query is “{query}”. Select all body regions physically affected by the impact in this frame.
The possible body regions are: head, neck, chest, back, left upper arm, left forearm, left hand, right upper arm, right forearm, right hand, butt, hip, left thigh, left calf, left foot, right thigh, right calf, right foot.
Answer with only the selected body-region labels, separated by commas.
Appendix
Table 24
Answer Format
The affected body regions are {list of body-region labels}.
Appendix
Table 25
Figure 14: The 7 body-region groups for per-region evaluation. Left: front; right: back.
Table 8: Effect of Temporal Sampling Interval. Moment-DETR and QD-DETR are trained on the HAPACT training split and evaluated on the test split under different temporal sampling intervals.
Table 9: Effect of Pre-training and Fine-tuning. Retrieval-based models implemented in the Lighthouse [ 48 ] codebase are evaluated under three settings: QVHighlights pre-trained checkpoints without fine-tuning ( Zero-shot ), training from scratch on the training split ( HAPACT ), and fine-tuning the QVHighlights checkpoints on the training split ( Fine-tune ). All models use CLIP features with a temporal sampling interval of 0.05 seconds and are evaluated on the HAPACT test split.
Method
Modality
F1 ↑
Precision ↑
Recall ↑
geo. (cm) ↓
F1 Scores per Body-part ↑
Head
Arms
Torso
Hips
Legs
Hands
Feet
Cosmos-Reason2-8B
T
15.98
14.03
39.36
37.84
20.90
13.90
19.68
24.35
7.66
12.29
8.13
I
16.46
14.48
36.02
39.65
19.83
15.99
19.26
26.47
10.76
17.79
7.60
T + I
18.59
16.17
44.11
33.31
21.21
15.92
20.01
29.95
12.59
17.87
8.71
Appendix
Table 10: Effect of Input Modality in the MLLM Baseline. Text-only ( T ), image-only ( I ), and text-image ( T+I ) inputs are compared under the same query-guided region selection setting. The table reports overall localization performance and per-region F1 over seven coarse body-region groups.
Table 11: Effects of Impact Duration and Moment Count on Temporal Localization. Values in parentheses denote relative changes from the reference group, with darker red indicating larger decreases. For GPT-5.4, results are reported as mean ± standard deviation over five runs, with relative changes computed from the mean. Bold indicates the smallest decrease for each comparison.
Method
Setting
F1 (%) ↑
Precision (%) ↑
Recall (%) ↑
geo. (cm) ↓
DECO
Zero-shot
8.28
9.87
11.84
83.84
HAPACT
5.05
68.92
6.05
55.75
Fine-tune
5.41
68.16
6.18
50.34
GraphiContact
Zero-shot
7.88
8.85
10.23
87.42
HAPACT
7.74
8.71
10.10
88.11
Fine-tune
7.34
30.12
14.18
50.02
Appendix
Table 12: HSC Baseline Performance under Different Training Settings. Zero-shot denotes evaluation using the released DAMON checkpoints, HAPACT denotes training from scratch on HAPACT, and Fine-tune denotes fine-tuning the DAMON checkpoints on HAPACT.
Training Set
Region-level F1 (%) ↑
Disparity ↓
Mean Region
Head/Torso
Limb
Region Gap
Across-Region SD
Original (HAPACT)
36.57
49.54
30.09
19.45
12.63
Random Down-Sampling
37.25
50.37
30.68
19.69
13.54
Region-Balanced Down-Sampling
38.04
49.05
32.54
16.51
11.51
Appendix
Table 13: Effect of Region-Label Imbalance on the Spatial Performance of the MLLM baseline. Mean Region F1 is the mean F1 across all 18 body-region labels. Head/Torso comprises the head, neck, chest, back, hip, and buttocks, and the remaining 12 labels form Limb. Region Gap is the difference between the mean F1 scores of the Head/Torso and Limb groups. Across-Region SD is the standard deviation of F1 scores across all regions. Lower disparity values indicate more uniform performance. Bold indicates the best result.
Figure 15: Examples of HAPACT. The examples show three representative impact categories: (a) human-to-human contact (e.g., grabbing), (b) human-to-environment contact (e.g., being thrown against a surface), and (c) impact from external forces (e.g., blast explosions).
Figure 16: Annotation subjectivity in spatial localization. The reference annotation and individual annotator labels for the same impact frame and query show similar spatial tendencies, while precise vertex labels differ due to occlusion, narrow contact, and annotator interpretation.
Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular video. Our approach augments a human reconstruction foundation model with learnable contact-force tokens and a temporal transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world videos. We also introduce a real-world climbing benchmark ForceWall with climbing videos and corresponding ground-truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from video.
Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin +5
We introduce HumanPhys, a human-centric benchmark for real-world physical perception, and a training corpus emitted by the same pipeline. Each question targets a physically grounded quantity related to people, including depth, inter-person distance, gaze angle, displacement, velocity, heading-step trajectory, blink count, and contact-deformation strength. The rendered probes additionally test illumination inference from body and face cues and camera ego-motion, both with and without a person in the frame. All labels are derived programmatically from sensor calibration, source metadata, or renderer state, rather than written by annotators or produced by estimators on the benchmark frames. HumanPhys systematically probes six dimensions of human-centric physical perception: spatial state, kinematics, blink events, deformation, illumination, and camera pose. To the best of our knowledge, no existing MLLM benchmark-and-training corpus jointly supervises this set of human-centered physical quantities under direct metadata- or renderer-derived labels. The six dimensions appear only partially dissociable across the tested MLLMs. We benchmark a range of open- and closed-source MLLMs on HumanPhys and find that all six capability dimensions remain challenging for current models. We hope HumanPhys will support future research on multimodal systems that perceive and reason about people in the physical world.
Yiming Zhang, Enzhi Zhou, Junan Lin +4
Nanyang Technological University, Singapore · Ant Group, Hangzhou, China
Physical video understanding requires more than naming an event correctly. A model can answer a question about pouring, sliding, or collision from textual regularities while still failing to localize the event in time or space. We introduce a grounded benchmark for physical video understanding that extends the what--when--where evaluation structure of V-STaR to four video sources, six physics domains, three prompt families (physics, vstar_like, and neutral_rstr), and four input conditions (original, shuffled, ablated, and frame-masked). The benchmark contains 1,560 base video clips from SSV2, YouCook2, HoloAssist, and Roundabout-TAU. Each clip is first converted into a shared grounded event record, and the three query families are derived from that record. Temporal and spatial targets are shared across prompt families, while the non-physics families use deterministic family-appropriate semantic a_what targets derived from the same record. Across models and prompt families, physics remains the strongest regime overall, vstar_like is the clearest non-physics semantic comparison, and neutral_rstr behaves as a harder templated control. Prompt-family robustness is selective rather than universal, perturbation gains cluster in weak original cases, and spatial grounding is the weakest across settings. These results suggest that video Q&A reasoning benchmarks shall report physically grounded, prompt-aware, and perturbation-aware diagnostics alongside aggregate accuracy.