HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies

Youngrae Kim; Yejin Jang; Eunho Kim; Youjin Sung; Kun-Woo Song; Sangpil Kim; Sang Ho Yoon

cs.CVOct 16, 2026

HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies

Authors: Youngrae Kim, Yejin Jang, Eunho Kim, Youjin Sung, Kun-Woo Song, Sangpil Kim, Sang Ho Yoon

Organizations: Graduate School of Metaverse, KAIST, South Korea · Department of AX, KAIST College of AI, South Korea · Graduate School of Culture Technology, KAIST, South Korea · Department of Artificial Intelligence, Korea University, South Korea

Abstract

Physical reasoning about humans in videos has largely focused on everyday interactions, where contact, pressure, pose, and motion support embodied perception and contact-aware applications. Still, short and forceful properties like impact events remain relatively underexplored, despite their relevance to broader understanding of how abrupt physical events affect the human body. To fill this gap, we introduce human-centric impact localization, a new task that localizes target physical impacts temporally within a video and spatially on the human body. We further present HAPACT, a HumAn-centric imPACT dataset and benchmark built from movie clips, containing over 2.8K video-query pairs with more than 20K fine-grained human-annotated frames. We built HAPACT around viewer-grounded annotation, text-conditioned localization, and supervision at frame-level temporal and SMPL-X-based spatial granularity. To demonstrate the effectiveness of our benchmark, we conduct extensive evaluations of existing methods, employing video moment retrieval models for temporal localization and human-scene contact models for spatial localization. For the spatial task, we propose a text-conditioned baseline by reformulating localization as query-guided body-region selection. Benchmark results show that physical impact localization remains challenging for existing methods, motivating HAPACT as a dedicated benchmark to support physical impact reasoning in videos.

Figures & tables

Figure 1: Human-Centric Impact Localization. Given a video and a text query describing a target impact event, the task requires localizing when the queried impact affects the person and which body regions are affected in each frame.

Figure 2: Overview of the HAPACT Construction Pipeline. Videos from the Movieclips YouTube channel are filtered by genre priors, processed with Gemini 2.5 Pro [ 13 ] to identify impacts and generate captions. Then, we refined videos through human-targeted filtering and manual revision. The clips are labeled in two phases, including temporal localization and spatial on-body regions.

Dataset	Domain	Scale	#Text Queries	Avg. #M / Query	Avg. M Length	Physical Cue	Body Rep.
Image/video body-level physical datasets
PROX [ 24 ]	Indoor scenes	100,000 frames	✗	✗	✗	Scene contact	SMPL-X
BEHAVE [ 5 ]	Natural scenes	15,200 frames	✗	✗	✗	Object contact	SMPL-H
RICH [ 27 ]	Natural scenes	577,000 frames	✗	✗	✗	Scene contact	SMPL-X
HOT [ 11 ]	In-the-wild	15,082 images	✗	✗	✗	Object/scene contact	2D regions
DAMON [ 64 ]	In-the-wild	5,522 images	✗	✗	✗	Object/scene contact	SMPL

Table 1: Comparison with Representative Body-Centric Physical Understanding and Video Moment Retrieval Datasets. #Text Queries, Avg. #M / Query, and Avg. M Length denote the number of queries, average moments per query, and average moment duration, respectively. HAPACT contains shorter moments than existing video grounding datasets, with an average duration of

0.20

Figure 3: Data Distribution Overview. (a) Distribution of impact occurrences on body mesh, where 1 indicates high and 0 indicates low occurrence probability; (b) Distribution of start-end positions of impact moments, normalized from 0 to 1 relative to the total video length; (c) Distribution of movie genres; (d) Distribution of moment lengths by frame count; (e) Distribution of single- and multi-moment queries; (f) Word cloud of verbs and nouns in the description of HAPACT. It shows that verbs such as “strike” and nouns such as “man” appear frequently, emphasizing that the dataset focuses on physical interactions. More detailed statistical information can be found in Section A.5 .

Figure 4: MLLM Baseline for Spatial Impact Localization with Query-Guided Region Selection.

Method				mAP				mR@k
Method	Setting	[email protected]	[email protected]	G-mAP	@1_tgt	@2_tgt	@3+tgt	@1	@2	@3
Random		4.14	2.31	7.16	7.30	6.73	5.50	4.11	6.55	7.91
Moment-DETR [ 37 ] NeurIPS ’21	ZS	8.05	5.77	10.04	11.69	7.07	5.56	6.93	5.88	7.29
Moment-DETR [ 37 ] NeurIPS ’21	FT	18.16	13.63	20.39	21.94	19.11	12.92	15.57	16.61	16.59
QD-DETR [ 47 ] CVPR ’23	ZS	9.55	6.00	11.98	13.57	9.50	7.81	8.22	8.12	8.53
QD-DETR [ 47 ] CVPR ’23	FT	21.27	16.17	22.06	23.91	19.98	15.39	16.86	18.67	19.69

Table 2: Performance Comparison on Temporal Impact Localization of Retrieval-based Moment Retrieval Models. ZS denotes zero-shot evaluation using QVHighlights [ 37 ] checkpoints, and FT denotes fine-tuning on the HAPACT training split. Random evaluates randomly sampled intervals within each clip. Bold indicates the best result and underline indicates the second-best result.

Method			mAP				mR@k
Method	[email protected]	[email protected]	G-mAP	@1_tgt	@2_tgt	@3+tgt	@1	@2	@3
VMR-specific Models
VTimeLLM [ 25 ]	1.04	1.15	1.92	2.69	0.82	0.26	2.22	0.00	0.00
TRACE [ 22 ]	4.21	4.16	4.76	5.18	3.67	3.25	4.80	6.25	7.50
Grounded-Video-LLM [ 65 ]	7.97	10.62	10.06	11.72	8.44	1.37	13.30	0.00	0.00
UniTime [ 40 ]	7.93	9.01	12.41	16.42	6.72	1.95	15.06	0.00	0.00

Table 3: Zero-shot Performance Comparison on Temporal Impact Localization of Generation-based Moment Retrieval Models. Closed-source MLLM results are reported as mean (standard deviation) over five runs. Detailed evaluation settings are provided in Section A.6.2 . Orange denotes the best result, while purple indicates the second best.

Method	Setting	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	geo. (cm) $\downarrow$	F1 Scores per Body-part $\uparrow$
Method	Setting	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	geo. (cm) $\downarrow$	Head	Arms	Torso	Hips	Legs	Hands	Feet
BSTRO [ 27 ]	ZS	6.13	10.69	5.68	81.53	0.00	2.75	0.00	0.35	0.01	14.82	16.61
BSTRO [ 27 ]	FT	5.72	11.20	5.93	51.35	1.65	3.72	7.44	13.40	4.30	7.18	3.66
DECO [ 64 ]	ZS	8.28	9.87	11.84	83.84	0.00	0.25	1.45	4.21	1.26	12.87	28.86
DECO [ 64 ]	FT	5.41	68.16	6.18	50.34	1.46	4.78	8.79	10.80	0.85	6.77	0.20
GraphiContact [ 41 ]	ZS	7.88	8.85	10.23	87.42	0.09	0.28	1.34	8.52	1.84	0.63	29.34

Table 4: Performance Comparison on Spatial On-body Impact Localization. ZS , FT , and RS denote zero-shot, fine-tuning on HAPACT, and fine-tuning with query-guided region selection, respectively. Per-region F1 is reported over seven region groups. Bold indicates the best result.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt for Query-Clip Candidate Generation

Identify segments where a person experiences a physical impact. Consider only sudden and salient physical forces affecting the subject. Exclude light contact, minor bumps, distant events, off-screen events, or events that are not physically relevant to the affected subject.

Answer in the format “[MM:SS - MM:SS] Description of the impact”. If there are multiple events, return multiple lines. If none, answer “None”.

Appendix

Table 10

Prompt for Human-Targeted Impact Filtering

Determine whether the described impact physically involves a human. Count as True if a human is the recipient of the impact, causes the impact through direct physical action, or is directly affected by the resulting physical force. Do not count cases where a human only observes, hears, or is merely near the impact without being physically affected.

Answer with only “True” or “False”.

Appendix

Table 11

Instructions for Temporal Impact Annotation

You are given a movie clip and an impact description. Your task is to mark the start and end frames of the physical impact described in the video. Review the impact description and identify when the impact is applied to the person. Mark only the impact timing itself, excluding the approach before the impact and the aftermath after the impact. If the described impact occurs multiple times, mark each occurrence separately. If the queried impact is not visible or cannot be reliably inferred, leave the comment and skip.

Appendix

Table 12

Instructions for Spatial On-body Impact Annotation

You are given a movie clip, an impact description, and the temporal interval(s) for the impact. Carefully review the clip, the impact description, and the provided interval(s). For every frame within the interval(s), identify which part of the human body is affected by the impact and mark the affected region using the web-based annotation tool. All frames inside the provided interval(s) must be annotated. For human-human impacts, use the grammatical subject of the impact description as the target person, and annotate only the body region of that subject. If the impact is not perceived on the subject in a frame, or if the affected body region cannot be inferred, briefly explain the reason in the comment and remove that frame from the interval.

Appendix

Table 13

Figure 7: Spatial annotation interface. Annotators mark affected body regions on an SMPL-X mesh using draw and erase modes, with controls for mesh navigation, brush size, and shortcut views.

Panel of Figure 8 — Figure 8: Distribution of nouns (left) and verbs (right) in the descriptions.

	Total	Train	Test	Val
Video Clip Statistics
Number of Movies	212	148	32	32
Number of Clips	2,884	2,019	433	432
Mean Clip Duration	6.49s	6.49s	6.44s	6.55s
Query Statistics
Total Word Count	32,039	22,395	4,920	4,724

Appendix

Table 5: Statistics on splits of HAPACT.

Moments per Query	Total	Train	Test	Val
1	1,900	1,355	272	273
2	753	507	119	127
3+	231	157	42	32
Total	2,884	2,019	433	432

Appendix

Table 6: Distribution of the number of moments per query for each split.

Figure 9: Distribution of movie genres for each split. This shows that action and adventure comprise a similar proportion of each split.

Figure 10: Distribution of start-end positions for each split. Start and end positions of impact moments, normalized from 0 to 1 relative to the total video length. The distribution of moment positions is comparable across all splits.

Figure 11: Distribution of impact occurrences on the body mesh for each split. 1 indicates a high probability of contact and 0 indicates a low probability. This indicates that the affected body vertices were distributed similarly across the splits.

Figure 12: Distribution of affected body parts weighted by face-level intensity across the SMPL-X mesh. Each bar represents the face-level intensity score (%) for each body part.

Model	Prompt
VTimeLLM	This is a video with {num_frames} frames: <video> During which frames can we see the impact event “{query}” happening in the video? If the event occurs multiple separate times, return all intervals, not just the most salient one. Localize only the applied moment of impact, not the approach or aftermath. Return all intervals in chronological order. Answer only in the format: From s1 to e1; From s2 to e2.
TRACE	<video> Localize a series of moments in the video where the impact on a person’s body described by “{query}” is physically applied, and output the start and end timestamps for each matching moment. The query may match multiple separate impact moments in the same video. Return all matching intervals, not just the most salient one. Localize only the applied moment of impact, not the approach or aftermath.
Grounded-Video-LLM	<video> <grounding> Give you a textual query: “{query}”. The described content may occur multiple times in the video, where a physical impact is applied to a person’s body. When do all the described impact moments occur in the video? Localize only the moment when the impact is physically applied to the person, not the approach or aftermath. Please return all start and end timestamps in chronological order.
UniTime	This is a sequence interleaved with timestamps and frames. Your task is to identify the temporal window(s) (start and end timestamps) when the physical impact on a person’s body described by the given query is applied. The query may match multiple separate impact moments in the video. Localize only the applied moment of impact, not the approach or aftermath. Query: {query} Answer:
TimeLens	You are given a video with multiple frames. The numbers before each video frame indicate its sampling timestamp (in seconds). Please find the physical impact on a person’s body described by the sentence “{query}”, determining its starting and ending times. The query may match multiple separate impact moments in the same video. Localize only the moment of impact, not the approach or aftermath. The format should be: “The event happens in <start time 1> - <end time 1> seconds; <start time 2> - <end time 2> seconds; …”.
Qwen3-VL / InternVL-3.5	Identify one or more time intervals in this video where the impact on a person’s body described by “{query}” is physically applied. The query may match multiple separate impact moments in the same video. Return all matching intervals, not just the most salient one. Localize only the moment when the impact is applied to the person, not the approach or aftermath. Return only the list of relevant intervals in seconds as [[start_1, end_1], ..., [start_n, end_n]] , for example [[1.040, 1.375], [4.833, 5.125], [7.208, 7.417]] . If there is only one interval, return it as a nested list [[start, end]] . Do not return any explanation or extra text.

Appendix

Table 7: Prompts for Generation-based Temporal Localization Baselines. Each prompt follows the model’s native input convention while preserving the same impact-localization constraints.

Figure 13: The 18 body-part labels for query-guided region selection. Left: front; right: back.

Prompt for Query-Guided Region Selection

You are observing a movie frame. The impact query is “{query}”. Select all body regions physically affected by the impact in this frame.

The possible body regions are: head, neck, chest, back, left upper arm, left forearm, left hand, right upper arm, right forearm, right hand, butt, hip, left thigh, left calf, left foot, right thigh, right calf, right foot.

Answer with only the selected body-region labels, separated by commas.

Appendix

Table 24

Answer Format

The affected body regions are {list of body-region labels}.

Appendix

Table 25

Figure 14: The 7 body-region groups for per-region evaluation. Left: front; right: back.

Method	Sample Int.			mAP				mR@k
Method	Sample Int.	[email protected]	[email protected]	G-mAP	@1_tgt	@2_tgt	@3+tgt	@1	@2	@3
Moment-DETR	2.0s	15.65	12.70	16.54	17.44	16.66	6.99	13.67	14.91	12.87
	1.0s	10.29	9.93	11.89	13.07	11.31	8.98	11.96	10.54	6.51
	0.5s	11.12	12.01	11.40	12.10	11.27	8.01	12.75	11.03	8.99
	0.1s ∗	16.98	13.86	18.65	19.82	17.30	12.22	14.60	16.00	19.38
	0.05s ∗	16.25	12.24	17.70	17.19	20.11	13.25	13.63	18.85	18.14

Appendix

Table 8: Effect of Temporal Sampling Interval. Moment-DETR and QD-DETR are trained on the HAPACT training split and evaluated on the test split under different temporal sampling intervals.

Method	Setting			mAP				mR@k
Method	Setting	[email protected]	[email protected]	G-mAP	@1_tgt	@2_tgt	@3+tgt	@1	@2	@3
Moment-DETR	Zero-shot	8.05	5.77	10.04	11.69	7.07	5.56	6.93	5.88	7.29
	HAPACT	16.25	12.24	17.70	17.19	20.11	13.25	13.63	18.85	18.14
	Fine-tune	18.16	13.63	20.39	21.94	19.11	12.92	15.57	16.61	16.59
QD-DETR	Zero-shot	9.55	6.00	11.98	13.57	9.50	7.81	8.22	8.12	8.53
	HAPACT	20.06	16.63	21.36	22.36	21.01	12.83	18.20	20.67	18.29

Appendix

Table 9: Effect of Pre-training and Fine-tuning. Retrieval-based models implemented in the Lighthouse [ 48 ] codebase are evaluated under three settings: QVHighlights pre-trained checkpoints without fine-tuning ( Zero-shot ), training from scratch on the training split ( HAPACT ), and fine-tuning the QVHighlights checkpoints on the training split ( Fine-tune ). All models use CLIP features with a temporal sampling interval of

0.05

seconds and are evaluated on the HAPACT test split.

Method	Modality	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	geo. (cm) $\downarrow$	F1 Scores per Body-part $\uparrow$
Method	Modality	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	geo. (cm) $\downarrow$	Head	Arms	Torso	Hips	Legs	Hands	Feet
Cosmos-Reason2-8B	T	15.98	14.03	39.36	37.84	20.90	13.90	19.68	24.35	7.66	12.29	8.13
	I	16.46	14.48	36.02	39.65	19.83	15.99	19.26	26.47	10.76	17.79	7.60
	T + I	18.59	16.17	44.11	33.31	21.21	15.92	20.01	29.95	12.59	17.87	8.71

Appendix

Table 10: Effect of Input Modality in the MLLM Baseline. Text-only ( T ), image-only ( I ), and text-image ( T+I ) inputs are compared under the same query-guided region selection setting. The table reports overall localization performance and per-region F1 over seven coarse body-region groups.

(a) Impact Duration				(b) Moment Count
	[email protected] (%)			[email protected] (%)
Method	Long (7+f, ref.)	Middle (3–6f)	Short (1–2f)	Single (1, ref.)	Multiple (2+)
Sim-DETR	40.22	24.71 (-38.6%)	10.17 (-74.7%)	26.58	24.23 (-8.8%)
UniTime	27.31	1.04 (-96.2%)	0.00 (-100.0%)	16.42	5.90 (-64.1%)
Qwen3-VL-8B	46.92	26.81 (-42.9%)	3.22 (-93.1%)	28.31	15.04 (-46.9%)
GPT-5.4	51.32 $\pm$ 1.38	44.60 $\pm$ 1.76 (-13.1%)	25.72 $\pm$ 1.85 (-49.9%)	53.90 $\pm$ 1.38	23.58 $\pm$ 0.39 (-56.3%)

Appendix

Table 11: Effects of Impact Duration and Moment Count on Temporal Localization. Values in parentheses denote relative changes from the reference group, with darker red indicating larger decreases. For GPT-5.4, results are reported as mean

\pm

standard deviation over five runs, with relative changes computed from the mean. Bold indicates the smallest decrease for each comparison.

Method	Setting	F1 (%) $\uparrow$	Precision (%) $\uparrow$	Recall (%) $\uparrow$	geo. (cm) $\downarrow$
DECO	Zero-shot	8.28	9.87	11.84	83.84
	HAPACT	5.05	68.92	6.05	55.75
	Fine-tune	5.41	68.16	6.18	50.34
GraphiContact	Zero-shot	7.88	8.85	10.23	87.42
	HAPACT	7.74	8.71	10.10	88.11
	Fine-tune	7.34	30.12	14.18	50.02

Appendix

Table 12: HSC Baseline Performance under Different Training Settings. Zero-shot denotes evaluation using the released DAMON checkpoints, HAPACT denotes training from scratch on HAPACT, and Fine-tune denotes fine-tuning the DAMON checkpoints on HAPACT.

Training Set	Region-level F1 (%) $\uparrow$			Disparity $\downarrow$
Training Set	Mean Region	Head/Torso	Limb	Region Gap	Across-Region SD
Original (HAPACT)	36.57	49.54	30.09	19.45	12.63
Random Down-Sampling	37.25	50.37	30.68	19.69	13.54
Region-Balanced Down-Sampling	38.04	49.05	32.54	16.51	11.51

Appendix

Table 13: Effect of Region-Label Imbalance on the Spatial Performance of the MLLM baseline. Mean Region F1 is the mean F1 across all 18 body-region labels. Head/Torso comprises the head, neck, chest, back, hip, and buttocks, and the remaining 12 labels form Limb. Region Gap is the difference between the mean F1 scores of the Head/Torso and Limb groups. Across-Region SD is the standard deviation of F1 scores across all regions. Lower disparity values indicate more uniform performance. Bold indicates the best result.

Figure 15: Examples of HAPACT. The examples show three representative impact categories: (a) human-to-human contact (e.g., grabbing), (b) human-to-environment contact (e.g., being thrown against a surface), and (c) impact from external forces (e.g., blast explosions).

Figure 16: Annotation subjectivity in spatial localization. The reference annotation and individual annotator labels for the same impact frame and query show similar spatial tendencies, while precise vertex labels differ due to occlusion, narrow contact, and annotator interpretation.

Physical Reasoning in Video

Physical reasoning in video studies how models infer, predict, and generate motion and interactions consistent with physical laws.

See all 74 papers

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video
Sep 30, 2026Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin +5Contact Force Estimation Human Pose Estimation
MBZUAI · ETH Zürich · EPFL
HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning
Oct 10, 2026Yiming Zhang, Enzhi Zhou, Junan Lin +4Human Motion Understanding VLM Reasoning
Nanyang Technological University, Singapore · Ant Group, Hangzhou, China
Grounding Video Reasoning in Physical Signals
Apr 23, 2026Alibay Osmanli, Zixu Cheng, Shaogang GongTemporal Video Grounding Video QA
Queen Mary University of London

Sep 30, 2026J/K move · Enter open · S save

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin, Peter Wolf+4

MBZUAI · ETH Zürich · EPFL

Contact Force Estimation Human Pose Estimation 3D Human Reconstruction Physics-Informed ML Human-Object Interaction

Open paper AskPDF arXiv

Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular video. Our approach augments a human reconstruction foundation model with learnable contact-force tokens and a temporal transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world videos. We also introduce a real-world climbing benchmark ForceWall with climbing videos and corresponding ground-truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from video.

Abstract

Physical Reasoning in Video

Explore similar work

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual Reasoning

Grounding Video Reasoning in Physical Signals