Grounded Human-Attributed Description and Activity Recognition in Videos

Jay N. Paranjape; Jurijs Nazarovs; Rohan Sarkar; Yue Guo; Lipeng Ke; Sankar Venkataraman; Vishal M. Patel; Nataraj Jammalamadaka

cs.CVOct 15, 2026

Grounded Human-Attributed Description and Activity Recognition in Videos

Authors: Jay N. Paranjape, Jurijs Nazarovs, Rohan Sarkar, Yue Guo, Lipeng Ke, Sankar Venkataraman, Vishal M. Patel, Nataraj Jammalamadaka

Organizations: Johns Hopkins University, USA · Amazon

Abstract

Understanding characteristics and activities of individuals in complex multi-person environments is crucial for real-world applications. However, existing datasets often simplify this problem to grounded group activity recognition, single-person activity recognition, or close-set activity recognition, thus limiting the generalizability of models trained on these datasets. In this work, we introduce the task of Grounded Human-Attributed Description and Activity Recognition (GHADAR), which involves describing the characteristics and activity descriptions of every person in a video, provided the location of the person in the video, which is more practical for real-world applications. To facilitate this, we introduce a new dataset derived from AVA-Actions by generating open-set captions for the person description and activity. In addition, we propose a novel method to effectively utilize the information contained in grounding during training by constraining the cross-attention masks during training in VLMs to improve performance for this task. Our experiments show that our method outperforms SOTA VLMs on this task. Finally, we demonstrate the limitations of existing evaluation metrics, which are overly reliant on human-annotations and exact text-text matching. As an added video-based evaluation, we propose a holistic VLM-based evaluation schema that compares concepts {\em directly} between the video and the generated predictions. Thus, in this work, we develop a complete framework for GHADAR, including a dataset, a novel method and an evaluation schema, thereby establishing a strong foundation for future research in this domain.

Figures & tables

Figure 1: Our proposed task with respect to existing tasks. While existing tasks are more relaxed, we propose an open-set human description and activity recognition that is more generic.

Dataset	Size (# clips)	Human-Centric?	Grounded for every frame?	Open-Set Captions	Multi-Activity?	Multi-Person?
HC-STVG ( [ 30 ] )	15K	✓	✓	✓	✓
VidSTG ( [ 36 , 28 ] )	7.8K		✓	✓
ActivityNet Entities ( [ 40 ] )	15K			✓		✓
AVA-Actions ( [ 9 ] )	450	✓	✓		✓	✓
AVA-Kinteics ( [ 16 ] )	230k	✓			✓	✓
MSR-VTT ( [ 34 ] )	10K			✓

Table 1: Existing datasets. Multi-Activity represents whether captions are available for multiple activities done by the person in the video. Multi-Person represents whether annotations are available for multiple people appearing in the video. Most of the video clips in these datasets are short (0-20s) except for AVA-Actions and MEVA, which have 15 min long and 5 min long annotated videos, respectively. * - can be obtained with minimal processing

Figure 2: Examples from the AVA-Captions dataset. The captions in black are generated using expert-VLM while red action labels are provided in the original AVA-Actions dataset. More examples are provided in the supplementary.

Figure 3: The two phases of CAMP. In the first phase, the modified causal mask for the answer tokens makes sure they can only attend to the vision tokens corresponding to the relaxed bounding box around the person for which the caption is being generated. In the second phase, the model is trained with a lower learning rate with a default attention mask, which allows insertion of global context into the attention weights. Here, $T_{V}$ : vision tokens, $T_{Q}$ : question tokens, $T_{A}$ : Answer tokens. The illustration shows only one frame but the masking is done using all frames of the video.

Method	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	SBert Sim ( $\uparrow$ )	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	Sbert Sim ( $\uparrow$ )
	AVA-Captions					HC-STVG
LLAVA-OV 0.5B	28.29	54.22	59.47	0.75	0.54	18.70	53.94	57.47	0.86	0.48
InternVL3-1B	31.07	58.17	63.09	0.77	0.58	25.87	55.08	65.54	0.87	0.60
Qwen2.5VL-3B	36.46	63.17	68.06	0.82	0.65	25.66	55.07	65.81	0.88	0.62
InternVL3 + CAMP (Ours)	32.02	59.90	64.07	0.79	0.60	27.11	57.90	66.10	0.88	0.61
Qwen2.5VL-3B + CAMP (Ours)	36.48	63.62	68.43	0.83	0.67	27.33	57.84	67.82	0.88	0.64

Table 2: Comparison of CAMP against existing methods for Subject Description

Method	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	SBert Sim ( $\uparrow$ )	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	Sbert Sim ( $\uparrow$ )
	AVA-Captions					HC-STVG
LLAVA-OV 0.5B	13.39	37.71	39.69	0.85	0.53	8.34	32.43	41.78	0.82	0.40
InternVL3-1B	19.34	44.91	46.28	0.89	0.63	8.60	32.66	38.03	0.85	0.42
Qwen2.5VL-3B	19.30	43.02	47.35	0.90	0.63	8.97	31.52	40.01	0.86	0.47
InternVL3 + CAMP (Ours)	20.11	46.15	46.82	0.90	0.65	11.17	35.67	42.23	0.87	0.46
Qwen2.5VL-3B + CAMP (Ours)	20.14	43.40	47.90	0.91	0.65	10.64	33.08	40.65	0.86	0.47

Table 3: Comparison of CAMP against existing methods for Activity Description

	( $\Delta w,\Delta h$ )
	(0,0)	(20,20)	(50,50)	(100,100)	(100,0)	(100,20) (used)	(100,50)	(20,50)
Subject Sbert	0.54	0.54	0.57	0.58	0.60	0.61	0.60	0.55
Subject METEOR	52.21	52.54	54.96	57.09	56.98	57.90	56.59	53.64
Subject ROUGE	61.73	61.67	63.88	65.87	66.38	66.10	65.92	62.30
Activity Sbert	0.44	0.44	0.45	0.46	0.46	0.46	0.46	0.44
Activity METEOR	33.32	33.04	34.00	34.40	35.00	35.67	34.87	33.55

Table 4: Effect of relaxations of bounding box during CAMP on HC-STVG dataset. The columns represent the one-sided changes to width (

\Delta w

) and height (

\Delta h

) in terms of percentage of the total width and height. For example, (20,20) signifies 20% relaxation in width and height on each side, thus making the relaxed bounding box as

[max(0,x_{1}-0.2w),max(0,y_{1}-0.2h),min(x_{2}+0.2w,448),min(y_{2}+0.2h,448)]

Figure 4: Qualitative examples from HC-STVG (left two) and AVA-Captions (right). HC-STVG has captions only for one person, whereas AVA-Captions has captions for both people in the video.

Figure 5: Examples of predictions wrongly given low metrics. In the first row, the predicted events occur in the video but are not captured by the GT annotation, leading to low SBert scores. Similarly, in the second row, the GT and predicted ways to describe the woman are valid, yet subject SBert score is low due to lexical mismatch. This necessitates the need for VLM-based metrics. More examples in the supplementary

VLM Evaluator	Nova Pro v1						Qwen2.5 VL 32B
Dataset	AVA-Captions			HC-STVG			AVA-Captions			HC-STVG
Method	Avg	Avg	Acc ( $\uparrow$ )	Avg	Avg	Acc ( $\uparrow$ )	Avg	Avg	Acc ( $\uparrow$ )	Avg	Avg	Acc ( $\uparrow$ )
	CRT ( $\uparrow$ )	UNQ( $\uparrow$ )		CRT ( $\uparrow$ )	UNQ ( $\uparrow$ )		CRT ( $\uparrow$ )	UNQ( $\uparrow$ )		CRT ( $\uparrow$ )	UNQ( $\uparrow$ )
LLAVA-OV 0.5B	2.84	2.06	0.29	3.99	1.94	0.34	2.89	2.29	0.38	2.86	2.46	0.30
InternVL3-1B	3.03	2.07	0.32	4.03	2.09	0.40	3.05	2.34	0.41	3.03	2.47	0.37
Qwen2.5VL-3B	3.46	2.35	0.43	4.36	2.04	0.39	3.44	2.58	0.66	3.25	2.53	0.33

Table 5: Comparison of CAMP against existing methods for Subject Description with VLM-based metrics using Nova Pro v1 and Qwen2.5 VL 32B as expert VLMs

VLM Evaluator	Nova Pro v1						Qwen2.5 VL 32B
Dataset	AVA-Captions			HC-STVG			AVA-Captions			HC-STVG
Method	Avg	Avg	Acc ( $\uparrow$ )	Avg	Avg	Acc ( $\uparrow$ )	Avg	Avg	Acc ( $\uparrow$ )	Avg	Avg	Acc ( $\uparrow$ )
	CRT ( $\uparrow$ )	CMP( $\uparrow$ )		CRT ( $\uparrow$ )	CMP ( $\uparrow$ )		CRT ( $\uparrow$ )	CMP( $\uparrow$ )		CRT ( $\uparrow$ )	CMP( $\uparrow$ )
LLAVA-OV 0.5B	3.24	2.39	0.54	2.21	1.17	0.08	3.59	3.17	0.75	3.07	2.43	0.53
InternVL3-1B	3.54	2.65	0.64	3.17	1.94	0.41	3.80	3.40	0.83	3.93	3.28	0.88
Qwen2.5VL-3B	3.61	2.83	0.68	3.51	2.07	0.48	3.86	3.51	0.85	3.96	3.18	0.87

Table 6: Comparison of CAMP against existing methods for Activity Description with VLM-based metrics using Nova Pro v1 and Qwen2.5 VL 32B as expert VLMs

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Examples from the AVA-Captions dataset. The captions in black are generated using expert-VLM while red action labels are provided in the original AVA-Actions dataset.

Phase	Total video clips	Minimum # People	Maximum # People	Mean # People	Median # People
Training	36815	1	10	1.96	2
Testing	11095	1	11	1.98	2

Appendix

Table 1: Statistics for the number of people per video clip in the AVA-Captions dataset

		HC-STVG Subject					HC-STVG Activity
Phase 1	Phase 2	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	SBert Sim ( $\uparrow$ )	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	SBert Sim ( $\uparrow$ )
✓		20.09	45.05	52.65	0.58	0.36	5.15	23.88	24.55	0.57	0.26
	✓	25.87	55.08	65.54	0.87	0.60	8.60	32.66	38.03	0.85	0.42
✓	✓	27.11	57.90	66.10	0.88	0.61	11.17	35.67	42.23	0.87	0.46
		AVA-Captions Subject					AVA-Captions Activity
Phase 1	Phase 2	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	SBert Sim ( $\uparrow$ )	BLEU4 ( $\uparrow$ )	METEOR ( $\uparrow$ )	ROUGE ( $\uparrow$ )	CLIP Sim ( $\uparrow$ )	SBert Sim ( $\uparrow$ )

Appendix

Table 2: Ablation on the phases of CAMP. The first row denotes the baseline without CAMP, next row denotes only Phase 1 without Phase 2 and last row denotes the best method, which includes both Phase 1 and Phase 2,

Figure 2: Examples of predictions which are lexically dissimilar from the ground truth but are equally valid for the given video

Figure 3: A limiting case of VLM-based evaluation metrics. CMP and UNQ are pessimistically given a score of 1 even though the predicted caption describes the person uniquely and describes some of the activities of the subject.

Human Activity Recognition

Human activity recognition classifies actions and behaviors from wearable sensors, video, radio signals, and other measurements.

See all 229 papers

HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding
May 29, 2026Andrea Zenotto, Simone Alberto Peirone, Francesca Pistilli +1Temporal Video Grounding Zero-Shot Learning
Politecnico di Torino
ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding
Apr 1, 2026Lala Shakti Swarup Ray, Mengxi Liu, Alcina Pinto +4Human Activity Recognition Cross-Modal Alignment
DFKI, Germany · RPTU
FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding
May 19, 2026Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh +2VLM Evaluation Temporal Video Grounding
National Taiwan University · Google · Independent Researcher

May 29, 2026J/K move · Enter open · S save

HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding

Andrea Zenotto, Simone Alberto Peirone, Francesca Pistilli, Giuseppe Averta

Politecnico di Torino

Temporal Video Grounding Zero-Shot Learning Egocentric Video Understanding

Open paper AskPDF arXiv

Procedural activities follow well-defined structures: whether we consider a cooking recipe or a mechanic repairing a car, these activities naturally decompose in a hierarchy of steps and sub-steps. Traditional approaches for step grounding require extensive annotations and scale poorly. Instead, we argue that such hierarchical structure can emerge naturally from uncurated videos of human activities through recurring patterns of co-occurring actions and activities. Our approach builds on HiERO, a weakly-supervised representation learning approach that maps close in the feature space actions that are functionally related to each other, leveraging only fine-grained action-level narrations. In this feature space, procedure steps can be detected by a simple clustering, with no additional task-specific fine-tuning. For the Ego4D Step Grounding challenge, we augment this approach by ensuring fine and coarse level agreement in step assignments, enforcing strict temporal monotonicity of the grounded steps and post-processing the detected steps to reduce the impact of noisy predictions. We call this approach HiERO-StepG and it achieves 56.27 % on the R@1 (IoU = 0.3) metric on the global leaderboard at submission time, ranking second while being completely zero-shot and not requiring procedure-specific annotations. Project page: https://github.com/andreazenotto/HiERO-StepG.

Abstract

Human Activity Recognition

Explore similar work

HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding

ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding

FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding