Egocentric video captioning requires fine-grained detail and accurate timing, yet current multimodal large language models (MLLMs) often miss brief actions, confuse their order, or misinterpret the wearer's motion. Existing caption formats also offer limited support for combining detailed narration with precise timing. We present EgoCaptioner, which pairs a video-level overview of the setting, participants, and the wearer's overall activity with timed steps whose narration adapts to varying perceptual demands and marks key moments at specific timestamps. We develop a coarse-to-fine annotation pipeline that establishes event structure before expanding step descriptions, yielding 50,000 training examples from eight egocentric video sources. Complementary rewards assess factual accuracy and coverage, measure step-level temporal alignment, and penalize fragmented or redundant narration, guiding caption refinement through Group Relative Policy Optimization (GRPO). To improve the reliability of factual supervision, we supplement caption references with pose-derived motion facts when sensor data are available. Across five egocentric benchmarks, EgoCaptioner achieves leading performance among evaluated open-source models in caption quality, caption-conditioned question answering, and temporal alignment. It also surpasses Gemini 3.1 Pro on PhysCapBench and all evaluated proprietary baselines on Minerva-Ego under the caption-to-QA protocol. Ablations support the complementary roles of factual and temporal rewards and show that coarse-to-fine annotation provides more effective supervision than single-pass annotation.
Figures & tables
Figure 1: Data construction pipeline. (a) Gemini-3.1-Pro and Seed2.1 pro annotate each video coarse to fine, from a global overview and event outlines to timestamped steps. (b) The draft caption is reviewed against the full video and corrected. (c) Each validated caption can yield full-video, event-conditioned, and overview instructions. (d) Body and camera poses are converted into timed motion facts.
Figure 2: Illustration of the rewards designed for egocentric captioning. Fact, time, and style rewards are combined for step-level captions, while overview captions pair the content reward with a format reward. The content reward Rfact is the F1 of judge-scored precision and recall over extracted facts, the time reward Rtime greedily matches predicted and reference steps by temporal IoU, and the style reward Rstyle penalizes steps that judges flag as fragmented or redundant.
EgoMe-Overview
EgoMe-Steps
EgoTempo
Minerva-Ego
EgoHome
PhysCapBench
Model
B@4 ↑
METEOR ↑
R-L ↑
CIDEr ↑
METEOR ↑
SODA-c ↑
SODA-d ↑
Acc ↑
Score ↑
Acc open ↑
PSJ ↑
Overall ↑
Proprietary MLLMs
Gemini 3.1 Pro
3.32
18.62
24.45
5.61
3.98
7.41
46.80
23.60
1.444
4.29
52.86
75.06
Gemini 3.8 Flash
2.62
19.27
22.12
0.79
6.26
10.34
48.31
41.20
2.310
7.14
71.09
82.77
GPT-6-Astra
1.28
16.12
15.95
0.00
5.91
6.25
43.12
52.60
2.842
15.71
79.42
84.87
Open-source MLLMs and Captioners
Table 1: Caption quality, video understanding, and temporal alignment (RQ1). All scores are scaled by 100 except EgoTempo Score (0–5). B@4 and R-L denote BLEU-4 and ROUGE-L; PhysCapBench Overall averages seven dimensions. † denotes direct video QA results. Best and second-best indicate rankings among open-source models.
Variant
METEOR ↑
SODA-d ↑
PSJ ↑
Base
17.58
50.25
46.90
SFT (10k)
16.06
42.20
67.79
SFT (25k)
16.37
43.80
68.63
SFT (50k)
19.91
53.25
71.63
w/o Rfact
18.93
51.05
67.97
w/o Rtime
20.17
53.50
66.81
Table 2: Training-stage, data-scale, and reward ablations (RQ2). METEOR and SODA-d are on EgoMe Overview and Steps; PSJ is on EgoHome. The SFT rows vary only the amount of SFT data under the same recipe. Fact reward is ablated across all three training tasks, whereas temporal and style rewards are ablated only for full-video and event-conditioned captioning.
Overview
Steps
EgoTempo
Minerva
EgoHome
Method
METEOR ↑
SODA-d ↑
Acc ↑
Acc ↑
PSJ ↑
Single-pass
17.57
48.43
26.60
7.14
62.89
Ours
19.91
53.25
36.40
18.57
71.63
Table 3: Annotation-procedure ablation after SFT (RQ2). Both variants use matched training videos and data volume.
Figure 3: Task-focused detail and temporal granularity (RQ3). Each panel shows EgoCaptioner’s timed step descriptions for one activity type, with task-relevant details highlighted.
Figure 4: Per-dimension PhysCapBench scores (RQ3). EgoCaptioner vs. CapRL-Video-4B at comparable caption length. EC: event coverage, OR: objects and roles, AM: actions and motion, CI: contact and interaction, SO: states and outcomes, MC: physical mechanisms and causality, TO: temporal order.
Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.
Yifan Xu, Zihao Wang, Zhixiao Wang +6
State Key Laboratory for Novel Software Technology, Nanjing Univerisity · Kling Team, Kuaishou Technology · Shanghai AI Laboratory
Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce EgoVITA, a framework that decomposes egocentric video reasoning into a structured plan-then-verify process. The model first generates an egocentric plan: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an exocentric verification stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification. EgoVITA achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by +7.7 on EgoBlind and +4.4 on EgoOrient, while maintaining strong generalization on exocentric video tasks with only 52k training samples.
We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo
Junbin Xiao, Shenglang Zhang, Pengxiang Zhu +1
University of Science and Technology of China · National University of Singapore