Meaningful Moments: A Corpus of VLM-Derived Temporal Importance Labels for Video Action Recognition

Cameron Keith; SouYoung Jin

cs.CVOct 12, 2026

Meaningful Moments: A Corpus of VLM-Derived Temporal Importance Labels for Video Action Recognition

Authors: Cameron Keith, SouYoung Jin

Organizations: SEE Lab, Dartmouth College

Abstract

Not every moment in a video is equally useful for recognizing an action. A few decisive moments often carry the evidence: the instant an object moves, the peak of a dive, or one object placed on another. Existing video datasets rarely provide large-scale, action-conditioned labels for which short segments carry that evidence. This work asks whether vision-language models can identify meaningful moments at scale and whether those labels improve action recognition. We introduce Meaningful Moments (MM), a corpus of about 4.58 million VLM-generated per-segment importance scores for nearly 500,000 class-labeled videos spanning Something-Something v2 (SSv2), Kinetics-400 (K400), and Diving-48 across 622 action classes. We freeze the action classifier and vary only temporal sampling. An alpha-parameterized comparison of importance-led selection against an importance-inverted control separates gains from the VLM signal from gains caused by variable-density sampling alone. Across all three datasets, VLM-derived importance is informative: at the hard-cut endpoint, keeping VLM-important segments outperforms keeping VLM-designated filler. Usefulness depends on class structure. On Diving-48, importance-led fast-forwarding preserves full-video recognition while the inverted control collapses by roughly 30 percentage points. On K400, hard-cut selection beats uniform sampling by 1.45 percentage points, rising to 2.05 under threshold selection. On SSv2, the labels carry signal but do not recover full-video performance. Direct scoring matches greedy removal within its noise floor at roughly 25 times fewer oracle calls, and a four-VLM cross-oracle study shows consistent downstream trends. We release the importance labels and evaluation protocol under CC-BY-4.0; SSv2 class strings remain under CC-BY-NC-4.0.

Figures & tables

Figure 1: A Meaningful Moments example. A K400 test-split clip labeled barbecuing is segmented at $\Delta t=1.0$ s; the number under each segment is the oracle’s continuous importance weight, and the border color marks the binary label (green VLM-important, red VLM-designated filler). Off-task content (a cat at 0–1 s, post-action and indoor scenes at 7–10 s) receives low weight, while the grilling action at 2–6 s forms the kept set.

Figure 2: The MM construction pipeline. Videos are segmented on a fixed grid, paired with their class label and a dataset-specific prompt, and scored in one oracle call. The response yields per-segment scores and a kept set; a two-stage logit rule yields precheck metadata.

Figure 3: Mean continuous weight and fraction labeled VLM-important by normalized temporal position (

20

bins, about

2{,}000

sampled train videos per dataset). Kept fraction peaks mid-clip and falls toward the boundaries (first/last bins: SSv2

0.11

0.05

, K400

0.37

0.12

, Diving-48

0.00

0.00

Dataset	Attempts	NO	SKIP	Passed	Fail %	Cls.
SSv2	220,847	10,715	290	209,842	4.98	174
K400	298,337	23,264	6	275,067	7.80	400
Diving-48	16,997	2,577	30	14,390	15.34	48
Total	536,181	36,556	326	499,299	6.88	622

Table 1: MM corpus composition. Parse failures are zero throughout, so “Fail %” is the combined precheck-NO and precheck-SKIP attrition rate. “Cls.” is each dataset’s label space;

47

of Diving-48’s

48

classes appear in the indices used here.

Figure 4: Variable-density sampling at $\alpha=0.25$ on a real eight-segment SSv2 example with a sixteen-frame budget (production labels keep $s_{2}$ through $s_{5}$ ). Cell shading shows the per-segment density $d_{i}$ and dots show frames allocated by largest-remainder rounding. Importance-Led FF concentrates frames on VLM-important segments; the inverted control concentrates the same budget on filler.

Dataset	full	v-sel	low-ev	Best $T$	Best $f$	$\Delta$
Diving-48	$87.82$	$82.08$	$52.13$	88.58 (0)	87.87 (90)	$+0.76$
SSv2	$65.77$	$62.45$	$53.41$	63.37 (50)	63.65 (70)	$-2.12$
K400	$83.95$	$85.40$	$80.15$	86.00 (70)	85.75 (30)	$+2.05$

Table 2: Headline results, top-1 (%): the uniform reference, both hard-cut endpoints, and the best threshold and budget operating points with the winning

T

f

in parentheses.

\Delta

is best minus full ; of the three, only the K400 entry survives Bonferroni correction (Table 3 ). v-sel and low-ev abbreviate vlm-selected and lowest-evidence .

Contrast	$n$	$\Delta$ (pp)	$95\%$ CI
$\alpha$ -sweep
D48 Imp-Led $\alpha{=}0.70$	1970	$+0.56$	$[-0.10,+1.27]$
SSv2 vlm-selected	2080	$-3.32^{\ast\dagger}$	$[-4.90,-1.73]$
K400 vlm-selected	2000	$+1.45^{\ast\dagger}$	$[+0.35,+2.55]$
$\alpha=0$ continuous-score endpoints
D48 $T{=}0$	1970	$+0.76^{\ast}$	$[+0.05,+1.47]$

Table 3: Paired-bootstrap CIs on all eight primary contrasts, each against full , from

B=10{,}000

resamples. “

\ast

” marks intervals excluding

0

at the raw

95\%

level, “

\dagger

” those additionally surviving Bonferroni correction over the

k=8

family. D48

T{=}0

is the one raw-significant contrast that does not survive; the two unmarked D48 rows were never significant.

Figure 5: Top-1 versus

\alpha

for Diving-48 (left), K400 (center), and SSv2 (right), Importance-Led FF against the matched inverted control. Diving-48 shows the intended signature, the arms fanning out as

\alpha\to 0

; K400’s interior is flat; SSv2’s interior curves overlap, the control marginally above, a protocol artifact discussed in the text. Decile tables are in the supplementary material.

Template	$n$	Full	Cut	$\Delta$ (pp)
Largest losses: boundary-state
Removing something, revealing something behind	77	36.36	15.58	$+20.78$
Dropping something next to something	106	36.79	16.04	$+20.75$
Putting something on a surface	492	51.83	31.10	$+20.73$
Laying something on the table on its side	145	35.86	16.55	$+19.31$
Largest gains: motion

Table 4: SSv2 per-template split under kept-set closed-set classification (templates with

n\geq 10

). Positive

\Delta

means full beats the cut. Losses are boundary-state-discriminated, their evidence being the clip-boundary state that cut-and-concatenate repacking disturbs; gains are motion-discriminated.

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

May 4, 2026

VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition

Videos are unique in their ability to capture actions which transcend multiple frames. Accordingly, for many years action recognition was the quintessential task for video understanding. Unfortunately, due to a lack of sufficiently diverse and challenging data, modern vision-language models (VLMs) are no longer evaluated on their action recognition capabilities. To revitalize action recognition in the era of VLMs, we advocate for a returned focus on domain-specific actions. To this end, we introduce VideoNet, a domain-specific action recognition benchmark covering 1,000 distinct actions from 37 domains. We begin with a multiple-choice evaluation setting, where the difference between closed and open models is stark: Gemini 3.1 Pro attains 69.9% accuracy while Qwen3-VL-8B gets a mere 45.0%. To understand why VLMs struggle on VideoNet, we relax the questions into a binary setting, where random chance is 50%. Still, Qwen achieves only 59.2% accuracy. Further relaxing the evaluation setup, we provide

k\in\{1,2,3\}

in-context examples of the action. Some models excel in the few-shot setting, while others falter; Qwen improves

+7.0\%

, while Gemini declines

-4.8\%

. Notably, these gains fall short of the

+13.6\%

improvement in non-expert humans when given few-shot examples. Finding that VLMs struggle to fully exploit in-context examples, we shift from test-time improvements to the training side. We collect the first large-scale training dataset for domain-specific actions, totaling nearly 500k video question-answer pairs. Fine-tuning a Molmo2-4B model on our data, we surpass all open-weight 8B models on the VideoNet benchmark.

Tanush Yadav, Mohammadreza Salehi, Jae Sung Park +6

University of Washington · Stanford University · Allen Institute for AI

VLM Evaluation VLM Adaptation Video QA Video Action Recognition

Ask PDF

Jun 13, 2026

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame-based inference. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.

Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3

University of Alicante · ETH Zürich · Microsoft

Egocentric Video Understanding In-Context Learning VLM Reasoning

Ask PDF

Jun 1, 2026

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

Video multimodal large language models (MLLMs) have made rapid progress in general and long-form video understanding, yet their ability to capture and use brief, answer-critical visual evidence remains underexplored. Momentary actions or state transitions may determine an answer while occupying only a small fraction of the video. Performance on broad video understanding tasks alone does not establish whether models can reliably identify and use this temporally localized evidence. We introduce Moment-Video, a benchmark for evaluating temporal fidelity through momentary visual event understanding. Questions are constructed around visually observable events or event sequences, requiring models to notice, count, describe, or reason about transient evidence. The benchmark contains 1,000 human-verified video-QA pairs across seven domains and 25 subcategories, covering four tasks: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. We evaluate 34 proprietary and open-source MLLMs. Under the main evaluation settings, the best-performing model, Seed-2.0-Pro, achieves 39.6% overall accuracy, while most open-source models remain below 25%. Diagnostic analyses show that denser frame sampling improves some models, but substantial errors persist even when sampled frames cover all annotated necessary event intervals in an audited subset. Performance also declines on longer videos, suggesting additional temporal-localization challenges. By exposing these limitations, Moment-Video provides a diagnostic benchmark to guide the evaluation and development of video MLLMs that more reliably understand brief, answer-critical visual events.

Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9

Shandong University · Southeast University · Shanghai Jiao Tong University +1

VLM Evaluation Video QA Video-Language Models Temporal Video Understanding

Ask PDF

Abstract

Temporal Video Understanding

Explore similar work

VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events