Meaningful Moments: A Corpus of VLM-Derived Temporal Importance Labels for Video Action Recognition

Cameron Keith; SouYoung Jin

cs.CVOct 12, 2026

Meaningful Moments: A Corpus of VLM-Derived Temporal Importance Labels for Video Action Recognition

Authors: Cameron Keith, SouYoung Jin

Organizations: SEE Lab, Dartmouth College

Abstract

Not every moment in a video is equally useful for recognizing an action. A few decisive moments often carry the evidence: the instant an object moves, the peak of a dive, or one object placed on another. Existing video datasets rarely provide large-scale, action-conditioned labels for which short segments carry that evidence. This work asks whether vision-language models can identify meaningful moments at scale and whether those labels improve action recognition. We introduce Meaningful Moments (MM), a corpus of about 4.58 million VLM-generated per-segment importance scores for nearly 500,000 class-labeled videos spanning Something-Something v2 (SSv2), Kinetics-400 (K400), and Diving-48 across 622 action classes. We freeze the action classifier and vary only temporal sampling. An alpha-parameterized comparison of importance-led selection against an importance-inverted control separates gains from the VLM signal from gains caused by variable-density sampling alone. Across all three datasets, VLM-derived importance is informative: at the hard-cut endpoint, keeping VLM-important segments outperforms keeping VLM-designated filler. Usefulness depends on class structure. On Diving-48, importance-led fast-forwarding preserves full-video recognition while the inverted control collapses by roughly 30 percentage points. On K400, hard-cut selection beats uniform sampling by 1.45 percentage points, rising to 2.05 under threshold selection. On SSv2, the labels carry signal but do not recover full-video performance. Direct scoring matches greedy removal within its noise floor at roughly 25 times fewer oracle calls, and a four-VLM cross-oracle study shows consistent downstream trends. We release the importance labels and evaluation protocol under CC-BY-4.0; SSv2 class strings remain under CC-BY-NC-4.0.

Figures & tables

Figure 1: A Meaningful Moments example. A K400 test-split clip labeled barbecuing is segmented at $\Delta t=1.0$ s; the number under each segment is the oracle’s continuous importance weight, and the border color marks the binary label (green VLM-important, red VLM-designated filler). Off-task content (a cat at 0–1 s, post-action and indoor scenes at 7–10 s) receives low weight, while the grilling action at 2–6 s forms the kept set.

Figure 2: The MM construction pipeline. Videos are segmented on a fixed grid, paired with their class label and a dataset-specific prompt, and scored in one oracle call. The response yields per-segment scores and a kept set; a two-stage logit rule yields precheck metadata.

Figure 3: Mean continuous weight and fraction labeled VLM-important by normalized temporal position (

20

bins, about

2{,}000

sampled train videos per dataset). Kept fraction peaks mid-clip and falls toward the boundaries (first/last bins: SSv2

0.11

0.05

, K400

0.37

0.12

, Diving-48

0.00

0.00

Dataset	Attempts	NO	SKIP	Passed	Fail %	Cls.
SSv2	220,847	10,715	290	209,842	4.98	174
K400	298,337	23,264	6	275,067	7.80	400
Diving-48	16,997	2,577	30	14,390	15.34	48
Total	536,181	36,556	326	499,299	6.88	622

Table 1: MM corpus composition. Parse failures are zero throughout, so “Fail %” is the combined precheck-NO and precheck-SKIP attrition rate. “Cls.” is each dataset’s label space;

47

of Diving-48’s

48

classes appear in the indices used here.

Figure 4: Variable-density sampling at $\alpha=0.25$ on a real eight-segment SSv2 example with a sixteen-frame budget (production labels keep $s_{2}$ through $s_{5}$ ). Cell shading shows the per-segment density $d_{i}$ and dots show frames allocated by largest-remainder rounding. Importance-Led FF concentrates frames on VLM-important segments; the inverted control concentrates the same budget on filler.

Dataset	full	v-sel	low-ev	Best $T$	Best $f$	$\Delta$
Diving-48	$87.82$	$82.08$	$52.13$	88.58 (0)	87.87 (90)	$+0.76$
SSv2	$65.77$	$62.45$	$53.41$	63.37 (50)	63.65 (70)	$-2.12$
K400	$83.95$	$85.40$	$80.15$	86.00 (70)	85.75 (30)	$+2.05$

Table 2: Headline results, top-1 (%): the uniform reference, both hard-cut endpoints, and the best threshold and budget operating points with the winning

T

f

in parentheses.

\Delta

is best minus full ; of the three, only the K400 entry survives Bonferroni correction (Table 3 ). v-sel and low-ev abbreviate vlm-selected and lowest-evidence .

Contrast	$n$	$\Delta$ (pp)	$95\%$ CI
$\alpha$ -sweep
D48 Imp-Led $\alpha{=}0.70$	1970	$+0.56$	$[-0.10,+1.27]$
SSv2 vlm-selected	2080	$-3.32^{\ast\dagger}$	$[-4.90,-1.73]$
K400 vlm-selected	2000	$+1.45^{\ast\dagger}$	$[+0.35,+2.55]$
$\alpha=0$ continuous-score endpoints
D48 $T{=}0$	1970	$+0.76^{\ast}$	$[+0.05,+1.47]$

Table 3: Paired-bootstrap CIs on all eight primary contrasts, each against full , from

B=10{,}000

resamples. “

\ast

” marks intervals excluding

0

at the raw

95\%

level, “

\dagger

” those additionally surviving Bonferroni correction over the

k=8

family. D48

T{=}0

is the one raw-significant contrast that does not survive; the two unmarked D48 rows were never significant.

Figure 5: Top-1 versus

\alpha

for Diving-48 (left), K400 (center), and SSv2 (right), Importance-Led FF against the matched inverted control. Diving-48 shows the intended signature, the arms fanning out as

\alpha\to 0

; K400’s interior is flat; SSv2’s interior curves overlap, the control marginally above, a protocol artifact discussed in the text. Decile tables are in the supplementary material.

Template	$n$	Full	Cut	$\Delta$ (pp)
Largest losses: boundary-state
Removing something, revealing something behind	77	36.36	15.58	$+20.78$
Dropping something next to something	106	36.79	16.04	$+20.75$
Putting something on a surface	492	51.83	31.10	$+20.73$
Laying something on the table on its side	145	35.86	16.55	$+19.31$
Largest gains: motion

Table 4: SSv2 per-template split under kept-set closed-set classification (templates with

n\geq 10

). Positive

\Delta

means full beats the cut. Losses are boundary-state-discriminated, their evidence being the clip-boundary state that cut-and-concatenate repacking disturbs; gains are motion-discriminated.

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition
May 4, 2026Tanush Yadav, Mohammadreza Salehi, Jae Sung Park +6VLM Evaluation VLM Adaptation
University of Washington · Stanford University · Allen Institute for AI
From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
Jun 13, 2026Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3Egocentric Video Understanding In-Context Learning
University of Alicante · ETH Zürich · Microsoft
Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
Jun 1, 2026Xiaolin Liu, Yilun Zhu, Xiangyu Zhao +9VLM Evaluation Video QA
Shandong University · Southeast University · Shanghai Jiao Tong University +1

May 4, 2026J/K move · Enter open · S save

VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition

Tanush Yadav, Mohammadreza Salehi, Jae Sung Park, Vivek Ramanujan+5

University of Washington · Stanford University · Allen Institute for AI

VLM Evaluation VLM Adaptation Video QA Video Action Recognition

Open paper AskPDF arXiv

Videos are unique in their ability to capture actions which transcend multiple frames. Accordingly, for many years action recognition was the quintessential task for video understanding. Unfortunately, due to a lack of sufficiently diverse and challenging data, modern vision-language models (VLMs) are no longer evaluated on their action recognition capabilities. To revitalize action recognition in the era of VLMs, we advocate for a returned focus on domain-specific actions. To this end, we introduce VideoNet, a domain-specific action recognition benchmark covering 1,000 distinct actions from 37 domains. We begin with a multiple-choice evaluation setting, where the difference between closed and open models is stark: Gemini 3.1 Pro attains 69.9% accuracy while Qwen3-VL-8B gets a mere 45.0%. To understand why VLMs struggle on VideoNet, we relax the questions into a binary setting, where random chance is 50%. Still, Qwen achieves only 59.2% accuracy. Further relaxing the evaluation setup, we provide

k\in\{1,2,3\}

in-context examples of the action. Some models excel in the few-shot setting, while others falter; Qwen improves

+7.0\%

, while Gemini declines

-4.8\%

. Notably, these gains fall short of the

+13.6\%

improvement in non-expert humans when given few-shot examples. Finding that VLMs struggle to fully exploit in-context examples, we shift from test-time improvements to the training side. We collect the first large-scale training dataset for domain-specific actions, totaling nearly 500k video question-answer pairs. Fine-tuning a Molmo2-4B model on our data, we surpass all open-weight 8B models on the VideoNet benchmark.

Abstract

Temporal Video Understanding

Explore similar work

VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition

From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events