Drop the Details, Get the Gist: Dynamic Conceptual Representation for Video Recognition

Ryoya Yoshiyama; Bowen Wang; Yuta Nakashima

cs.CVOct 27, 2026

Drop the Details, Get the Gist: Dynamic Conceptual Representation for Video Recognition

Authors: Ryoya Yoshiyama, Bowen Wang, Yuta Nakashima

Organizations: The University of Osaka, Japan

Abstract

Concept Bottleneck Models (CBMs), which route predictions through a layer of human-interpretable concepts, have substantially improved the interpretability of visual classification by exposing an explicit, language-based rationale for each prediction. Most existing work focuses on static images, and temporal dynamics in videos have been underexplored in conceptual representation. We propose a framework that extends concept-based image recognition to procedural video understanding. It represents a multi-step video as a temporal sequence of segment-level concept score vectors and feeds this sequence into a Transformer encoder to predict the class label. We also explore a way to estimate the contribution of each concept for further interpretability. We evaluated three concept sets of different granularities on cooking and household-activity videos and found that coarser-grained concepts are sufficient for classifying these videos, which implies that fine-grained textual descriptions may be unnecessary.

Figures & tables

Figure 1: Our approach mirrors human video understanding: a video is parsed into a sequence of semantic events (concepts), and a temporal reasoning module integrates these events to produce an interpretable class prediction.

Figure 2: Overview of the proposed framework. Frozen encoders $E_{V}$ and $E_{T}$ compute the similarity between each video segment and each concept, and a Transformer trained on the resulting temporal concept scores predicts the class.

Figure 3: An overview of concept set construction.

Concept set	Example concepts
$\mathcal{C}_{\text{raw}}$	add soy sauce fish sauce and salt, cut the sushi roll into pieces, cook the dough in a pan, heat a pan and add oil, place the onions in the pan, stir the soup, brush the pastry with butter …
$\mathcal{C}_{\text{verb}}$	drain a sliced potato into a bowl, chop a tuna, fry a sausage in a pan, sprinkle cheese on a pizza, knead dough well, whisk yogurt and water, serve boiled eggs …
$\mathcal{C}_{\text{oa}}$	cook a sauce, add vanilla, wrap a hot dog, spread sauce, mince ginger, maple syrup, green chilli, honey, crouton …

Table 1: Representative examples of concepts from each concept set.

	YouCook2			ActivityNet
Ours	$\mathcal{C}_{\text{raw}}$	$\mathcal{C}_{\text{verb}}$	$\mathcal{C}_{\text{oa}}$	$\mathcal{C}_{\text{raw}}$	$\mathcal{C}_{\text{verb}}$	$\mathcal{C}_{\text{oa}}$
All concepts	$72.4\pm 1.1$	$75.3\pm 0.9$	$73.7\pm 1.4$	$91.4\pm 0.3$	$89.7\pm 0.2$	$90.2\pm 0.6$
Top-5	$84.5\pm 1.8$	$85.3\pm 1.5$	$87.2\pm 0.6$	$91.8\pm 0.1$	$92.6\pm 0.2$	$93.0\pm 0.3$
Top-10	$88.6\pm 0.6$	$87.7\pm 0.9$	$88.6\pm 0.7$	$92.1\pm 0.1$	$93.1\pm 0.1$	$93.5\pm 0.2$
Top-20	$88.8\pm 0.3$	$89.3\pm 0.5$	$89.5\pm 1.0$	$93.5\pm 0.5$	$93.0\pm 0.2$	$93.3\pm 0.3$
Top-40	$89.4\pm 0.5$	$89.1\pm 0.8$	$88.4\pm 0.8$	$93.1\pm 0.3$	$93.5\pm 0.2$	$93.4\pm 0.2$

Table 2: Top-1 accuracy (%) on YouCook2 and ActivityNet (Household Activities). Columns indicate the concept set (

\mathcal{C}_{\text{raw}}

: raw step descriptions,

\mathcal{C}_{\text{verb}}

: verb phrases,

\mathcal{C}_{\text{oa}}

: object-and-action). Results are the mean and standard deviation over 10 runs for YouCook2 and 3 runs for ActivityNet.

Figure 4: Visualization of the prediction rationale based on the importance score $q_{i}$ for two correctly classified test videos, “miso soup” and “spaghetti and meatballs.”

Figure 5: Visualization of the important concepts for an incorrectly classified test video. Concepts are ranked by $|q_{i}|$ with respect to the ground-truth class.

Model variant	Top-1 Acc. (%)
w/ Transformer	$89.5\pm 1.0$
w/o Transformer	$86.0\pm 0.4$

Table 3: Effect of temporal modeling (YouCook2, concept set

\mathcal{C}_{\text{oa}}

, Top-20).

Concept set	Top-1 Acc. (%)
$\mathcal{C}_{\text{oa}}$ (dataset-specific)	$89.5\pm 1.0$
LLM-generated (dataset-agnostic)	$86.6\pm 0.9$

Table 4: Effect of the concept source (YouCook2, Top-20).

Video Understanding

Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.

See all 116 papers

Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test
Oct 23, 2026Jianghuai Li, Andrew Ren, Greg ShakhnarovichSpatiotemporal Transformers Statistical Hypothesis Testing
University of Chicago · Toyota Technological Institute at Chicago
Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
May 13, 2026Nicola Debole, Andrea Passerini, Stefano Teso +2Concept Bottleneck Models Human-in-the-Loop Annotation
DISI, University of Trento, Italy · CIMeC, University of Trento, Italy
Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
Oct 18, 2026Junwei Wen, Deshui Miao, Guangming Lu +3Temporal Reasoning in VLMs Video Object Segmentation
Harbin Institute of Technology, Shenzhen · University of California at Merced · Pengcheng Laboratory +1

Oct 23, 2026J/K move · Enter open · S save

Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test

Jianghuai Li, Andrew Ren, Greg Shakhnarovich

University of Chicago · Toyota Technological Institute at Chicago

Spatiotemporal Transformers Statistical Hypothesis Testing

Open paper AskPDF arXiv

Video models have the architecture to support encoding temporal aspects of the input, i.e. concepts that require more than one frame to define, like object movement. However, because motion and appearance change are entangled, a feature that appears to encode temporal concepts like object movement may instead encode non-temporal concepts like spatial presence at different positions whose activations merely correlate with motion. To resolve this, we introduce the No-False-Positives (NFP) test, a statistical test with a formal guarantee of statistical specificity for identifying features encoding temporal concepts. Using specially designed stimuli, we apply this test to representations produced by VideoMAE, V-JEPA2, and TimeSformer, and observe that temporal features exist in each model. We validate the test first by showing it yields no temporal features for the DINOv2 image-encoder. Then, using a synthetic representation space, we show that detection rate increases with a direction's projection onto the temporal subspace, and that Sparse Autoencoder features test-flagged as temporal project much more of their norm onto the temporal subspace than unflagged features. Interventions on temporal SAE features yield mixed results across models, suggesting that intervening on temporal features may not outperform a non-temporal feature baseline.

Abstract

Video Understanding

Explore similar work

Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation