MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation

Lahiru Wijayasingha

cs.CVOct 18, 2026

MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation

Authors: Lahiru Wijayasingha

Organizations: Independent Researcher

Abstract

We show that a video can be represented with just a few keyframes without sacrificing a given model's belief about the video. Our method is the first of its kind that extracts these keyframes conditioned on the model's prediction, which is massively helpful for building explainable video models. We then fairly distribute the importance of the video to the model among the keyframes with game theoretically fair Shapley values. Through qualitative and quantitative experiments, we show that keyframe attribution is superior to the traditional frame attribution. Further, we conduct preliminary exploration on highlighting the important changes among the frames by looking at the differences between adjacent keyframes.

Figures & tables

Figure 1: MoCKEFA framework. Keyframe extraction: model selects keyframes that are important for model’s decision. Continuous temporal regions can be represented with just a selected keyframe with minimal change to the model belief. Fair attribution: Shapley values are calculated for each keyframe that gives game-theoretically ”fair” attributions

Figure 2: Comparison of frame-wise (F) and group-wise (G) attribution. Keyframes have a red dot on the top. The three moments represent three stages of the bending of the left leg of the person.

Figure 3: Example of a distilled video. Frames 0 to 4 can be represented just by the frame 0. Frames 5, 6 and 7 are groups with just one frame. Frame from 8 to 15 can be represented with just the frame 14. Temporal Freezing can be used to create the new video with the original length. Check Sec 5.2 for metric definitions

Thr	$\mu_{k}$	$\Delta^{L}$	$\Delta^{m1}$	$\Delta^{m3}$	$\Delta^{m5}$	$\Delta^{entr}$	$Ag\%$
-1e-1	14.98	-0.011	-0.587	-0.078	-0.051	0.043	99.55
-1e-2	7.01	-0.024	-1.327	-0.131	-0.089	0.119	98.65
-1e-3	4.72	-0.016	-1.408	-0.118	-0.076	0.017	98.28
-1e-5	4.51	-0.015	-1.406	-0.116	-0.074	-0.020	98.31
1e-4	4.48	-0.022	-1.410	-0.116	-0.074	-0.021	98.28
5e-4	4.41	-0.014	-1.407	-0.115	-0.072	-0.037	98.26

Table 1: UCF101 Metrics after distillation

Thr	$\mu_{k}$	$\Delta^{L}$	$\Delta^{m1}$	$\Delta^{m3}$	$\Delta^{m5}$	$\Delta^{entr}$	$Ag\%$
-1	15.89	-0.006	-0.183	-0.077	-0.043	0.012	99.81
-1e-1	11.87	-0.031	-3.690	-0.272	-0.187	0.170	96.65
1e-5	6.45	0.039	-6.608	-0.116	-0.054	-0.566	90.52
1e-4	6.44	0.039	-6.606	-0.115	-0.053	-0.570	90.52
5e-4	6.45	0.038	-6.644	-0.117	-0.054	-0.566	90.61
1e-3	6.39	0.040	-6.641	-0.113	-0.051	-0.577	90.52

Table 2: SSV2 Metrics after distillation

Figure 4: Iterative distillation till the desired similarity to the original video is achieved. Top figure is the original video. Then from top to bottom we decrease the threshold to distill with more finer groups.

Method	UCF-101		SSV2
Method	L2	KL-Div	L2	KL-Div
future	0.00875	0.04164	0.01168	0.01321
past	0.00877	0.04182	0.01174	0.01396
late	0.00867	0.04116	0.01175	0.01400
mid	0.00872	0.04137	0.01174	0.01369
random	0.00870	0.04124	0.01169	0.01354

Table 3: We removed a single randomly chosen group from each crystallized video and filled in the missing values using different methods. Then we obtained the model embeddings of these filled videos. This table shows the difference in the distributions of the original embeddings and the filled embeddings. L2: Average sample-wise Euclidian distance. KL-DIV: Average sample-wise KL divergence

Panel of Figure 5 — Figure 5: Attribution examples - UCF101. Keyframes are shown with red a circle

Panel of Figure 6 — Figure 6: Attribution examples - SSV2. Keyframes are shown with red a circle

	Method	$AUC_{asc}\uparrow$	$AUC_{des}\downarrow$	$AUC_{rand}$	$AUC_{lr}$	$AUC_{rl}$
F	Partition	0.708	0.663	0.688	0.674	0.697
	IG	0.690	0.688	0.688	0.674	0.697
	Gradam	0.686	0.692	0.688	0.674	0.697
	Occlusion	0.693	0.681	0.689	0.674	0.697
G	Kernel	0.722	0.650	0.688	0.674	0.697
	Partition	0.725	0.646	0.688	0.674	0.697

Table 4: Removal test for the probability AUC of group attribution - UCF101. F- frame-wise attribution. G-Group-wise attribution

	Method	$AUC_{asc}\uparrow$	$AUC_{des}\downarrow$	$AUC_{rand}$	$AUC_{lr}$	$AUC_{rl}$
F	Partition	0.743	0.706	0.725	0.717	0.732
	IG	0.726	0.726	0.726	0.717	0.732
	Gradam	0.723	0.729	0.727	0.717	0.732
	Occlusion	0.730	0.720	0.726	0.717	0.732
G	Kernel	0.752	0.697	0.726	0.717	0.732
	Partition	0.755	0.693	0.726	0.717	0.732

Table 5: Removal test for the Logit AUC of group attribution - UCF101. F- frame-wise attribution. G-Group-wise attribution

	Method	$AUC_{asc}\uparrow$	$AUC_{des}\downarrow$	$AUC_{rand}$	$AUC_{lr}$	$AUC_{rl}$
F	Partition	0.516	0.380	0.462	0.441	0.402
	Gradam	0.471	0.477	0.463	0.441	0.402
	Occlusion	0.471	0.439	0.463	0.441	0.402
G	Partition	0.550	0.340	0.460	0.441	0.402

Table 6: Removal test for the probability AUC of group attribution - SSV2. F- frame-wise attribution. G-Group-wise attribution

	Method	$AUC_{asc}\uparrow$	$AUC_{des}\downarrow$	$AUC_{rand}$	$AUC_{lr}$	$AUC_{rl}$
F	Partition	0.621	0.505	0.577	0.549	0.529
	Gradam	0.584	0.585	0.578	0.549	0.529
	Occlusion	0.586	0.556	0.576	0.549	0.529
G	Partition	0.646	0.470	0.576	0.549	0.529

Table 7: Removal test for the logit AUC of group attribution - SSV2. F- frame-wise attribution. G-Group-wise attribution

Figure 7: Effect of distillation quality on attribution

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 330 papers

Sufficient Frame Subsets for Video Understanding
Oct 18, 2026Lahiru WijayasinghaNeural Network Interpretability Video Understanding
Independent Researcher
MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models
Sep 22, 2026Anirudh Prabhakaran, Alexandre Rocchi, Gianni FranchiFeature Attribution SHAP Feature Attribution
AMIAD, Pˆole Recherche, Palaiseau
When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs
Oct 13, 2026Anubhav Kataria, Shreya Ghosh, Jingwen Ye +1VLM Interpretability Facial Expression Recognition
Monash University, Melbourne, Australia · Indian Institute of Technology Kharagpur, Kharagpur, India

Oct 18, 2026J/K move · Enter open · S save

Sufficient Frame Subsets for Video Understanding

Lahiru Wijayasingha

Independent Researcher

Neural Network Interpretability Video Understanding Temporal Video Understanding Video Action Recognition

Open paper AskPDF arXiv

Understanding when during a video a model makes its critical observations is essential for interpreting video-understanding systems. However, existing attribution methods often spread evidence diffusely across near-duplicate frames. We introduce Sufficient Frame Subsets: compact sets of frames in which all other frames are masked, yet each subset independently preserves the model's prediction. We evaluate our method on two complementary benchmarks, UCF101 and Something-Something v2, using both CNN- and transformer-based architectures. Our results show that the proposed algorithm identifies highly compact and effective frame subsets for explaining model decisions. We also find that models capable of relying on a greater number of sufficient frame subsets tend to perform better, possibly because they are more robust, consistent with prior research. Finally, the extracted frame subsets help us analyze model failure cases, reveal which aspects of a video the model has learned to emphasize, and identify opportunities for improving future training procedures.

Abstract

Neural Network Interpretability

Explore similar work

Sufficient Frame Subsets for Video Understanding

MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models

When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs