Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning

Lingyu Shen; Wei Tang; Fakhri Karray; Min-Ling Zhang

cs.CVOct 5, 2026

Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning

Authors: Lingyu Shen, Wei Tang, Fakhri Karray, Min-Ling Zhang

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates

Abstract

Multi-instance partial-label learning (MIPL) addresses inexact supervision in both the instance and label spaces, which can be applied to video classification. However, bag-level labels do not explicitly supervise the correspondence between candidate classes and temporal evidence. We propose {\ours}, which couples label disambiguation with temporal evidence allocation through a joint class--time assignment. Occupancy-regularized spherical matching associates contextualized video features while learning nonuniform temporal mass and discouraging excessive concentration. During training, candidate-restricted inference recomputes the assignment within the candidate label set. A dual-marginal KL projection then constructs a structured teacher that incorporates momentum-refined class beliefs while preserving the proposal's temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and shows that, under the stated construction, the joint objective decomposes into class-marginal and class-conditional temporal supervision. We construct VCMIPL benchmarks from Breakfast, DoTA, and FineAction using model-generated candidate labels and evaluate the method across four feature representations. Extensive experimental results demonstrate that PIVOTMIPL outperforms existing MIPL algorithms in both effectiveness and efficiency.

Figures & tables

1	Initialize parameters and $q_{ic}^{(0)}=\mathbf{1}[c\in S_{i}]/\|S_{i}\|$ .
2	For each epoch $e$ and each training bag $(X_{i},S_{i})$ :
3	Encode valid ordered positions and compute $\mathbf{C}_{i}$ .
4	Solve full-space actor $\bm{\Pi}_{i}^{A}=\mathcal{T}(\mathbf{C}_{i};\mathcal{Y})$ .
5	Solve proposal $\widetilde{\bm{\Pi}}_{i}=\mathcal{T}(\operatorname{sg}(\mathbf{C}_{i});S_{i})$ .
6	Update class belief using Eq. ( 16 ).
7	Scale the proposal to $(\mathbf{q}_{i}^{(e)},\widetilde{\mathbf{m}}_{i})$ using Eq. ( 18 ).

Table 2

Dataset	Method	DINOv3	VideoMAEv2	ResNet	SlowFast
Breakfast -MIPL	DeMipl	$11.4\%\,{\scriptstyle\pm\,0.4\%}\,\bullet$	$25.7\%\,{\scriptstyle\pm\,10.0\%}\,\bullet$	$10.1\%\,{\scriptstyle\pm\,0.7\%}\,\bullet$	$11.0\%\,{\scriptstyle\pm\,0.5\%}\,\bullet$
	EliMipl	$27.8\%\,{\scriptstyle\pm\,0.8\%}\,\bullet$	$52.2\%\,{\scriptstyle\pm\,1.9\%}\,\bullet$	$26.1\%\,{\scriptstyle\pm\,1.7\%}\,\bullet$	$11.3\%\,{\scriptstyle\pm\,0.1\%}\,\bullet$
	MiplMa	$33.9\%\,{\scriptstyle\pm\,1.5\%}\,\bullet$	$63.6\%\,{\scriptstyle\pm\,2.1\%}\,\bullet$	$30.5\%\,{\scriptstyle\pm\,2.4\%}\,\bullet$	$11.3\%\,{\scriptstyle\pm\,0.1\%}\,\bullet$
	ProMipl	$\underline{62.2\%}\,{\scriptstyle\pm\,1.2\%}\,\bullet$	$83.8\%\,{\scriptstyle\pm\,3.4\%}\,\bullet$	$\underline{43.7\%}\,{\scriptstyle\pm\,1.2\%}\,\bullet$	$41.6\%\,{\scriptstyle\pm\,1.1\%}\,\bullet$
	PsMipl	$47.3\%\,{\scriptstyle\pm\,5.0\%}\,\bullet$	$70.3\%\,{\scriptstyle\pm\,7.3\%}\,\bullet$	$11.0\%\,{\scriptstyle\pm\,1.4\%}\,\bullet$	$33.0\%\,{\scriptstyle\pm\,0.6\%}\,\bullet$
	DualG	$45.7\%\,{\scriptstyle\pm\,3.2\%}\,\bullet$	$64.9\%\,{\scriptstyle\pm\,4.4\%}\,\bullet$	$34.3\%\,{\scriptstyle\pm\,2.0\%}\,\bullet$	$14.4\%\,{\scriptstyle\pm\,1.6\%}\,\bullet$

Table 1: Classification accuracy on the benchmark datasets. Results are reported as (mean

\pm

std %). The best result in each column is bolded, and the second-best results are underlined.

\bullet

\circ

indicates whether PivotMipl is statistically superior/inferior to the corresponding method under a two-sided paired

t

-test at

p<0.05

Dataset	Feature	PivotMipl	A1	A2	A3	A4	A5	A6
Breakfast -MIPL	DINOv3	$\underline{66.4\%}$	$62.6\%$	$\mathbf{72.1\%}$	$66.1\%$	$64.8\%$	$64.3\%$	$57.1\%$
	VideoMAEv2	$\underline{77.4\%}$	$\mathbf{77.8\%}$	$77.1\%$	$70.3\%$	$67.1\%$	$72.6\%$	$75.6\%$
	ResNet	$\underline{52.2\%}$	$42.6\%$	$\mathbf{54.4\%}$	$51.4\%$	$50.3\%$	$47.9\%$	$35.4\%$
	SlowFast	$\mathbf{53.9\%}$	$48.4\%$	$53.4\%$	$52.6\%$	$51.6\%$	$\underline{53.6\%}$	$49.9\%$
DoTA -MIPL	DINOv3	$\mathbf{54.3\%}$	$50.3\%$	$51.7\%$	$\underline{53.5\%}$	$52.6\%$	$51.3\%$	$49.3\%$
	VideoMAEv2	$\underline{72.8\%}$	$72.0\%$	$70.9\%$	$72.2\%$	$72.7\%$	$\mathbf{73.0\%}$	$72.7\%$

Table 2: Ablation study of PivotMipl .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Videos	Classes	Mean $\|S_{i}\|$	Range $\|S_{i}\|$
Breakfast	1,989	10	3.3896	1–9
DoTA	4,577	9	3.2733	1–8
FineAction	10,675	78	11.5592	2–64

Appendix

Table 3: Video-level statistics of the constructed MIPL benchmarks.

Feature	Total instances	Maximum	Minimum	Mean	Dimension
Breakfast: 1,989 videos, 10 classes, 100 fine-tuning epochs
Candidate size: 1–9, mean 3.3896
ResNet	4,103,316	9,745	186	2,063.00	2,048
DINOv3	4,103,316	9,745	186	2,063.00	4,096
SlowFast	1,026,577	2,437	47	516.13	2,304
VideoMAEv2	1,026,577	2,437	47	516.13	768

Appendix

Table 4: Instance and feature statistics for all twelve dataset–feature configurations.

Setting	Value or implementation
Framework / hardware	PyTorch / NVIDIA H100
Optimizer	AdamW; $(\beta_{1},\beta_{2})=(0.9,0.999)$ ; weight decay $0$
Learning rates	Prototypes: $5\times 10^{-3}$ ; temporal encoder: $5\times 10^{-4}$
Gradient clipping	Global $\ell_{2}$ norm clipped to $5.0$
Training budget	$40$ epochs
Motion module	Average-filter kernel $3$ ; convolution kernel $5$ ; dilations $1,2$ ; motion-residual dropout $0.1$

Appendix

Table 5: Experimental settings and configurable implementation defaults of PivotMipl .

Dataset	Method	DINOv3	VideoMAEv2	ResNet	SlowFast
Breakfast -MIPL	DeMipl	$10.2\%\,{\scriptstyle\pm\,0.4\%}$	$24.4\%\,{\scriptstyle\pm\,10.2\%}$	$10.6\%\,{\scriptstyle\pm\,0.6\%}$	$10.0\%\,{\scriptstyle\pm\,0.0\%}$
	EliMipl	$27.9\%\,{\scriptstyle\pm\,0.8\%}$	$52.4\%\,{\scriptstyle\pm\,2.1\%}$	$26.0\%\,{\scriptstyle\pm\,1.9\%}$	$10.0\%\,{\scriptstyle\pm\,0.0\%}$
	MiplMa	$34.2\%\,{\scriptstyle\pm\,1.5\%}$	$63.8\%\,{\scriptstyle\pm\,2.0\%}$	$30.8\%\,{\scriptstyle\pm\,2.6\%}$	$10.0\%\,{\scriptstyle\pm\,0.0\%}$
	ProMipl	$\underline{61.9\%}\,{\scriptstyle\pm\,1.1\%}$	$83.3\%\,{\scriptstyle\pm\,3.5\%}$	$\underline{43.7\%}\,{\scriptstyle\pm\,1.0\%}$	$41.4\%\,{\scriptstyle\pm\,1.1\%}$
	PsMipl	$47.7\%\,{\scriptstyle\pm\,4.9\%}$	$70.1\%\,{\scriptstyle\pm\,7.1\%}$	$10.9\%\,{\scriptstyle\pm\,1.4\%}$	$32.9\%\,{\scriptstyle\pm\,0.5\%}$
	DualG	$45.2\%\,{\scriptstyle\pm\,2.7\%}$	$65.1\%\,{\scriptstyle\pm\,4.1\%}$	$34.7\%\,{\scriptstyle\pm\,2.2\%}$	$13.8\%\,{\scriptstyle\pm\,1.7\%}$

Appendix

Table 6: Balanced accuracy on the benchmark datasets. Results are reported as (mean

\pm

std %). The best result in each column is bolded, and the second-best results are underlined.

Figure 2: Accuracy–training-time trade-off across the twelve dataset–feature configurations. Each panel corresponds to one dataset–feature setting and compares PivotMipl with the MIPL baselines in terms of mean classification accuracy and total training time over five folds. Higher accuracy and shorter training time are preferred.

Variant	Modification	Full-model wins	Mean gain
A1	No occupancy regularization	10/12	$+3.25\%$
A2	Uniform temporal occupancy	9/12	$+0.72\%$
A3	Class-level KL	12/12	$+3.26\%$
A4	Class-marginal-only teacher	12/12	$+2.53\%$
A5	Single prototype per class	11/12	$+2.52\%$
A6	No temporal encoder	11/12	$+5.77\%$

Appendix

Table 7: Summary of the ablations across twelve dataset–feature configurations.

Dataset	Feature	AgopMipl	AgopMipl $+$ Temporal Encoder	PivotMipl
Breakfast	DINOv3	$46.4\%$	$\underline{53.1\%}$	$\mathbf{66.4\%}$
	VideoMAEv2	$\mathbf{79.3\%}$	$\underline{78.1\%}$	$77.4\%$
	ResNet	$31.4\%$	$\underline{45.4\%}$	$\mathbf{52.2\%}$
	SlowFast	$50.6\%$	$\underline{53.1\%}$	$\mathbf{53.9\%}$
DoTA	DINOv3	$\underline{49.2\%}$	$48.4\%$	$\mathbf{54.3\%}$
	VideoMAEv2	$69.5\%$	$\underline{69.9\%}$	$\mathbf{72.8\%}$

Appendix

Table 8: Accuracy for AgopMipl , AgopMipl augmented with the same temporal encoder architecture as PivotMipl , and PivotMipl . The best and second-best values within each row are shown in bold and underlined, respectively.

Figure 3: Sensitivity of PivotMipl to the entropic coefficient

\varepsilon

and occupancy coefficient

\lambda

on three representative dataset–feature configurations.

Explore similar work

Dec 19, 2025cs.LG

Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning

Multi-instance partial-label learning (MIPL) is a weakly supervised framework that extends the principles of multi-instance learning (MIL) and partial-label learning (PLL) to address the challenges of inexact supervision in both instance and label spaces. However, existing MIPL approaches often suffer from poor calibration, undermining classifier reliability. In this work, we propose a plug-and-play calibratable disambiguation loss (CDL) for classification and calibration, which modulates a disambiguation objective by a top-vs-competitor prediction margin. The competitor is instantiated either as the second strongest candidate label or as the strongest non-candidate label, yielding two variants that respectively emphasize candidate-level separation and candidate-vs-non-candidate suppression. Theoretically, we analyze CDL as a margin-modulated momentum-based disambiguation loss (MDL) objective, derive a lower-bound and a pseudo-label confidence-alignment bound for calibration, and show through gradient and momentum analyses how margin shaping affects weight updates. Experimental results on benchmark and real-world MIPL datasets, together with representative PLL adaptation, confirm that our CDL significantly improves both classification accuracy and expected calibration error.

Wei Tang, Yin-Fang Yang, Weijia Zhang +1

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration (Southeast University), MoE, China · School of Computer and Information Sciences, The University of Newcastle, Callaghan, NSW 2308, Australia

Multiple Instance Learning Disambiguation Suppression

Ask PDF

May 5, 2026cs.CV

Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence

The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/

Zhiyuan Li, Rongzhen Zhao, Wenyan Yang +3

Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland

Object-Centric Representations

Ask PDF

May 7, 2026cs.CV

Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval

Partially relevant video retrieval aims to retrieve untrimmed videos using text queries that describe only partial content. However, the inherent asymmetry between brief queries and rich video content inevitably introduces uncertainty into the retrieval process. In this setting, vague queries often induce semantic ambiguity across videos, a challenge that is further exacerbated by the sparse temporal supervision within videos, which fails to provide sufficient matching evidence. To address this, we propose Holmes, a hierarchical evidential learning framework that aggregates multi-granular cross-modal evidence to quantify and model uncertainty explicitly. At the inter-video level, similarity scores are interpreted as evidential support and modeled via a Dirichlet distribution. Based on the proposed three-fold principle, we perform fine-grained query identification, which then guides query-adaptive calibrated learning. At the intra-video level, to accumulate denser evidence, we formulate a soft query-clip alignment via flexible optimal transport with an adaptive dustbin, which alleviates sparse temporal supervision while suppressing spurious local responses. Extensive experiments demonstrate that Holmes outperforms state-of-the-art methods. Code is released at https://github.com/lijun2005/ICML26-Holmes.

Jun Li, Peifeng Lai, Xuhang Lou +5

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China

Fine-Grained Video Understanding Multimodal Retrieval Evidential Uncertainty Uncertainty

Ask PDF