Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning

Lingyu Shen; Wei Tang; Fakhri Karray; Min-Ling Zhang

cs.CVOct 5, 2026

Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning

Authors: Lingyu Shen, Wei Tang, Fakhri Karray, Min-Ling Zhang

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates

Abstract

Multi-instance partial-label learning (MIPL) addresses inexact supervision in both the instance and label spaces, which can be applied to video classification. However, bag-level labels do not explicitly supervise the correspondence between candidate classes and temporal evidence. We propose {\ours}, which couples label disambiguation with temporal evidence allocation through a joint class--time assignment. Occupancy-regularized spherical matching associates contextualized video features while learning nonuniform temporal mass and discouraging excessive concentration. During training, candidate-restricted inference recomputes the assignment within the candidate label set. A dual-marginal KL projection then constructs a structured teacher that incorporates momentum-refined class beliefs while preserving the proposal's temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and shows that, under the stated construction, the joint objective decomposes into class-marginal and class-conditional temporal supervision. We construct VCMIPL benchmarks from Breakfast, DoTA, and FineAction using model-generated candidate labels and evaluate the method across four feature representations. Extensive experimental results demonstrate that PIVOTMIPL outperforms existing MIPL algorithms in both effectiveness and efficiency.

Figures & tables

1	Initialize parameters and $q_{ic}^{(0)}=\mathbf{1}[c\in S_{i}]/\|S_{i}\|$ .
2	For each epoch $e$ and each training bag $(X_{i},S_{i})$ :
3	Encode valid ordered positions and compute $\mathbf{C}_{i}$ .
4	Solve full-space actor $\bm{\Pi}_{i}^{A}=\mathcal{T}(\mathbf{C}_{i};\mathcal{Y})$ .
5	Solve proposal $\widetilde{\bm{\Pi}}_{i}=\mathcal{T}(\operatorname{sg}(\mathbf{C}_{i});S_{i})$ .
6	Update class belief using Eq. ( 16 ).
7	Scale the proposal to $(\mathbf{q}_{i}^{(e)},\widetilde{\mathbf{m}}_{i})$ using Eq. ( 18 ).

Table 2

Dataset	Method	DINOv3	VideoMAEv2	ResNet	SlowFast
Breakfast -MIPL	DeMipl	$11.4\%\,{\scriptstyle\pm\,0.4\%}\,\bullet$	$25.7\%\,{\scriptstyle\pm\,10.0\%}\,\bullet$	$10.1\%\,{\scriptstyle\pm\,0.7\%}\,\bullet$	$11.0\%\,{\scriptstyle\pm\,0.5\%}\,\bullet$
	EliMipl	$27.8\%\,{\scriptstyle\pm\,0.8\%}\,\bullet$	$52.2\%\,{\scriptstyle\pm\,1.9\%}\,\bullet$	$26.1\%\,{\scriptstyle\pm\,1.7\%}\,\bullet$	$11.3\%\,{\scriptstyle\pm\,0.1\%}\,\bullet$
	MiplMa	$33.9\%\,{\scriptstyle\pm\,1.5\%}\,\bullet$	$63.6\%\,{\scriptstyle\pm\,2.1\%}\,\bullet$	$30.5\%\,{\scriptstyle\pm\,2.4\%}\,\bullet$	$11.3\%\,{\scriptstyle\pm\,0.1\%}\,\bullet$
	ProMipl	$\underline{62.2\%}\,{\scriptstyle\pm\,1.2\%}\,\bullet$	$83.8\%\,{\scriptstyle\pm\,3.4\%}\,\bullet$	$\underline{43.7\%}\,{\scriptstyle\pm\,1.2\%}\,\bullet$	$41.6\%\,{\scriptstyle\pm\,1.1\%}\,\bullet$
	PsMipl	$47.3\%\,{\scriptstyle\pm\,5.0\%}\,\bullet$	$70.3\%\,{\scriptstyle\pm\,7.3\%}\,\bullet$	$11.0\%\,{\scriptstyle\pm\,1.4\%}\,\bullet$	$33.0\%\,{\scriptstyle\pm\,0.6\%}\,\bullet$
	DualG	$45.7\%\,{\scriptstyle\pm\,3.2\%}\,\bullet$	$64.9\%\,{\scriptstyle\pm\,4.4\%}\,\bullet$	$34.3\%\,{\scriptstyle\pm\,2.0\%}\,\bullet$	$14.4\%\,{\scriptstyle\pm\,1.6\%}\,\bullet$

Table 1: Classification accuracy on the benchmark datasets. Results are reported as (mean

\pm

std %). The best result in each column is bolded, and the second-best results are underlined.

\bullet

\circ

indicates whether PivotMipl is statistically superior/inferior to the corresponding method under a two-sided paired

t

-test at

p<0.05

Dataset	Feature	PivotMipl	A1	A2	A3	A4	A5	A6
Breakfast -MIPL	DINOv3	$\underline{66.4\%}$	$62.6\%$	$\mathbf{72.1\%}$	$66.1\%$	$64.8\%$	$64.3\%$	$57.1\%$
	VideoMAEv2	$\underline{77.4\%}$	$\mathbf{77.8\%}$	$77.1\%$	$70.3\%$	$67.1\%$	$72.6\%$	$75.6\%$
	ResNet	$\underline{52.2\%}$	$42.6\%$	$\mathbf{54.4\%}$	$51.4\%$	$50.3\%$	$47.9\%$	$35.4\%$
	SlowFast	$\mathbf{53.9\%}$	$48.4\%$	$53.4\%$	$52.6\%$	$51.6\%$	$\underline{53.6\%}$	$49.9\%$
DoTA -MIPL	DINOv3	$\mathbf{54.3\%}$	$50.3\%$	$51.7\%$	$\underline{53.5\%}$	$52.6\%$	$51.3\%$	$49.3\%$
	VideoMAEv2	$\underline{72.8\%}$	$72.0\%$	$70.9\%$	$72.2\%$	$72.7\%$	$\mathbf{73.0\%}$	$72.7\%$

Table 2: Ablation study of PivotMipl .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Videos	Classes	Mean $\|S_{i}\|$	Range $\|S_{i}\|$
Breakfast	1,989	10	3.3896	1–9
DoTA	4,577	9	3.2733	1–8
FineAction	10,675	78	11.5592	2–64

Appendix

Table 3: Video-level statistics of the constructed MIPL benchmarks.

Feature	Total instances	Maximum	Minimum	Mean	Dimension
Breakfast: 1,989 videos, 10 classes, 100 fine-tuning epochs
Candidate size: 1–9, mean 3.3896
ResNet	4,103,316	9,745	186	2,063.00	2,048
DINOv3	4,103,316	9,745	186	2,063.00	4,096
SlowFast	1,026,577	2,437	47	516.13	2,304
VideoMAEv2	1,026,577	2,437	47	516.13	768

Appendix

Table 4: Instance and feature statistics for all twelve dataset–feature configurations.

Setting	Value or implementation
Framework / hardware	PyTorch / NVIDIA H100
Optimizer	AdamW; $(\beta_{1},\beta_{2})=(0.9,0.999)$ ; weight decay $0$
Learning rates	Prototypes: $5\times 10^{-3}$ ; temporal encoder: $5\times 10^{-4}$
Gradient clipping	Global $\ell_{2}$ norm clipped to $5.0$
Training budget	$40$ epochs
Motion module	Average-filter kernel $3$ ; convolution kernel $5$ ; dilations $1,2$ ; motion-residual dropout $0.1$

Appendix

Table 5: Experimental settings and configurable implementation defaults of PivotMipl .

Dataset	Method	DINOv3	VideoMAEv2	ResNet	SlowFast
Breakfast -MIPL	DeMipl	$10.2\%\,{\scriptstyle\pm\,0.4\%}$	$24.4\%\,{\scriptstyle\pm\,10.2\%}$	$10.6\%\,{\scriptstyle\pm\,0.6\%}$	$10.0\%\,{\scriptstyle\pm\,0.0\%}$
	EliMipl	$27.9\%\,{\scriptstyle\pm\,0.8\%}$	$52.4\%\,{\scriptstyle\pm\,2.1\%}$	$26.0\%\,{\scriptstyle\pm\,1.9\%}$	$10.0\%\,{\scriptstyle\pm\,0.0\%}$
	MiplMa	$34.2\%\,{\scriptstyle\pm\,1.5\%}$	$63.8\%\,{\scriptstyle\pm\,2.0\%}$	$30.8\%\,{\scriptstyle\pm\,2.6\%}$	$10.0\%\,{\scriptstyle\pm\,0.0\%}$
	ProMipl	$\underline{61.9\%}\,{\scriptstyle\pm\,1.1\%}$	$83.3\%\,{\scriptstyle\pm\,3.5\%}$	$\underline{43.7\%}\,{\scriptstyle\pm\,1.0\%}$	$41.4\%\,{\scriptstyle\pm\,1.1\%}$
	PsMipl	$47.7\%\,{\scriptstyle\pm\,4.9\%}$	$70.1\%\,{\scriptstyle\pm\,7.1\%}$	$10.9\%\,{\scriptstyle\pm\,1.4\%}$	$32.9\%\,{\scriptstyle\pm\,0.5\%}$
	DualG	$45.2\%\,{\scriptstyle\pm\,2.7\%}$	$65.1\%\,{\scriptstyle\pm\,4.1\%}$	$34.7\%\,{\scriptstyle\pm\,2.2\%}$	$13.8\%\,{\scriptstyle\pm\,1.7\%}$

Appendix

Table 6: Balanced accuracy on the benchmark datasets. Results are reported as (mean

\pm

std %). The best result in each column is bolded, and the second-best results are underlined.

Figure 2: Accuracy–training-time trade-off across the twelve dataset–feature configurations. Each panel corresponds to one dataset–feature setting and compares PivotMipl with the MIPL baselines in terms of mean classification accuracy and total training time over five folds. Higher accuracy and shorter training time are preferred.

Variant	Modification	Full-model wins	Mean gain
A1	No occupancy regularization	10/12	$+3.25\%$
A2	Uniform temporal occupancy	9/12	$+0.72\%$
A3	Class-level KL	12/12	$+3.26\%$
A4	Class-marginal-only teacher	12/12	$+2.53\%$
A5	Single prototype per class	11/12	$+2.52\%$
A6	No temporal encoder	11/12	$+5.77\%$

Appendix

Table 7: Summary of the ablations across twelve dataset–feature configurations.

Dataset	Feature	AgopMipl	AgopMipl $+$ Temporal Encoder	PivotMipl
Breakfast	DINOv3	$46.4\%$	$\underline{53.1\%}$	$\mathbf{66.4\%}$
	VideoMAEv2	$\mathbf{79.3\%}$	$\underline{78.1\%}$	$77.4\%$
	ResNet	$31.4\%$	$\underline{45.4\%}$	$\mathbf{52.2\%}$
	SlowFast	$50.6\%$	$\underline{53.1\%}$	$\mathbf{53.9\%}$
DoTA	DINOv3	$\underline{49.2\%}$	$48.4\%$	$\mathbf{54.3\%}$
	VideoMAEv2	$69.5\%$	$\underline{69.9\%}$	$\mathbf{72.8\%}$

Appendix

Table 8: Accuracy for AgopMipl , AgopMipl augmented with the same temporal encoder architecture as PivotMipl , and PivotMipl . The best and second-best values within each row are shown in bold and underlined, respectively.

Figure 3: Sensitivity of PivotMipl to the entropic coefficient

\varepsilon

and occupancy coefficient

\lambda

on three representative dataset–feature configurations.

Explore similar work

Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning
Dec 19, 2025Wei Tang, Yin-Fang Yang, Weijia Zhang +1Multiple Instance Learning Disambiguation
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration (Southeast University), MoE, China · School of Computer and Information Sciences, The University of Newcastle, Callaghan, NSW 2308, Australia
Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence
May 5, 2026Zhiyuan Li, Rongzhen Zhao, Wenyan Yang +3Object-Centric Representations
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland
Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval
May 7, 2026Jun Li, Peifeng Lai, Xuhang Lou +5Fine-Grained Video Understanding Multimodal Retrieval
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China

Dec 19, 2025, cs.LGJ/K move · Enter open · S save

Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning

Wei Tang, Yin-Fang Yang, Weijia Zhang, Min-Ling Zhang

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China, and the Key Laboratory of Computer Network and Information Integration (Southeast University), MoE, China · School of Computer and Information Sciences, The University of Newcastle, Callaghan, NSW 2308, Australia

Multiple Instance Learning Disambiguation Suppression

Open paper AskPDF arXiv

Multi-instance partial-label learning (MIPL) is a weakly supervised framework that extends the principles of multi-instance learning (MIL) and partial-label learning (PLL) to address the challenges of inexact supervision in both instance and label spaces. However, existing MIPL approaches often suffer from poor calibration, undermining classifier reliability. In this work, we propose a plug-and-play calibratable disambiguation loss (CDL) for classification and calibration, which modulates a disambiguation objective by a top-vs-competitor prediction margin. The competitor is instantiated either as the second strongest candidate label or as the strongest non-candidate label, yielding two variants that respectively emphasize candidate-level separation and candidate-vs-non-candidate suppression. Theoretically, we analyze CDL as a margin-modulated momentum-based disambiguation loss (MDL) objective, derive a lower-bound and a pseudo-label confidence-alignment bound for calibration, and show through gradient and momentum analyses how margin shaping affects weight updates. Experimental results on benchmark and real-world MIPL datasets, together with representative PLL adaptation, confirm that our CDL significantly improves both classification accuracy and expected calibration error.

Abstract

Explore similar work

Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning

Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence

Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval