Live Interactive Training for Video Segmentation

Xinyu Yang; Haozheng Yu; Yihong Sun; Bharath Hariharan; Jennifer J. Sun

cs.CVMar 27, 2026

Live Interactive Training for Video Segmentation

Authors: Xinyu Yang, Haozheng Yu, Yihong Sun, Bharath Hariharan, Jennifer J. Sun

Organizations: Cornell University

Abstract

Interactive video segmentation often requires many user interventions for robust performance in challenging scenarios (e.g., occlusions, object separations, camouflage, etc.). Yet, even state-of-the-art models like SAM2 use corrections only for immediate fixes without learning from this feedback, leading to inefficient, repetitive user effort. To address this, we introduce Live Interactive Training (LIT), a novel framework for prompt-based visual systems where models also learn online from human corrections at inference time. Our primary instantiation, LIT-LoRA, implements this by continually updating a lightweight LoRA module on-the-fly. When a user provides a correction, this module is rapidly trained on that feedback, allowing the vision system to improve performance on subsequent frames of the same video. Leveraging the core principles of LIT, our LIT-LoRA implementation achieves an average 18-34% reduction in total corrections on challenging video segmentation benchmarks, with a negligible training overhead of ~0.5s per correction. We further demonstrate its generality by successfully adapting it to other segmentation models and extending it to CLIP-based fine-grained image classification. Our work highlights the promise of live adaptation to transform interactive tools and significantly reduce redundant human effort in complex visual tasks. Project: https://youngxinyu1802.github.io/projects/LIT/.

Figures & tables

Figure 1: Comparison between the current non-learning system and our LIT-LoRA approach. The current system (top) does not learn from user feedback, leading to the same errors to reappear and requiring repeated corrections (e.g., 14 prompts to add the missing cards), which leads to substantial annotation time (e.g., 5.62 mins). In contrast, our LIT-LoRA method continuously adapts to user correction input and generalizes to similar future errors, reducing the number of required corrections (e.g., down to 5) and user annotation time (e.g., down to 3.18 mins).

Figure 2: Left: Overview of the LIT-LoRA framework on VOS . As the video progresses, segmentation errors may arise. When the user provides a correction (which can be time-consuming), the correction is used to train a LoRA module on-the-fly. The LoRA module is then consulted for later errors: if its prediction meets the validation criterion, it is accepted to correct the error; otherwise, the adapter is further refined using the latest correction. Right: LIT LoRA module illustration .

		(a) Average user corrections per video					(b) Average annotation time per video (min)
$\tau_{\mathrm{IoU}}$	Method	VOST	LVOSv2	MOSEv2	SA-V Val	SA-V Test	VOST	LVOSv2	MOSEv2	SA-V Val	SA-V Test
0.5	Original	27.43	33.59	31.78	20.66	20.90	18.42	14.83	22.49	12.90	13.09
	LIT	18.24	25.75	25.99	16.91	16.23	12.91	11.86	18.00	10.01	10.73
	Reduced	$\downarrow$ 33.51%	$\downarrow$ 23.35%	$\downarrow$ 18.22%	$\downarrow$ 18.16%	$\downarrow$ 22.35%	$\downarrow$ 29.94%	$\downarrow$ 20.03%	$\downarrow$ 19.98%	$\downarrow$ 22.44%	$\downarrow$ 18.01%
0.75	Original	40.40	56.32	41.47	27.48	29.83	39.47	46.15	41.96	26.18	28.16
	LIT	30.85	45.00	34.43	22.55	24.50	31.01	36.65	33.15	21.13	23.00

Table 1: Comparison of user corrections and annotation time across datasets under different IoU thresholds. LIT consistently reduces both user corrections and annotation time at

\tau_{\mathrm{IoU}}

= 0.5 and 0.75.

Figure 3: User interaction patterns and the impact across datasets. (a) The number of user corrections follows a clear long-tailed distribution: a small fraction of challenging videos accounts for the majority of interactions. (b) The challenging cases ( $\geq$ 10 corrections) require substantially more user inputs than the dataset average. (c) User feedback consistently improves segmentation performance, especially for the challenging subset. (d) Corrections are not uniformly distributed in time; most prompts occur in the early to late portions of each sequence, indicating the recurrence of errors.

Figure 4: Performance under different numbers of user corrections.

(a) VOS			(b) Fine-grained image classification
Dataset	VOST		CUB-200-2011	Stanford Cars	SUN397
Model	DAM4SAM	SAMURAI	CLIP
Original	34.60	26.96	13.04	13.38	13.92
LIT	22.46	21.23	8.53	7.57	8.95
Reduction	$\downarrow$ 35.09%	$\downarrow$ 21.25%	$\downarrow$ 34.55%	$\downarrow$ 43.40%	$\downarrow$ 35.70%

Table 2: User correction reduction across models and tasks.

Panel of Figure 6 — Figure 6: Ablation studies on VOST dataset

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Predicted IoU vs. Ground-Truth IoU (Accuracy, FPR, FNR). Accuracy, false positive rate (FPR), and false negative rate (FNR) across different predicted IoU thresholds, measuring how well the predicted IoU aligns with the ground-truth IoU for each dataset. Vertical dashed lines mark the threshold that achieves the highest alignment accuracy.

	# Clicks $\downarrow$	# Params
Original	27.43	–
$\text{LIT}_{\text{mask decoder}}$	18.24	$\sim$ 35k
$\text{LIT}_{\text{memory adapter}}$	18.84	$\sim$ 66k ( $\sim$ 2 $\times$ )

Appendix

Table 3: User-correction reduction with different adapter insertion locations.

	# Clicks $\downarrow$	Reduction	Time (s)
Original	27.43	–	–
1/1 (ours)	18.24	33.51%	0.58
3/3	18.74	31.69%	0.61
5/5	19.03	30.62%	0.62
3/1	17.73	35.39%	1.61
5/1	17.64	35.71%	2.32

Appendix

Table 4: Train on more corrections.

Figure 8: Using predicted IoU for correction triggering and validation.

Figure 9: Qualitative result of successful cases of LIT-LoRA on image classification.

Figure 10: Additional qualitative result of successful cases of LIT-LoRA on VOS.

Human-in-the-Loop AI

Human-in-the-loop AI combines human feedback, oversight, and decision-making with machine learning and autonomous systems.

See all 288 papers

From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks
Apr 8, 2026Carlos Schmidt, Simon ReißIn-Context Learning Promptable Image Segmentation
Fraunhofer IOSB Karlsruhe, Germany · Karlsruhe Institute of Technology Karlsruhe, Germany
Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts
Apr 18, 2026Dawar Jyoti DekaVideo Object Segmentation
Indian Institute of Technology Bombay · Mumbai, Maharashtra, India
X2SAM: Any Segmentation in Images and Videos
Apr 27, 2026Hao Wang, Limeng Qiao, Chi Zhang +4Image Segmentation Video Object Segmentation
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc

Apr 8, 2026J/K move · Enter open · S save

From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks

Carlos Schmidt, Simon Reiß

Fraunhofer IOSB Karlsruhe, Germany · Karlsruhe Institute of Technology Karlsruhe, Germany

In-Context Learning Promptable Image Segmentation Image Editing

Open paper AskPDF arXiv

Visual in-context learning models are designed to adapt to new tasks by leveraging a set of example input-output pairs, enabling rapid generalization without task-specific fine-tuning. However, these models operate in a fundamentally static paradigm: while they can adapt to new tasks, they lack any mechanism to incorporate user-provided guidance signals such as scribbles, clicks, or bounding boxes to steer or refine the prediction process. This limitation is particularly restrictive in real-world applications, where users want to actively guide model predictions, e.g., by highlighting the target object for segmentation, indicating a region which should be visually altered, or isolating a specific person in a complex scene to run targeted pose estimation. In this work, we propose a simple method to transform static visual in-context learners, particularly the DeLVM approach, into highly controllable, user-driven systems, i.e., Interactive DeLVM, enabling seamless interaction through natural visual cues such as scribbles, clicks, or drawing boxes. Specifically, by encoding interactions directly into the example input-output pairs, we keep the philosophy of visual in-context learning intact: enabling users to prompt models with unseen interactions without fine-tuning and empowering them to dynamically steer model predictions with personalized interactions. Our experiments demonstrate that SOTA visual in-context learning models fail to effectively leverage interaction cues, often ignoring user guidance entirely. In contrast, our method excels in controllable, user-guided scenarios, achieving improvements of

+7.95%

IoU for interactive segmentation,

+2.46

PSNR for directed super-resolution, and

-3.14%

LPIPS for interactive object removal. With this, our work bridges the gap between rigid static task adaptation and fluid interactivity for user-centric visual in-context learning.

Abstract

Human-in-the-Loop AI

Explore similar work

From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks

Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts

X2SAM: Any Segmentation in Images and Videos