VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

Tengju Ru; Weitong Lian; Zecong Tang; Lingyi Meng; Haoran Li; Zhejun Cui; Yichen Zhu; Hangshuo Cao

cs.CVNov 3, 2026

VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

Authors: Tengju Ru, Weitong Lian, Zecong Tang, Lingyi Meng, Haoran Li, Zhejun Cui, Yichen Zhu, Hangshuo Cao,

Organizations: Zhejiang University · The University of Hong Kong · Institute of Automation, Chinese Academy of Sciences · Huawei Technologies Ltd.

Abstract

The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.

Figures & tables

Figure 1: Example samples of VidUEU-100K. We illustrate three capability paradigms: Understanding (top), Editing (middle), and Unified (bottom).

Figure 2: Overview of VidUEU-Agent. The agent performs global perception with quality gating, then conducts keyframe mining using multimodal signals and tool verification, builds metadata , and finally synthesizes instruction data for understanding, editing, and unified tasks.

Figure 3: Failure Case Examples: Illustrating four common rejected cases in the processing pipeline: Global Rejection, Quality Failure, Gen-Unsuitability, and Task Mismatch.

Type	Model	Size	MMT-Bench $\uparrow$	MME $\uparrow$	LLaVA-Bench $\uparrow$	MMDU $\uparrow$
Und. Only	Qwen2.5-VL	3B	0.577	2133.6	72.3	42.5
	Qwen2.5-VL	7B	0.607	2308.6	82.7	59.5
	LLaMA-3.2	11B	0.456	1761.7	73.6	37.0
	InternVL3	1B	0.512	1871.9	46.4	39.0
	InternVL3	2B	0.532	2157.9	62.1	45.6
	InternVL3	8B	0.546	2308.0	69.0	53.1

Table 1: Comparison of fine-tuned InternVL3-8B & Bagel, and baseline models on benchmarks.

Type	Model	G_SC $\uparrow$	G_PQ $\uparrow$	G_O $\uparrow$
Edit Only	Instruct-Pix2Pix	3.575	5.491	3.684
	MagicBrush	4.677	5.656	4.518
	AnyEdit	3.178	5.820	3.212
	Step1X-Edit	7.091	6.763	6.701
Unified	Gemini-2.0-flash	6.732	6.606	6.315
	OmniGen	5.963	5.888	5.061

Table 2: Comparison of fine-tuned Bagel and baseline models on GEdit-Bench-En.

Type	Model	Size	Science $\uparrow$	Creation $\uparrow$	Logic $\uparrow$	Game $\uparrow$	Overall $\uparrow$
Unified	OmniGen2	7B	0.511	0.682	0.551	0.511	0.609
	Ovis-U1	3B	0.402	0.557	0.364	0.357	0.422
	Bagel	14B	0.378	0.475	0.406	0.365	0.446
	Bagel-Zebra	14B	0.399	0.456	0.393	0.396	0.449
	Ours(Bagel)	14B	0.426	0.501	0.480	0.419	0.501

Table 3: Comparison of fine-tuned Bagel and baseline models on WEAVEbench.

Table 7

Figure 4: Case study examples comparing baseline models and VidUEU-100K fine-tuned models.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Prompt for shot-level summarization.

Panel of Figure 6 — Figure 6: Prompt for chapter construction, chapter-level summarization, and video-level summarization.

Panel of Figure 7 — Figure 7: Prompt for Gate A shot admission.

Panel of Figure 8 — Figure 8: Prompt for clip summary generation.

Panel of Figure 9 — Figure 9: Prompt for hierarchical tool planning.

Panel of Figure 10 — Figure 10: Prompt for tool signal verbalization.

Panel of Figure 11 — Figure 11: Prompt for frame value scoring.

Panel of Figure 12 — Figure 12: Prompt and rule for quota allocation and weighted facility-location selection.

Panel of Figure 13 — Figure 13: Prompt for pairwise transition generation.

Panel of Figure 14 — Figure 14: Prompt for task routing.

Panel of Figure 15 — Figure 15: Prompt for instruction generation.

Setting	Final Samples/min $\uparrow$	Quality $\uparrow$
Original LLM setting	12.4	4.18
Direct VLM setting	5.6	2.72

Appendix

Table 6: Comparison between the metadata-driven LLM design and the direct VLM alternative.

VLM	Samples/min/GPU $\uparrow$	GPT-4o Quality $\uparrow$
Qwen3.5-9B	12.4	4.16
Qwen3.5-27B	4.1	4.21

Appendix

Table 7: Efficiency and quality comparison of VLM candidates under the same agent workflow.

Tool	Role	Applicable scenarios
PaddleOCR	Scene-text extraction from video frames	Used to read titles, labels, menu items, tables, subtitles, document text, and screen text, especially in slides, screen recordings, documents, and software demonstrations
Whisper ASR	Speech-content recovery around a shot	Used when speech provides semantic cues near candidate frames, including narration, spoken instructions, or dialogue context
YOLOv8	Object and entity detection	Used to identify people, objects, tools, and scene entities that provide visual grounding for frame understanding
RAFT	Motion estimation between adjacent candidate frames	Used when temporal dynamics or motion strength is informative for candidate-frame analysis
OmniParser	Structured GUI understanding	Used for GUI screenshots and software-interface frames where interface layout and interactive regions are important

Appendix

Table 8: Summary of auxiliary perception tools used for frame-level and shot-level information extraction.

Tool evidence	Verbalization rule	Value-scoring use	Reliability boundary
OCR	Use recognized textual content as primary evidence for readable text; describe the visible meaning, function, or role of the text rather than enumerating all detected tokens.	Reward readable and specific text evidence that reveals task-relevant titles, steps, labels, states, commands, or interface cues; assign higher value when text evidence substantially improves understanding beyond generic visual appearance.	Keep noisy, incomplete, or ambiguous text evidence conservative; do not reward repeated boilerplate, recognition artifacts, or isolated fragments unless they reveal important state or intent.
ASR	Use nearby spoken content as temporal contextual evidence; treat speech as information associated with the frame interval, rather than as on-screen text or direct visual evidence.	Reward specific and temporally relevant speech that contributes concrete instructions, explanations, actions, or contextual meaning; assign higher value when spoken content matches the apparent step, state, or action around the frame.	Stay conservative when the alignment between speech and frame is uncertain; do not over-reward vague transcripts, weakly aligned speech, or information already evident from the visual content alone.
YOLOv8	Use detected object categories as primary evidence for visible entities; use confidence or location only as supporting information, and describe only entities that are visually supported.	Reward clear and relevant visible entities that explain the subject, objects in use, workspace context, or interaction target; assign higher value when detected entities are specific, diverse, and informative.	Do not treat detection confidence, quantity, or coverage alone as evidence of value, and do not infer unsupported relationships, intentions, or actions.
RAFT	Use optical-flow evidence as primary support for motion or temporal change; interpret motion strength as an indicator of change intensity rather than semantic importance or action type.	Reward motion evidence when it marks a meaningful transition, interaction moment, or state change; assign higher value when motion helps distinguish the frame from static or redundant neighboring frames.	Describe staticness or motion-heavy appearance only when supported by motion evidence; do not reward motion magnitude alone without semantic relevance.
OmniParser	Use parsed interface structure and element descriptions as primary evidence for GUI layout, component meaning, and visible application state; prioritize actionable or state-revealing elements over low-level visual labels.	Reward actionable interface structure such as buttons, menus, form fields, toggles, tabs, or clear application states; assign higher value when parsed elements materially improve understanding of what the interface shows or allows.	Treat element counts, positions, and bounding regions as supporting evidence rather than meaning by themselves; do not reward dense but uninformative interface clutter.

Appendix

Table 9: Rules used to verbalize tool signals and score each frame.

Figure 17: Fine-grained perception example. The sample asks the model to identify a foreground object from a selected visual frame.

Figure 18: Visual reasoning example. The sample requires inferring the ongoing activity from visual evidence in the image.

Panel of Figure 19 — Figure 19: Multi-image logic example. The sample compares multiple related frames and verifies the shared scene structure and temporal changes.

Panel of Figure 20 — Figure 20: Image editing example. The sample edits the source image according to a localized visual instruction while preserving unrelated content.

Panel of Figure 21 — Figure 21: Grounded edit-to-select generation example. The sample plans a grounded visual edit, generates the target state, and selects the matching candidate.

Panel of Figure 22 — Figure 22: Interleaved understand-recall-generate-select example. The sample alternates between visual understanding, grounded recall, generation, and candidate selection.

Panel of Figure 23 — Figure 23: Stepwise sequence generate-and-select example. The sample follows a continuous action sequence, generates the next step, and selects the matching candidate.

Panel of Figure 24 — Figure 24: Middle-step gap generate-and-fill example. The sample infers and generates the missing intermediate frame between two temporally ordered endpoints.

Panel of Figure 25 — Figure 25: Instructional sequential generate-and-select example. The sample follows a stable natural blooming process, generates future visual steps, and selects the matching sequence.

Panel of Figure 26 — Figure 26: Prompt for shot admission.

Panel of Figure 27 — Figure 27: Prompt for quality assessment.

Panel of Figure 28 — Figure 28: Prompt for signal filtering.

Panel of Figure 29 — Figure 29: Prompt for metadata admission.

Domain	Rejection Rate (%)
Technology and Tutorial	37.3
Sports and Action	39.8
Daily Life and Human Activities	24.8
Nature and Scenery	47.3
Gaming and ACG	38.2
Overall	37.4

Appendix

Table 10: Gate C rejection rates by domain.

Figure 30: Distribution of the final dataset.

Quality Dimension	Annotator A	Annotator B	Mean	Cohen’s $\kappa$
Gate C Metadata Validity	4.12	4.08	4.10	0.69
Keyframe–Tool–Signal Alignment	4.30	4.20	4.25	0.76
Transition Correctness	4.06	4.01	4.04	0.77
Editing-Sample Validity	3.96	4.02	3.99	0.71

Appendix

Table 11: Human audit of synthesized-data quality.

Capability	Model / metric	Baseline std	Ours std
Understanding	InternVL3, LLaVA-Bench	0.317	0.292
Understanding	InternVL3, MMDU	0.150	0.172
Understanding	Bagel, LLaVA-Bench	0.306	0.328
Understanding	Bagel, MMDU	0.133	0.151
Editing	Bagel, GEdit-Bench-En G_SC	0.073	0.157
Editing	Bagel, GEdit-Bench-En G_PQ	0.076	0.146

Appendix

Table 12: Automatic-judge run-to-run standard deviation with fixed evaluated outputs. Mean scores are reported in the main experimental tables.

Setting	Understanding	Editing	Unified
Iteration 1	50.0	25.0	25.0
Iteration 2	35.0	33.0	32.0
Iteration 3	39.0	31.0	30.0
Iteration 4	40.0	30.0	30.0

Appendix

Table 14: Realized capability distributions (%) across autonomous iterations.

Setting	Technology	Sports	Daily Life	Nature	Gaming
Iteration 1	22.1	17.8	28.2	18.1	13.8
Iteration 2	18.1	21.9	24.8	16.1	19.1
Iteration 3	23.9	20.2	21.9	17.9	16.1
Iteration 4	25.2	21.1	19.8	17.8	16.1

Appendix

Table 15: Realized domain distributions (%) across autonomous iterations.

Setting	FP	VR	MIL	Edit	GES	RGS	SGS	GGF	ISG
Iteration 1	28.3	9.6	12.1	25.0	4.6	5.3	4.8	5.1	5.2
Iteration 2	22.4	6.8	5.8	33.0	9.0	6.4	6.1	4.8	5.7
Iteration 3	20.8	8.3	9.9	31.0	7.0	5.6	5.9	5.3	6.2
Iteration 4	19.8	8.3	11.9	30.0	5.6	6.3	5.8	6.2	6.1

Appendix

Table 16: Realized task-type distributions (%) across autonomous iterations.

Setting	MMBench $\uparrow$	MMStar $\uparrow$	VSI-Bench $\uparrow$
BAGEL Base	84.2	67.3	31.4
Iteration 1	84.9	67.4	32.8
Iteration 2	84.7	67.3	32.9
Iteration 3	85.0	67.5	33.6
Iteration 4	85.1	67.6	34.4
Final	85.3	67.8	36.8

Appendix

Table 17: Evaluation on benchmarks excluded from autonomous feedback.

Multimodal Dataset

See all 60 papers

SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding
Apr 14, 2026Tanzila Rahman, Renjie Liao, Leonid SigalSynthetic-to-Real Domain Adaptation Synthetic Data Augmentation
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
UniVideo: Unified Understanding, Generation, and Editing for Videos
Oct 9, 2025Cong Wei, Quande Liu, Zixuan Ye +5Video Generation Unified Multimodal Models
University of Waterloo · Kling Team, Kuaishou Technology
VideoAgent: All-in-One Framework for Video Understanding and Editing
Jun 22, 2026Hengji Zhou, Lingxuan Huang, Jian Wang +4Video Editing Benchmarks Agentic Video Editing
1Harbin Institute of Technology, Shenzhen · 2South China University of Technology · 3The University of Hong Kong +1

Apr 14, 2026J/K move · Enter open · S save

SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding

Tanzila Rahman, Renjie Liao, Leonid Sigal

University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair

Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation Video Object Segmentation Multimodal QA

Open paper AskPDF arXiv

Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as object counting, question answering, and segmentation. However, collecting and annotating multimodal video data in real-world is costly, slow, and inherently limited in diversity and coverage. To address this challenge, we propose~SynMulti dataset, along with a unified synthetic data generation pipeline capable of automatically producing unlimited multimodal video data with rich and diverse supervision. Our framework supports multiple task formats within a single pipeline, enabling scalable and consistent data creation across tasks. To further enhance reasoning ability, we introduce a VQA-based fine-tuning strategy that trains models to answer structured questions about visual content rather than relying solely on captions or simple instructions. This formulation encourages deeper visual grounding and reasoning. We evaluate our approach in three challenging tasks: video object counting, video-based visual question answering, and video object segmentation. Experimental results demonstrate that models trained predominantly on synthetic data generalize effectively to real-world datasets, often outperforming traditionally trained counterparts. Our findings highlight the potential of unified synthetic data pipelines as a scalable alternative to expensive real-world annotation for multimodal video understanding.

Abstract

Multimodal Dataset

Explore similar work

SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding

UniVideo: Unified Understanding, Generation, and Editing for Videos

VideoAgent: All-in-One Framework for Video Understanding and Editing