From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creation

Zhefan Rao; Bin Zou; Xuanhua He; Chong Hou Choi; Yanheng Li; Rui Liu; Haoxuan Che; Qifeng Chen

cs.CVAug 13, 2026

From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creation

Authors: Zhefan Rao, Bin Zou, Xuanhua He, Chong Hou Choi, Yanheng Li, Rui Liu, Haoxuan Che, Qifeng Chen

Organizations: The Hong Kong University of Science and Technology · Celia Research HK · City University of Hong Kong

Abstract

Unified image and video creation requires a model to follow diverse instructions while preserving identity, geometry, and temporal structure from visual context. However, semantic-only conditioning and creation-only training do not explicitly supervise the local structure needed for precise, temporally consistent editing. We therefore formulate depth and surface-normal prediction as image-form denoising targets, using these dense tasks as structured visual supervision within the same creation interface. Our framework decouples semantic interpretation from spatially aligned visual injection while sharing one multimodal diffusion transformer (MMDiT) backbone across all tasks. Mutual Context Attention (MCA), a paired-video data-construction procedure, and a progressive training curriculum then connect the learned structural cues to temporally localized editing and reference-conditioned creation. A single checkpoint obtains the highest overall score in the reported comparison of unified systems (4.15); adding dense supervision improves OpenVE Overall from 3.98 to 4.06 and Local Add from 3.92 to 4.18. These results support a deliberately bounded conclusion: perception-oriented dense supervision transfers useful structural knowledge to downstream creation, especially editing locality and preservation; we do not claim superiority as a standalone dense predictor.

Figures & tables

Fig. 1: Unified visual creation framework. The semantic pathway encodes instruction intent and cross-modal relations, while the visual pathway preserves spatial evidence from sources, references, and controls. Noisy target tokens, clean condition tokens, and their role identifiers are processed by one shared MMDiT. Creation outputs and depth/normal structured-supervision targets use the same target slots and denoising objective.

Task family	Primary input	Optional context	Target behavior
Text-to-image/video generation	Text prompt	None	Synthesize an image or video from scratch
In-context generation	Text prompt	Subject, frame, sketch, layout, pose, depth, normal	Generate content consistent with visual context
Instruction-based editing	Source image/video + instruction	Mask or structured control	Apply requested edits while preserving untouched regions
In-context editing	Source image/video	Reference image/video, edited keyframes	Transfer style, identity, structure, or edits across time
Dense prediction	Source image/video + instruction	Target representation type	Predict depth or normal maps as image-form outputs

TABLE I: Unified task formulation. Diverse creation and dense-prediction tasks vary the contents of a shared condition package rather than the model interface.

Fig. 2: Mutual Context Attention for aligned pair synthesis. Source and target branches are denoised in one shared DiT; scheduled cross-branch key/value interaction aligns scene layout, identity, and motion while branch-specific queries retain the requested edit.

Fig. 3: Automatic construction and filtering of video editing pairs. Prompt expansion is followed by MCA-coupled source/target synthesis, instruction generation, response filtering, and multi-round VQA verification.

Fig. 4: MCA-generated editing pairs. Examples include edits whose onset occurs after the first frame or only within an intermediate interval, while unchanged content remains aligned.

Method	Unified Overall $\uparrow$	Generation			Editing
		Overall $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	Overall $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	IFS $\uparrow$
UniVideo [ 34 ]	4.03	4.08	4.12	4.04	3.98	4.01	3.95	3.90
VINO [ 5 ]	4.12	4.16	4.19	4.13	4.08	4.11	4.06	4.03
Ours	4.15	4.15	4.17	4.13	4.16	4.18	4.15	4.14

TABLE II: Unified image/video creation. Qwen3-VL-32B scores use a 0–5 scale for one unified checkpoint per method; Overall aggregates T2I, T2V, S2V, TI2I, TV2V, II2I, and IV2V.

Fig. 5: Unified creation examples. One checkpoint performs subject-to-video generation, text-instruction video editing, and image-reference video editing while retaining reference identity, unedited scene content, and temporal structure.

Stage	Objectives	Capability introduced
I	Generation; VLM reconstruction; consistency preservation (7:2:1)	Condition alignment and prior retention
II	Instruction editing; depth/normals; MCA pairs; generation replay	Local editing and explicit structural supervision
III	Subject/reference generation; reference editing; keyframe propagation; multi-reference composition	Reference binding and compositional creation

TABLE III: Progressive training curriculum. The model interface and loss remain fixed; only task sampling changes.

Panel of Figure 9 — Fig. 6: MCA data value and adaptation behavior. (a) The editing instantiation reaches strong benchmark performance with $O(100)$ K video editing samples while also using larger image-editing and generation sources. (b) Temporally localized editing demonstrates a behavior underrepresented by first-frame propagation data.

Variant	Overall $\uparrow$	Local Add	Local Remove	Local Change	Background Change	Global Style
w/o Dense Prediction	3.98	3.92	3.84	3.89	3.76	3.89
w/ Dense Prediction	4.06	4.18	4.02	4.11	3.95	4.06

TABLE IV: Downstream transfer from dense supervision on OpenVE-Bench. Scores evaluate instruction video editing, not standalone depth/normal accuracy.

Method	OpenVE-Bench						InsEdit-Bench				Latency $\downarrow$
Method	Overall $\uparrow$	Local Add	Local Remove	Local Change	Subtitle Edit	Creative Edit	Overall $\uparrow$	IC $\uparrow$	TVQ $\uparrow$	URP $\uparrow$	Latency $\downarrow$
VACE-14B [ 13 ]	3.01	1.76	3.99	2.47	4.41	2.17	3.08	2.39	2.64	4.21	9.08
OmniVideo [ 28 ]	3.66	2.80	4.52	3.75	4.95	1.13	4.13	4.04	4.04	4.32	20.58
InsViE [ 36 ]	3.25	2.25	3.56	2.82	4.77	3.36	2.76	2.19	2.29	3.80	1.06
Lucy-Edit [ 29 ]	3.77	3.92	3.95	3.93	4.23	4.19	3.64	3.24	3.38	4.30	0.60
ICVE [ 19 ]	3.76	3.77	4.50	3.87	4.68	3.54	3.54	2.91	2.94	4.79	20.60

TABLE V: Instruction video editing. OpenVE-Bench reports representative categories; Table VI gives the complete category breakdown. Latency is minutes per 81-frame 480p output.

Method	Overall $\uparrow$	Local Add	Local Remove	Local Change	Background Change	Global Style	Subtitle Edit	Creative Edit	Camera Edit
VACE-14B [ 13 ]	3.01	1.76	3.99	2.47	2.81	3.46	4.41	2.17	3.09
OmniVideo [ 28 ]	3.66	2.80	4.52	3.75	4.11	3.41	4.95	1.13	3.62
InsViE [ 36 ]	3.25	2.25	3.56	2.82	2.68	3.63	4.77	3.36	3.61
Lucy-Edit [ 29 ]	3.77	3.92	3.95	3.93	3.25	3.64	4.23	4.19	3.54
ICVE [ 19 ]	3.76	3.77	4.50	3.87	3.51	3.87	4.68	3.54	2.84
Ditto [ 1 ]	3.44	2.48	3.53	2.89	3.52	4.48	3.69	4.14	3.33

TABLE VI: Complete category-level OpenVE-Bench results.

Method	G_SC $\uparrow$	G_PQ $\uparrow$	G_O $\uparrow$
Gemini2.5	7.48	8.30	7.17
GPT4o	8.06	7.80	7.48
Seedream4	8.33	8.00	7.72
UniWorld-V1 [ 20 ]	5.04	7.56	4.98
OmniGen2 [ 35 ]	6.79	6.68	6.18
Flux-Kontext-Dev [ 2 ]	7.23	7.28	6.53

TABLE VII: Image editing on GEdit. Bold marks the best value among reported open-source methods.

Fig. 7: Instruction video editing results. Examples span local object, attribute, background, motion, and style changes while preserving content outside the instruction.

Fig. 8: Qualitative comparison. InsEdit is compared with unified baselines on instruction compliance, temporal quality, and preservation of unedited regions.

Construction	Alignment	Prompt	Temporal	Artifact (%)
Independent generation	0.67	0.79	0.72	18.6
First-frame propagation	0.78	0.71	0.76	15.4
MCA-generated	0.84	0.82	0.81	9.8

TABLE VIII: Controlled audit of paired-video construction. Generator, prompts, seeds, resolution, denoising budget, and filtering thresholds are fixed. Artifact is a rate (

\downarrow

); other metrics are scores (

\uparrow

Category	Raw	Kept	Keep rate
Object add/remove	38.4K	30.6K	79.7%
Attribute/action	31.2K	24.5K	78.5%
Background/weather	26.8K	21.4K	79.9%
Style/relighting	18.6K	15.1K	81.2%
Temporal edits	33.7K	26.9K	79.8%

TABLE IX: Category-level MCA data audit. Raw/retained counts in the audited construction batch, keep rates.

Video editing data	InsEdit-Bench Overall $\uparrow$
4K non-MCA	3.96
2K non-MCA + 2K MCA	4.08

TABLE X: Matched-budget downstream MCA study. Both variants use 4K video editing pairs; only pair composition changes.

Variant	Overall $\uparrow$	IC $\uparrow$	TVQ $\uparrow$	URP $\uparrow$
Edit-only Baseline	4.41	4.24	4.30	4.70
+ Gen Data w/ SigLIP-1	4.46	4.32	4.33	4.72
+ Gen Data w/ SigLIP-3	4.30	4.09	4.17	4.63
+ Gen Data w/ SigLIP-3 Avg	4.32	4.11	4.21	4.65
+ Gen Data w/o VLM Vision	4.10	3.83	3.80	4.52

TABLE XI: Stage-II modeling variants on InsEdit-Bench. Generation replay, VLM visual input, and SigLIP temporal input are varied.

Group	Variant	Overall $\uparrow$	IC $\uparrow$	TVQ $\uparrow$	URP $\uparrow$
Baseline	Edit-only Baseline	4.41	4.24	4.30	4.70
Initialization	w/o Stage 1 Init	4.15	3.94	3.95	4.57
	w/o Stage 1 Consistency Init	4.26	3.99	4.04	4.74
	w/o Stage 1 VLM-Recon Init	4.29	4.09	4.15	4.63
Data Ratio	Image:Video = 1:1	4.34	4.15	4.16	4.71
Data Ratio	Image:Video = 1:4	4.27	4.07	4.12	4.62

TABLE XII: Edit-only Stage-II recipe on InsEdit-Bench. Groups vary Stage-I initialization, image/video ratio, and prompt mixture.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 179 papers

Unified Video Dense Prediction from Disjoint Data
Jul 23, 2026Yihong Sun, Seoung Wug Oh, Jiahui Huang +2Multi-Task Learning Dense Prediction
Adobe Research · Cornell University
SenseNova-U1.5: Towards Native Unified Visual Intelligence
Sep 10, 2026Haiwen Diao, Jiahao Wang, Chenjing Ding +62Unified Multimodal Models T2I Generation
TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation
Jun 6, 2026Qi Liu, Gang Yue, Mingyu Yin +7Video Diffusion Models Video Editing
Zhejiang University · Bilibili Inc.

Jul 23, 2026J/K move · Enter open · S save

Unified Video Dense Prediction from Disjoint Data

Yihong Sun, Seoung Wug Oh, Jiahui Huang, Bharath Hariharan+1

Adobe Research · Cornell University

Multi-Task Learning Dense Prediction 3D Scene Understanding

Open paper AskPDF arXiv

Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials-all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling. Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency. Code and video results are available at https://unid-video.github.io/.

Abstract

Video Editing

Explore similar work

Unified Video Dense Prediction from Disjoint Data

SenseNova-U1.5: Towards Native Unified Visual Intelligence

TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation