OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

Xiaoyan Cong; Tianyu Wang; Hyeonho Jeong; Zhifei Zhang; Yuqian Zhou; Yijun Li; Shaoteng Liu; Zhe Lin

cs.CVOct 15, 2026

OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

Authors: Xiaoyan Cong, Tianyu Wang, Hyeonho Jeong, Zhifei Zhang, Yuqian Zhou, Yijun Li, Shaoteng Liu, Zhe Lin,

Organizations: Brown University · Adobe Research

Abstract

Creative intent in video editing can be expressed through text instructions (e.g., "remove the people"), image references (e.g., subject appearance), video references (e.g., camera movement), or a combination of the above. We introduce OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine variable numbers of text, image, and video references to control diverse attributes of a source video. We first devise a unified input design with a reference-bank interface and introduce $R^2$ -RoPE to support generalization to more references. Then our model, OmniXRef-Model, uses a single-stream diffusion transformer to jointly process the source video and multimodal context from an MLLM in an interleaved sequence. To address the scarcity of training data for video-reference editing, we construct OmniXRef-Data using synthesis pipelines for different editing tasks and combinations of references. We use curriculum learning to progressively increase task complexity and resolution. To support systematic evaluation of video as a control modality, we introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer across Camera, Motion, Content, Style, and Visual Effects (VFX). Comprehensive evaluations show that our OmniXRef-Model outperforms task-specific baselines and commercial models.

Figures & tables

Figure 1: OmniXRef supports multimodal (text, image and video) multi-reference video editing, transferring the desired attribute from each reference to the source video.

Figure 2: Overview of OmniXRef-Model. Reference banks provide a unified interface for text, image, and video references, enabling multimodal multi-reference video editing. An MLLM extracts semantic context from the interleaved inputs. A single-stream DiT jointly processes the contextual text tokens, visual tokens, and noise through full self-attention, allowing multiple reference banks to condition the edited output simultaneously.

Task	Weight		#Samples
	Stage 1	Stage 2/3
$\mathrm{T}^{\prime}\mathrm{V2V}$ — Text-only
General	20%	20%	7M
$(\mathrm{I}/\mathrm{V})^{\prime}\mathrm{V2V}$ — Single visual reference
Camera	16%	8%	1.5M
Motion	24%	12%	100K

Table 1: Statistics of OmniXRef-Data with the sampling weight of each task at each curriculum stage.

Open-source
Model	Content				Style				VFX
	RA	VQ	SP	Score	RA	VQ	SP	Score	RA	VQ	SP	Score
RefVFX	0.94	3.35	4.55	1.98	1.12	3.55	4.90	2.24	1.69	3.40	4.39	2.49
Ours
OmniXRef (MiniMax-H3)	4.22	3.61	4.08	3.70	3.75	3.74	4.40	3.55	3.56	3.63	4.85	3.83
OmniXRef	4.69	3.94	4.67	4.49	3.71	3.60	4.65	3.75	3.56	3.25	4.52	3.55

Table 2: Per-category VLM evaluation of video-reference attribute transfer on OmniXRef-Bench.

Figure 3: Qualitative comparison of OmniXRef-Model and Seedance 2.5 on $\mathrm{V}^{\prime}\mathrm{V2V}$ across visual effects, style, content, and camera editing.

Model	RotErr $\downarrow$	TransErr $\downarrow$	CamMC $\downarrow$
CamCloneMaster	2.086	9.329	10.332
OmniXRef	0.874	3.961	4.437
Commercial
Seedance 2.0	4.443	22.455	24.008
Seedance 2.5	4.373	22.445	24.001

Table 3: Camera trajectory errors on OmniXRef-Bench, summed over frames; RotErr is in radians.

Variant	Content				Style				VFX
	RA	VQ	SP	Score	RA	VQ	SP	Score	RA	VQ	SP	Score
w/o Mix-Patchify	3.75	3.80	3.30	2.97	3.40	3.22	3.58	2.71	4.12	3.61	3.64	3.30
Reference-bank dropout	3.06	3.73	3.30	2.48	3.25	3.50	4.45	3.29	3.72	3.47	4.14	3.42
w/o R 2 -RoPE	3.31	3.47	3.89	3.06	2.83	3.62	4.44	3.14	3.41	3.48	4.66	3.64
w/o curriculum learning	3.69	3.56	3.70	3.07	3.08	3.44	4.32	3.15	3.69	3.37	4.22	3.51
OmniXRef (full)	4.69	3.87	4.70	4.48	3.71	3.60	4.65	3.74	3.56	3.25	4.52	3.55

Table 4: Per-category ablations of OmniXRef-Model on OmniXRef-Bench.

Figure 4: Responsiveness to different visual references with a fixed source video and edit instruction.

Figure 5: Reference-order robustness when swapping reference bank slots in R 2 -RoPE.

Figure 6: Compositional generalization to three video references.

Figure 7: Analysis of the inference latency and computational cost.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Task definition
Camera	“This is a camera edit — change only the camera trajectory, keep everything else exactly as in the source.”
Motion	“This is a motion edit — change only the motion, keep everything else exactly as in the source.”
Content	“This is a content edit — change only the indicated subject or background, keep everything else exactly as in the source.”
VFX	“This is a visual effect edit — add only the indicated phenomenon, keep everything already in the source unchanged.”
Style	“This is a style edit — change only the lighting or stylization, keep everything else exactly as in the source.”

Appendix

Table 5: Task definition

T_{i}^{\mathrm{task}}

of each category

Criterion	Judged items	Label $\rightarrow$ score	Caps
RA (Camera)	Direction, extent, timing, framing	match / partial / mismatch $\rightarrow 5/2.5/0$	None (preservation and leakage are judged by SP)
RA (Content, Style, VFX)	Category-specific properties	match / partial / mismatch $\rightarrow 5/2.5/0$	Wrong localization $\rightarrow 0$ ; spillover $\rightarrow\leq 2.5$ ; Style broken structure $\rightarrow\leq 1.25$ ; Content clear extra import $\rightarrow\leq 2.5$
RA (Motion)	Pose match at six moments	match / partial / mismatch $\rightarrow 5/2.5/0$	Reference share of appearance votes $\geq 0.5\rightarrow 0$ , $\geq 0.2\rightarrow\leq 2.5$ ; subject absent in more than half of the moments $\rightarrow 0$
VQ	Worst defect in appeal, plausibility, coherence	none / subtle / noticeable / obvious $\rightarrow 5/3.75/2/0$	None
SP	Source attributes	preserved / side effect / altered / absent / from reference $\rightarrow 5/5/2.5/1.25/0$	Different video $\rightarrow 0$ ; partly changed $\rightarrow\leq 4$ ; leakage: suspected $\leq 3.125$ , minor $\leq 1.25$ , major $0$
Score	RA, VQ, SP	Eq. ( 6 )	Major leakage $\rightarrow 0$ ; minor leakage or $\mathrm{SP}\leq 1.25\rightarrow\leq 1.25$ ; suspected leakage $\rightarrow\leq 3.125$

Appendix

Table 6: Conversion of judge labels into scores. The label scores of the judged items are averaged into a base score, and the listed caps are then applied.

Commercial
Model	IF $\uparrow$	RQ $\uparrow$	EE $\uparrow$	GeoAgg $\uparrow$
Kling o3 Omni	3.033	3.588	3.043	3.057
Kling o1	3.040	3.534	2.976	2.985
Runway Gen-4.5	2.817	3.319	2.923	2.912
Seedance 2.0	2.811	3.421	3.088	2.766
Grok Imagine	2.606	3.346	3.376	2.723

Appendix

Table 7: VEFX-Bench leaderboard for

\mathrm{T}^{\prime}\mathrm{V2V}

Model	Editing accuracy $\uparrow$	Video naturalness $\uparrow$	Video quality $\uparrow$	Total $\uparrow$
Add
InsViE	2.60	3.10	3.46	3.05
Lucy-Edit	6.47	5.70	6.77	6.31
Ditto	6.70	7.57	8.41	7.56
ReCo	8.54	7.55	8.61	8.23
ReCo_Ref	7.96	7.39	8.69	8.02

Appendix

Table 8: ReCo-Bench leaderboard for

\mathrm{T}^{\prime}\mathrm{V2V}

across four tasks: Add, Remove, Replace, Style.

Figure 8: Coupled transfer of facial expression and body pose from a Motion reference.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 178 papers

ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships
Jul 16, 2026Xinyu Liu, Shihao Li, Weihong Lin +10Multimodal Grounding Video Diffusion Models
HKUST · Kling Team · NJU +3
MiVE: Multiscale Vision-language features for reference-guided video Editing
May 14, 2026Tong Wang, Meng Zou, Chengjing Wu +4Video Diffusion Models Video Editing
MT Lab, Meitu Inc., Beijing 100083, China · Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Computer Science and Technology, BNRist, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance
Mar 2, 2026Yiqi Lin, Guoqiang Liang, Ziyun Zeng +3Video Editing Instruction-Based Video Editing
Show Lab, National University of Singapore

Jul 16, 2026J/K move · Enter open · S save

ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships

Xinyu Liu, Shihao Li, Weihong Lin, Xinlong Chen+9

HKUST · Kling Team · NJU+3

Multimodal Grounding Video Diffusion Models Video Editing Instruction-Based Video Editing Multimodal Instruction Tuning

Open paper AskPDF arXiv

Recent diffusion-based video generation models have made significant progress in multi-reference image-conditioned video editing. However, existing methods still struggle to coordinate information from multiple visual sources accurately. We identify a critical deficiency in existing approaches. Existing editing instructions lack explicit reference relationships, and most multimodal large language models (MLLMs) cannot generate them reliably. To address this problem, we propose ReBind, a systematic framework that introduces semantic instructions with embedded reference tokens as the intermediate representation for multi-reference image-conditioned video editing. Our key insight is embedding reference tokens at semantic positions to eliminate ambiguity and establish precise bindings between visual attributes and their sources. We develop ReBind-Instruct, a specialized MLLM that learns to establish explicit bindings between visual attributes and their reference sources through a two-stage progressive scheme for precise reference relationships. We further develop ReBind-Edit, which enables lightweight adaptation of text-to-video models to coordinate multiple references by binding visual attributes to their designated sources. Extensive experiments demonstrate that ReBind substantially outperforms general-purpose MLLMs in instruction quality and achieves state-of-the-art performance among open-source methods on reference image conditioned video editing. Our project webpage: https://rebind-mrv2v.github.io/.

Abstract

Video Editing

Explore similar work

ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships

MiVE: Multiscale Vision-language features for reference-guided video Editing

Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance