AVControl: Efficient Framework for Training Audio-Visual Controls

Matan Ben-Yosef; Tavi Halperin; Naomi Ken Korem; Mohammad Salama; Harel Cain; Asaf Joseph; Anthony Chen; Urska Jelercic

cs.CVMar 25, 2026

AVControl: Efficient Framework for Training Audio-Visual Controls

Authors: Matan Ben-Yosef, Tavi Halperin, Naomi Ken Korem, Mohammad Salama, Harel Cain, Asaf Joseph, Anthony Chen, Urska Jelercic,

Organizations: Lightricks

Abstract

Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.

Figures & tables

Panel of Figure 1 — Figure 1: AVControl trains each control modality as a lightweight LoRA. Each column shows control input (top) and generated output (bottom), covering spatial controls, camera trajectory, motion, editing, and audio-visual generation.

Figure 2: Overview of AVControl. The reference signal is placed on a parallel canvas as additional tokens in self-attention. A LoRA adapter is the only trainable component; the backbone remains frozen.

Panel of Figure 3 — Figure 3: Spatial concatenation for depth-guided generation. Each panel shows the input depth map (top) and the output from a concatenation-based LoRA (bottom). The model captures general scene semantics but fails to faithfully follow the spatial structure of the depth signal, motivating our adoption of the parallel canvas approach.

Task	Method	AQ	BC	DD	IQ	MS	SC	Avg.
Depth	Control-A-Video [ 13 ]	50.6	91.7	70.0	67.8	97.6	88.1	77.6
	VideoComposer [ 57 ]	50.0	94.2	70.0	59.4	96.2	89.8	76.6
	ControlVideo [ 72 ]	63.3	95.0	10.0	65.1	96.5	92.3	70.4
	VACE	56.7	96.1	60.0	66.4	98.8	94.1	78.7
	Ours	62.9	95.1	68.4	70.4	99.0	94.1	81.6
Pose	Text2Video-Zero [ 34 ]	57.6	87.7	100.0	70.7	79.7	84.8	80.1

Table 1: VBench evaluation on the VACE Benchmark. All methods use the same inputs. Baseline numbers from [ 33 ] . Bold : best per metric.

Control Input	Ours	VACE

Figure 4: Qualitative comparison on the VACE Benchmark (depth and pose). Each triplet: control input, ours , VACE [ 33 ] . Our outputs show higher structural fidelity, consistent with Table 1 .

Panel of Figure 5 — Figure 5: Partial gallery of control modalities. Each row pair shows control input (top) and generated output (bottom, blue border ) across five sampled frames. Each modality is an independent LoRA trained in 200–15,000 steps. We provide additional video examples in the supplementary.

Method	CLIP-F (%) $\uparrow$	RotErr ( ∘ ) $\downarrow$
GCD [ 54 ]	95.66	2.27
Traj-Attn [ 63 ]	96.52	2.18
DaS [ 21 ]	98.32	1.45
ReCamMaster [ 1 ]	98.74	1.22
Ours	99.13	6.00 †

Table 2: Camera control on the ReCamMaster Benchmark (200 videos). Baselines from [ 1 ] . † COLMAP Structure-from-Motion (SfM) fails on 27% of our videos; SpatialTrackerV2 [ 62 ] (all videos) yields 3.55 ∘ .

Method	FAD $\downarrow$	KL $\downarrow$	IS $\uparrow$	IB $\uparrow$	Train Data
ReWaS [ 31 ]	14.71	2.69	8.45	0.15	160K
CAFA [ 4 ]	12.60	2.02	13.45	0.21	200K
MMAudio [ 14 ]	5.32	1.64	17.18	0.33	180K
Ours	57.25	7.74	34.51	0.13	7.8K

Table 3: Audio intensity evaluation on VGGSound. Baselines are dedicated V2A methods; ours generates both audio and video jointly via a single LoRA. Baseline numbers from [ 4 ] . Bold : best per metric.

Method	Sync-C $\uparrow$	Sync-D $\downarrow$	E-FID $\downarrow$	FID $\downarrow$
AniPortrait [ 60 ]	3.09	10.94	1.32	32.83
Hallo3 [ 15 ]	6.55	8.49	1.12	33.98
Sonic [ 32 ]	8.35	6.43	1.22	29.53
MultiTalk [ 36 ]	8.54	6.69	1.00	24.01
Ours	4.50	10.31	0.18	12.31

Table 4: Talking-head comparison on HDTF test set. Sync-C (confidence,

\uparrow

) and Sync-D (distance,

\downarrow

) measure lip-audio synchronization via SyncNet. E-FID measures expression quality. FID measures visual quality. Baseline numbers are from [ 36 ] .

Cat.	Modality	Steps	Size	Rank	Modules
Spatial	Depth	3K	2K	128	V: SA
	Pose	3K	2K	128	V: SA
	Canny / edges	3K	2K	32	V: SA
Editing	Inpaint / outpaint	1K	1K	128	V: SA
	Local edit	1K	1K	128	V: SA, FF
	Video detailing	200	200	128	V: SA, CA, FF

Table 5: Per-modality training details. Steps : optimization steps at the selected checkpoint. Size : number of training samples. Rank : LoRA rank (alpha

=

rank in all cases). Modules : which transformer blocks receive LoRA adapters (SA = self-attention, CA = cross-attention, FF = feed-forward; prefixed by V = video, A = audio). Video-only LoRAs skip the audio loss; audio-only LoRAs skip the video loss.

Figure 6: Training efficiency comparison. Our per-modality LoRAs range from 200 steps (video detailing) to 15,000 steps (cut-on-action), with most spatially-aligned controls converging in approximately 3,000 steps. Even summing across all 13 trained LoRAs ( ${\sim}$ 55K steps total), the training budget is less than one-third of a single VACE training run (200K steps) and comparable to camera-specific methods like BulletTime (40K iterations). This efficiency stems from fine-tuning only existing weights via LoRA adapters on a frozen backbone, with no new layers to train from scratch.

Figure 7: Small-to-large control grid, illustrated on a depth-guided example. The reference canvas resolution scales with information density: each highlighted cell in the reference (top row) maps to the corresponding region in the full-resolution target (bottom row). At $2\times$ downscale, one reference token covers $2{\times}2$ target tokens; at $4\times$ , one covers $4{\times}4$ . Sparser controls thus require fewer reference tokens, reducing self-attention cost and yielding 35–50% inference speedup at $4\times$ downscale.

Figure 8: VBench average score vs. training steps for the depth LoRA. Performance rises steeply from 500 to 1,000 steps and plateaus beyond 2,000 steps.

Figure 9: Inference-time strength modulation for depth-guided generation. Each row shows four evenly-spaced frames. Top row: depth condition. Subsequent rows: outputs at global strength 0.0, 0.25, 0.5, and 1.0. At lower strengths the model respects the coarse scene layout while exercising creative freedom; at strength 1.0 the output closely follows the depth map. † † footnotemark:

Figure 10: Canny edge-guided generation. Control input (top) and generated output (bottom).

Figure 11: Sparse track-guided generation. Point trajectories rendered as colored dots on a black canvas (top) and generated output (bottom).

Figure 12: Video inpainting. Masked input (top) and completed output (bottom).

Figure 13: Video outpainting. Masked input with border regions (top) and extended output (bottom).

Figure 14: Local video editing. Reference video (top) and edited output (bottom). The LoRA propagates a first-frame edit consistently across the video.

Figure 15: Video detailing (upscaling). Low-resolution input (top) and upscaled output (bottom).

Figure 16: Camera trajectory from a single image. Input image and canonical grid reference (top) and generated video with the target camera motion (bottom).

Figure 17: Diverse camera trajectories from the same input image, demonstrating controllable camera motion.

Figure 18: Camera trajectory from video. Source video re-rendered at a new camera trajectory while preserving scene motion.

Figure 19: Diverse re-rendered camera trajectories from the same source video.

Figure 20: Cut-on-action. The source video (top) is re-rendered from a substantially different camera angle (bottom).

Figure 21: Who-is-talking control. An abstract reference layout of colored bounding boxes with voice activity (top) produces multi-person talking video with synchronized lip motion and audio (bottom).

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 321 papers

MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation
Apr 21, 2026Liyang Li, Wen Wang, Canyu Zhao +4Diffusion Model Guidance Audio-Video Generation
Zhejiang University, Hangzhou, China
ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
Apr 16, 2026Jianxuan Yang, Xinyue Guo, Zhi Cheng +10Audio-Video Generation Multimodal Generation
MiLM Plus, Xiaomi Inc. · Wuhan University
InstructAV2AV: Instruction-Guided Audio-Video Joint Editing
May 18, 2026Haojie Zheng, Yixin Yang, Siqi Yang +2Audio-Video Generation Video Diffusion Models
Beijing Academy of Artificial Intelligence, Beijing, China · Peking University, Beijing, China

Apr 21, 2026J/K move · Enter open · S save

MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation

Liyang Li, Wen Wang, Canyu Zhao, Tianjian Feng+3

Zhejiang University, Hangzhou, China

Diffusion Model Guidance Audio-Video Generation Conditional Diffusion Models

Open paper AskPDF arXiv

Recent advances in Diffusion Transformers (DiTs) have enabled high-quality joint audio-video generation, producing videos with synchronized audio within a single model. However, existing controllable generation frameworks are typically restricted to video-only control. This restricts comprehensive controllability and often leads to suboptimal cross-modal alignment. To bridge this gap, we present MMControl, which enables users to perform Multi-Modal Control in joint audio-video generation. MMControl introduces a dual-stream conditional injection mechanism. It incorporates both visual and acoustic control signals, including reference images, reference audio, depth maps, and pose sequences, into a joint generation process. These conditions are injected through bypass branches into a joint audio-video Diffusion Transformer, enabling the model to simultaneously generate identity-consistent video and timbre-consistent audio under structural constraints. Furthermore, we introduce modality-specific guidance scaling, which allows users to independently and dynamically adjust the influence strength of each visual and acoustic condition at inference time. Extensive experiments demonstrate that MMControl achieves fine-grained, composable control over character identity, voice timbre, body pose, and scene layout in joint audio-video generation.

Abstract

Controllable Video Generation

Explore similar work

MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation

ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling

InstructAV2AV: Instruction-Guided Audio-Video Joint Editing