Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
Figures & tables
Figure 1: AVControl trains each control modality as a lightweight LoRA. Each column shows control input (top) and generated output (bottom), covering spatial controls, camera trajectory, motion, editing, and audio-visual generation.
Figure 2: Overview of AVControl. The reference signal is placed on a parallel canvas as additional tokens in self-attention. A LoRA adapter is the only trainable component; the backbone remains frozen.
Figure 3: Spatial concatenation for depth-guided generation. Each panel shows the input depth map (top) and the output from a concatenation-based LoRA (bottom). The model captures general scene semantics but fails to faithfully follow the spatial structure of the depth signal, motivating our adoption of the parallel canvas approach.
Task
Method
AQ
BC
DD
IQ
MS
SC
Avg.
Depth
Control-A-Video [ 13 ]
50.6
91.7
70.0
67.8
97.6
88.1
77.6
VideoComposer [ 57 ]
50.0
94.2
70.0
59.4
96.2
89.8
76.6
ControlVideo [ 72 ]
63.3
95.0
10.0
65.1
96.5
92.3
70.4
VACE
56.7
96.1
60.0
66.4
98.8
94.1
78.7
Ours
62.9
95.1
68.4
70.4
99.0
94.1
81.6
Pose
Text2Video-Zero [ 34 ]
57.6
87.7
100.0
70.7
79.7
84.8
80.1
Table 1: VBench evaluation on the VACE Benchmark. All methods use the same inputs. Baseline numbers from [ 33 ] . Bold : best per metric.
Control Input
Ours
VACE
Figure 4: Qualitative comparison on the VACE Benchmark (depth and pose). Each triplet: control input, ours , VACE [ 33 ] . Our outputs show higher structural fidelity, consistent with Table 1 .
Figure 5: Partial gallery of control modalities. Each row pair shows control input (top) and generated output (bottom, blue border ) across five sampled frames. Each modality is an independent LoRA trained in 200–15,000 steps. We provide additional video examples in the supplementary.
Method
CLIP-F (%) ↑
RotErr ( ∘ ) ↓
GCD [ 54 ]
95.66
2.27
Traj-Attn [ 63 ]
96.52
2.18
DaS [ 21 ]
98.32
1.45
ReCamMaster [ 1 ]
98.74
1.22
Ours
99.13
6.00 †
Table 2: Camera control on the ReCamMaster Benchmark (200 videos). Baselines from [ 1 ] . † COLMAP Structure-from-Motion (SfM) fails on 27% of our videos; SpatialTrackerV2 [ 62 ] (all videos) yields 3.55 ∘ .
Method
FAD ↓
KL ↓
IS ↑
IB ↑
Train Data
ReWaS [ 31 ]
14.71
2.69
8.45
0.15
160K
CAFA [ 4 ]
12.60
2.02
13.45
0.21
200K
MMAudio [ 14 ]
5.32
1.64
17.18
0.33
180K
Ours
57.25
7.74
34.51
0.13
7.8K
Table 3: Audio intensity evaluation on VGGSound. Baselines are dedicated V2A methods; ours generates both audio and video jointly via a single LoRA. Baseline numbers from [ 4 ] . Bold : best per metric.
Method
Sync-C ↑
Sync-D ↓
E-FID ↓
FID ↓
AniPortrait [ 60 ]
3.09
10.94
1.32
32.83
Hallo3 [ 15 ]
6.55
8.49
1.12
33.98
Sonic [ 32 ]
8.35
6.43
1.22
29.53
MultiTalk [ 36 ]
8.54
6.69
1.00
24.01
Ours
4.50
10.31
0.18
12.31
Table 4: Talking-head comparison on HDTF test set. Sync-C (confidence, ↑ ) and Sync-D (distance, ↓ ) measure lip-audio synchronization via SyncNet. E-FID measures expression quality. FID measures visual quality. Baseline numbers are from [ 36 ] .
Cat.
Modality
Steps
Size
Rank
Modules
Spatial
Depth
3K
2K
128
V: SA
Pose
3K
2K
128
V: SA
Canny / edges
3K
2K
32
V: SA
Editing
Inpaint / outpaint
1K
1K
128
V: SA
Local edit
1K
1K
128
V: SA, FF
Video detailing
200
200
128
V: SA, CA, FF
Table 5: Per-modality training details. Steps : optimization steps at the selected checkpoint. Size : number of training samples. Rank : LoRA rank (alpha = rank in all cases). Modules : which transformer blocks receive LoRA adapters (SA = self-attention, CA = cross-attention, FF = feed-forward; prefixed by V = video, A = audio). Video-only LoRAs skip the audio loss; audio-only LoRAs skip the video loss.
Figure 6: Training efficiency comparison. Our per-modality LoRAs range from 200 steps (video detailing) to 15,000 steps (cut-on-action), with most spatially-aligned controls converging in approximately 3,000 steps. Even summing across all 13 trained LoRAs ( ∼ 55K steps total), the training budget is less than one-third of a single VACE training run (200K steps) and comparable to camera-specific methods like BulletTime (40K iterations). This efficiency stems from fine-tuning only existing weights via LoRA adapters on a frozen backbone, with no new layers to train from scratch.
Figure 7: Small-to-large control grid, illustrated on a depth-guided example. The reference canvas resolution scales with information density: each highlighted cell in the reference (top row) maps to the corresponding region in the full-resolution target (bottom row). At 2× downscale, one reference token covers 2×2 target tokens; at 4× , one covers 4×4 . Sparser controls thus require fewer reference tokens, reducing self-attention cost and yielding 35–50% inference speedup at 4× downscale.
Figure 8: VBench average score vs. training steps for the depth LoRA. Performance rises steeply from 500 to 1,000 steps and plateaus beyond 2,000 steps.
Figure 9: Inference-time strength modulation for depth-guided generation. Each row shows four evenly-spaced frames. Top row: depth condition. Subsequent rows: outputs at global strength 0.0, 0.25, 0.5, and 1.0. At lower strengths the model respects the coarse scene layout while exercising creative freedom; at strength 1.0 the output closely follows the depth map. † † footnotemark:
Figure 10: Canny edge-guided generation. Control input (top) and generated output (bottom).
Figure 11: Sparse track-guided generation. Point trajectories rendered as colored dots on a black canvas (top) and generated output (bottom).
Figure 12: Video inpainting. Masked input (top) and completed output (bottom).
Figure 13: Video outpainting. Masked input with border regions (top) and extended output (bottom).
Figure 14: Local video editing. Reference video (top) and edited output (bottom). The LoRA propagates a first-frame edit consistently across the video.
Figure 15: Video detailing (upscaling). Low-resolution input (top) and upscaled output (bottom).
Figure 16: Camera trajectory from a single image. Input image and canonical grid reference (top) and generated video with the target camera motion (bottom).
Figure 17: Diverse camera trajectories from the same input image, demonstrating controllable camera motion.
Figure 18: Camera trajectory from video. Source video re-rendered at a new camera trajectory while preserving scene motion.
Figure 19: Diverse re-rendered camera trajectories from the same source video.
Figure 20: Cut-on-action. The source video (top) is re-rendered from a substantially different camera angle (bottom).
Figure 21: Who-is-talking control. An abstract reference layout of colored bounding boxes with voice activity (top) produces multi-person talking video with synchronized lip motion and audio (bottom).
Recent advances in Diffusion Transformers (DiTs) have enabled high-quality joint audio-video generation, producing videos with synchronized audio within a single model. However, existing controllable generation frameworks are typically restricted to video-only control. This restricts comprehensive controllability and often leads to suboptimal cross-modal alignment. To bridge this gap, we present MMControl, which enables users to perform Multi-Modal Control in joint audio-video generation. MMControl introduces a dual-stream conditional injection mechanism. It incorporates both visual and acoustic control signals, including reference images, reference audio, depth maps, and pose sequences, into a joint generation process. These conditions are injected through bypass branches into a joint audio-video Diffusion Transformer, enabling the model to simultaneously generate identity-consistent video and timbre-consistent audio under structural constraints. Furthermore, we introduce modality-specific guidance scaling, which allows users to independently and dynamically adjust the influence strength of each visual and acoustic condition at inference time. Extensive experiments demonstrate that MMControl achieves fine-grained, composable control over character identity, voice timbre, body pose, and scene layout in joint audio-video generation.
Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-grained controllability remains challenging. Existing methods suffer from weak textual controllability under visual-text conflict and imprecise stylistic control due to entangled temporal and timbre information in reference audio. Moreover, the lack of standardized benchmarks limits systematic evaluation. We propose ControlFoley, a unified multimodal V2A framework that enables precise control over video, text, and reference audio. We introduce a joint visual encoding paradigm that integrates CLIP with a spatio-temporal audio-visual encoder to improve alignment and textual controllability. We further propose temporal-timbre decoupling to suppress redundant temporal cues while preserving discriminative timbre features. In addition, we design a modality-robust training scheme with unified multimodal representation alignment (REPA) and random modality dropout. We also present VGGSound-TVC, a benchmark for evaluating textual controllability under varying degrees of visual-text conflict. Extensive experiments demonstrate state-of-the-art performance across multiple V2A tasks, including text-guided, text-controlled, and audio-controlled generation. ControlFoley achieves superior controllability under cross-modal conflict while maintaining strong synchronization and audio quality, and shows competitive or better performance compared to an industrial V2A system. Code, models, datasets, and demos are available at: https://github.com/xiaomi-research/controlfoley.
Recent diffusion-based methods have achieved impressive progress in video content manipulation. However, they typically ignore the accompanying audio, leaving the audio disjointed from the edited results. In this paper, we propose InstructAV2AV, the first end-to-end framework for instruction-guided audio-video joint editing. We first develop a scalable data synthesis pipeline and construct InsAVE-80K, the first large-scale audio-video editing dataset with high-quality source-to-target pairs. With this data foundation, we adapt an audio-video generation backbone to leverage its robust priors. We concatenate the audio-video input with noisy latent codes to anchor the source context, propose the source-instruction gated attention to improve instruction following and content preservation, and introduce a two-stage training strategy to effectively transfer these pre-trained priors. Extensive experiments demonstrate that InstructAV2AV outperforms state-of-the-art methods across 11 metrics spanning three aspects on two evaluation sets, highlighting its potential for controllable content creation. Project page: https://hjzheng.net/projects/InstructAV2AV/.
Haojie Zheng, Yixin Yang, Siqi Yang +2
Beijing Academy of Artificial Intelligence, Beijing, China · Peking University, Beijing, China