Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

Shivanshu Shekhar; Raghavendra Addanki; Tong Zhang; Uttaran Bhattacharya

cs.CVOct 8, 2026

Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

Authors: Shivanshu Shekhar, Raghavendra Addanki, Tong Zhang, Uttaran Bhattacharya

Organizations: University of Illinois Urbana-Champaign · Adobe Research

Abstract

Reward-guided inference-time alignment can improve joint audio-visual generation, but optimizing multiple alignment objectives requires many expensive model evaluations, making existing methods costly under limited compute. Particle-based search enables global exploration but incurs estimation error with a finite population, while Metropolis-Hastings (MH) can refine candidates locally but benefits from a strong initialization. We introduce Spend Where It Matters (\modelname), a two-stage, training-free, reward-agnostic framework that combines particle search with MH refinement under a fixed compute budget. In the first stage, we adaptively allocate particles across denoising checkpoints using online variance estimates, minimizing estimation variance while concentrating compute where it is most useful. This search provides a promising candidate for MH refinement, which uses local noise-and-denoise proposals on completed outputs to further improve reward. By starting from a strong candidate, MH can recover additional reward without requiring a larger particle population or costly global exploration. We show that, at matched compute, combining particle search with MH refinement achieves higher rewards than either approach alone. On JavisBench-Mini, at closely matched compute, \modelname achieves mean relative gains across seven metrics of $30.7\%$ over BoN and $14.1\%$ over EvoSearch with JavisDiT++, and $9.2\%$ over BoN and $17.0\%$ over EvoSearch with LTX-2. Beyond joint audio-visual generation, we also demonstrate the benefits of adaptive particle allocation on SD1.5 image generation, improving AestheticScore by $2.2\%$ and ImageReward by $6.0\%$ over uniform allocation.

Figures & tables

Panel of Figure 1 — Figure 1: Visual results on JavisBench-Mini ( Liu et al., 2026a ) . For each prompt, we show the ARW-based EvoSearch and SWIM outputs for both JavisDiT++ and LTX-2, each as a frame strip with its audio waveform below. Cues in the prompts are highlighted as visual or audio .

Model	Method	TV-IB $\uparrow$	TA-IB $\uparrow$	AV-IB $\uparrow$	AVH $\uparrow$	JS $\uparrow$	CLAP $\uparrow$	VR-TA $\uparrow$
JavisDiT++	BoN@400	$0.3780$ $(0.0002)$	$0.1496$ $(0.0006)$	$0.1859$ $(0.0015)$	$0.1711$ $(0.0016)$	$0.1021$ $(0.0015)$	$0.1474$ $(0.0015)$	$0.3711$ $(0.0015)$
	BoN@600	$0.3785$ $(0.0003)$	$0.1508$ $(0.0013)$	$0.1896$ $(0.0018)$	$0.1746$ $(0.0013)$	$0.1035$ $(0.0003)$	$0.1464$ $(0.0017)$	$0.3870$ $(0.0001)$
	EvoSearch@399	$0.3762$ $(0.0011)$	$0.1766$ $(0.0012)$	$0.2312$ $(0.0005)$	$0.2126$ $(0.0016)$	$0.1380$ $(0.0016)$	$0.1857$ $(0.0014)$	$0.3074$ $(0.0048)$
	EvoSearch@532	$0.3767$ $(0.0006)$	$0.1838$ $(0.0022)$	$0.2382$ $(0.0003)$	$0.2197$ $(0.0013)$	$0.1441$ $(0.0004)$	$\underline{0.1933}$ $(0.0024)$	$0.3223$ $(0.0271)$
	Uniform@400	$\mathbf{0.3824}$ $(0.0004)$	$0.1861$ $(0.0026)$	$0.2218$ $(0.0016)$	$0.2098$ $(0.0019)$	$0.1125$ $(0.0023)$	$0.1868$ $(0.0066)$	$\mathbf{0.6245}$ $(0.0036)$
	Uniform+MH@360	$0.3755$ $(0.0010)$	$\mathbf{0.1900}$ $(0.0002)$	$\underline{0.2434}$ $(0.0021)$	$\underline{0.2335}$ $(0.0023)$	$\underline{0.1692}$ $(0.0024)$	$\underline{0.1878}$ $(0.0022)$	$0.4417$ $(0.0132)$

Table 1: Results on JavisBench-Mini ( Liu et al., 2026a ) . Budget is reported as total NFE (forward passes). SWIM achieves the strongest results at lower budgets, the practical regime for expensive AV models. Numbers reported as

\mathrm{mean}(\mathrm{std\ dev})

\uparrow:

higher is better.

Method	SimpleAnimals: AestheticScore $\uparrow$	DrawBench: ImageReward $\uparrow$
Uniform+MH(Sys)@400	$6.506$ $(0.018)$	$0.9890$ $(0.027)$
Uniform+MH(Mul)@400	$6.395$ $(0.016)$	$0.9129$ $(0.030)$
Uniform+MH(FV)@400	$6.452$ $(0.024)$	$0.9677$ $(0.023)$
SWIM-Sys@400	$\mathbf{6.652}$ $(0.021)$	$\mathbf{1.048}$ $(0.022)$
SWIM-Mul@400	$\mathbf{6.528}$ $(0.029)$	$\mathbf{0.978}$ $(0.031)$
SWIM-FV@400	$\mathbf{6.602}$ $(0.017)$	$\mathbf{1.010}$ $(0.025)$

Table 2: Results on DrawBench (ImageReward) ( Saharia et al., 2022 ) and Simple Animals (AestheticScore) ( Black et al., 2024 ) . SWIM outperforms fixed-population baselines across all resampling strategies at matched NFE. Numbers reported as

\mathrm{mean}(\mathrm{std\ dev})

\uparrow:

higher is better.

Panel of Figure 2 — Figure 2: Human preference responses comparing SWIM JavisDiT++ and LTX-2 on JavisBench-Mini ( Liu et al., 2026a ) . Bars show relative preferences on text-visual, text-audio, and audio-visual alignments, as well as visual, audio, and overall aesthetics. SWIM is preferred over the corresponding baseline across all the preference criteria.

Panel of Figure 6 — Figure 6: TV-IB and estimator-variance surrogate versus NFE on JavisDiT++. The left panel shows TV-IB and the right panel shows the variance surrogate across compute budgets. Adaptive allocation achieves higher TV-IB and lower variance than uniform- $K$ throughout the tested range.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Effect of the staged reward parameters. The left panel shows the effect of when the text-audio reward is enabled, and the right panel shows its weight $\beta_{\mathrm{mix}}$ . An intermediate activation threshold and moderate weight give the best balance between TA-IB and VR-TA.

Panel of Figure 8 — Figure 8: Effect of the pacing floor $\gamma$ on the adaptive funnel. Left: final reward and realized NFE. Right: actual particle schedule across checkpoints. Larger $\gamma$ forces the schedule toward a more uniform allocation, and intermediate pacing achieves the best reward.

Panel of Figure 9 — Figure 9: Effect of the dual step size $\eta_{\nu}$ on the funnel’s compute schedule under JavisDiT++. The left panel shows remaining NFE budget and the right panel shows mean population size $K$ across checkpoints. Larger $\eta_{\nu}$ responds more aggressively to budget errors, while schedules remain identical until the first update to $\nu$ .

Panel of Figure 10 — Figure 10: Score and estimator variance under compute scaling for TV-IB in audio-visually aligned video generation. At full budget (NFE $\approx$ 500), both methods perform comparably on score but SWIM retains a substantial advantage on variance. As budget decreases, SWIM maintains higher quality with estimator variance increasing comparatively smoothly, while uniform- $K$ degrades more sharply. This behavior results from concentrating pruning at low-value checkpoints while preserving particles at informative middle checkpoints. Results on JavisDiT++ with 50 prompts and 3 seeds. Error bars show $\pm 1$ standard error across prompts.

Panel of Figure 11 — Figure 11: Score and estimator variance under compute scaling for AestheticScore in image generation. At full budget (NFE $\approx$ 1000), both methods perform comparably. As budget decreases, SWIM maintains higher quality with estimator variance increasing comparatively smoothly, while uniform- $K$ degrades more sharply. This behavior results from concentrating pruning at low-value checkpoints while preserving particles at informative middle checkpoints. Results on SD1.5 with 45 prompts and 3 seeds. Error bars show $\pm 1$ standard error across prompts.

Figure 12: Effect of population growth strategy on VR-TA improvement over base generation. Rescue achieves the largest gain among five strategies, combining reward-guided resampling with lineage preservation during population expansion.Results on JavisDiT++ with 50 prompts and 3 seeds.

Panel of Figure 13 — Figure 13: Reward reliability across denoising checkpoints. We plot the score difference between two dissimilar prompts. Shaded regions show standard deviation across prompts and seeds. TA-IB has little discriminative power early, while TV-IB distinguishes prompts earlier. Both become more discriminative as denoising progresses.

Independent Metropolis-Hastings

See all 8 papers

Oct 12, 2026

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-

N

selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at https://github.com/WeChatCV/SyncReward.

Kaisi Guan, Ying Ba, Wenjing Wang +4

Gaoling School of Artificial Intelligence, Renmin University of China · WeChat, Tencent Inc.

Audio-Visual Synchronization Audio-Video Generation Reward Modeling Generative AI Evaluation

Ask PDF

May 28, 2026

Native Audio-Visual Alignment for Generation

Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.

Longbin Ji, Guan Wang, Xuan Wei +6

ERNIE Team, Baidu Inc.

Audio-Video Generation Audio-Visual Synchronization Audio-Visual Alignment Multimodal Diffusion Models

Ask PDF

Jun 2, 2026

Inference-Time Scaling for Joint Audio-Video Generation

Joint audio-video generation aims to synthesize realistic audio-video pairs that are both semantically aligned with text prompts and precisely synchronized. While existing joint audio-video generation models often require substantial training resources to improve fidelity, Inference-Time Scaling (ITS) has recently emerged as a promising training-free alternative in single-modality domains. However, extending ITS from a single modality to multimodal domains is non-trivial, as it requires balancing multiple heterogeneous objectives. In this paper, we present the first comprehensive study of ITS for joint audio-video generation. We first demonstrate that a multi-verifier framework is essential to address the limitations of single-objective guidance, including asymmetric performance trade-offs and verifier hacking. Through systematic analysis, we then identify an optimal multi-verifier combination that yields balanced improvements across all quality dimensions. Finally, to effectively aggregate diverse reward signals, we propose Adaptive Reward Weighting (ARW), a novel test-time optimization algorithm. ARW treats reward aggregation as an online optimization problem, utilizing learnable parameters to calibrate reward variances without requiring prior knowledge of reward distributions, thereby ensuring robust multi-objective selection. Experimental results on VGGSound and JavisBench-mini benchmarks demonstrate that our framework significantly enhances semantic alignment, perceptual quality, and audio-visual synchronization of generated outputs. Synthesized samples and code are available on the project page: https://jung-jaemin.github.io/ITS-AVGen-Proj.

Jaemin Jung, Kyeongha Rho, Inkyu Shin +1

Korea Advanced Institute of Science and Technology · Luma AI

Audio-Video Generation Inference-Time Scaling Audio-Visual Synchronization Audio-Visual Alignment Multi-Objective Optimization

Ask PDF

Abstract

Independent Metropolis-Hastings

Explore similar work

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Native Audio-Visual Alignment for Generation

Inference-Time Scaling for Joint Audio-Video Generation