CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

Mengzhe Geng; Max W. Y. Lam; Yuewen Cao; Shoukang Hu; Jing Xu; Marc Tessier; Xu Li

cs.LGOct 11, 2026

CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

Authors: Mengzhe Geng, Max W. Y. Lam, Yuewen Cao, Shoukang Hu, Jing Xu, Marc Tessier, Xu Li

Organizations: National Research Council Canada · The Chinese University of Hong Kong

Abstract

Removing audio can shift a visual predictor's output without showing whether the task uses the intended audio information. We introduce Conditional Input and Visual Interpretation Checks (CIViC), a four-gate protocol that links an interpretation to an intervention, held-out task comparison, or endpoint control. We demonstrate CIViC on actor-disjoint face-raster tests, a synthetic audio-feature mixture, and source-video-disjoint SpeakerVid proxy tasks. In the static face-raster test, cross-actor audio replacement meets the registered detection rule with a small effect; on a separate temporal endpoint, constant-audio fits score better than audio-trained fits in four matched seeds. In the synthetic mixture, the condition-augmented fit has higher 1,000-step RMSE than the audio-only fit (0.1807 vs. 0.1707), with the gap shrinking during training. In the SpeakerVid proxy, the five-feature-zero fit (five low-level audio summaries zeroed; identity, dialogue, and condition inputs retained) has lower whole-frame grayscale RMSE than aligned audio across four held-out conditions and five paired seeds, while aligned audio has lower segment-drift error. The results show that output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks.

Figures & tables

Evaluation action	What it can establish	What CIViC adds before the stated interpretation
Audio removal or ablation	The registered path affects this output	A matched replacement for a value-specific statement
Input replacement	The stated alternative changes this endpoint	Held-out task evidence when benefit is asserted
Held-out task score	An ordering on the registered endpoint	A shortcut control for a transfer or proxy statement
Perturbation or defense test	Robustness to the tested disruption	A gate-specific stop rule and retained failed gate

Table 1: From common evaluation actions to bounded interpretations. The entries are complementary, not a ranking of prior work.

Gate	Required evidence	What a detection does not establish
Activation	Output changes after the registered removal	Which input values the model uses
Substitution	Matched replacement changes the stated endpoint	Task benefit or general audio use
Benefit	Matched held-out error comparison	A mechanism for any observed gap
Endpoint validity	Control excludes the named shortcut	Validity outside the checked endpoint

Table 2: Interpretation rules. A detection result excludes only the named alternative; practical use needs a task-scale rule.

Figure 1: Four interpretation gates (A) and two selected diagnostic summaries (B). CREMA-D meets its registered fixed-pair detection rule but has no practical threshold. In the synthetic feature-mixture proxy, the condition changes predictions while early held-out error is higher; that gap shrinks with training. Its matched endpoint control is untested. The newer SpeakerVid proxy comparison is described in Section 5 , not shown here. Color and text distinguish the gates.

Control	Control MAE $-$ aligned MAE $\uparrow$	Paired 95% interval
Zero audio	+0.131475	[0.127163, 0.135640]
Actor-audio replacement	+0.000109	[0.000034, 0.000186]
Identity-raster replacement	+0.064860	[0.062795, 0.066897]

Table 3: CREMA-D static confirmation on 732 fixed pairs from an actor-disjoint split. Differences are control MAE minus aligned MAE; aligned MAE is 0.02907. Paired intervals use 10,000 clip-level resamples with seed 1701 for the fixed checkpoint and pair set; they do not quantify actor-population uncertainty. Positive differences favor aligned input.

Figure 2: Registered 1,000-step feature-mixture evidence. (A) Adding the condition raises withheld-composition RMSE in all three matched seeds; the target is a dimensionless stored proxy. (B) Zeroing the condition changes proxy predictions in the same runs. Points are seed values; markers are mean

\pm

standard deviation. Activation is not a task-error comparison.

Training support	Audio-only RMSE $\downarrow$	Added-condition RMSE $\downarrow$	Gap $\downarrow$
Single clean	$0.4249\pm 0.0097$	$0.6921\pm 0.0218$	$+0.2672\pm 0.0276$
Single environment	$0.1638\pm 0.0026$	$0.4192\pm 0.0239$	$+0.2554\pm 0.0240$
Single music	$0.1681\pm 0.0025$	$0.4169\pm 0.0247$	$+0.2487\pm 0.0252$
Clean plus environment	$0.1685\pm 0.0027$	$0.4121\pm 0.0225$	$+0.2436\pm 0.0231$

Table 4: Twenty-four-seed, 1,000-step training-support audit on the held-out mixed condition (mean

\pm

population standard deviation). The gap is added-condition minus audio-only RMSE; positive gaps favor audio only. Each policy changes training support and is not pooled with the three-seed result.

Steps	Audio-only RMSE $\downarrow$	Added-condition RMSE $\downarrow$	Gap $\downarrow$
1,000	$0.1707\pm 0.0024$	$0.1807\pm 0.0059$	$+0.0100\pm 0.0040$
3,000	$0.1338\pm 0.0004$	$0.1366\pm 0.0023$	$+0.0028\pm 0.0019$
10,000	$0.1237\pm 0.0014$	$0.1256\pm 0.0042$	$+0.0019\pm 0.0032$

Table 5: Withheld mixed-composition error at three checkpoints of the same three-seed runs (mean

\pm

population standard deviation; lower RMSE is better). Audio only omits the seven-bit condition; both arms retain speaker and dialogue inputs. The gap is added-condition minus audio-only RMSE; positive favors audio only.

Control	Audio	Condition relation	Mixed RMSE $\downarrow$
Dummy condition	Observed	Deterministic dummy	$0.1822\pm 0.0025$
Shuffled condition	Observed	Train shuffled; test aligned	$0.4248\pm 0.0217$
Five-feature-zeroed	Zeroed	Aligned	$0.5003\pm 0.0278$

Table 6: Five-seed parameter-matched controls on the held-out mixture under clean plus environment training support (mean

\pm

population standard deviation; seeds 1234, 2027, 4096, 8191, 16384). All entries use seven condition coordinates and 4,908,416 parameters. Shuffled values are used in training but aligned values in evaluation; this is a policy-specific diagnostic.

Condition	H100 $\Delta$ RMSE $\downarrow$	A100-1 $\Delta$ RMSE $\downarrow$	A100-2 $\Delta$ RMSE $\downarrow$	V100 $\Delta$ RMSE $\downarrow$
Clean	$-0.0142\pm 0.0036$	$-0.0143\pm 0.0035$	$-0.0143\pm 0.0035$	$-0.0142\pm 0.0036$
Environment-like	$-0.0214\pm 0.0022$	$-0.0215\pm 0.0022$	$-0.0215\pm 0.0022$	$-0.0216\pm 0.0024$
Music-like	$-0.0210\pm 0.0019$	$-0.0211\pm 0.0019$	$-0.0211\pm 0.0019$	$-0.0213\pm 0.0020$
Mixed	$-0.0206\pm 0.0024$	$-0.0208\pm 0.0024$	$-0.0208\pm 0.0024$	$-0.0212\pm 0.0027$

Table 7: Paired test raster-reconstruction RMSE differences for the five-feature-zero control, computed as five-feature-zero minus aligned audio. Entries are mean

\pm

sample standard deviation over five paired seed differences. Negative values favor five-feature-zero. The four platform runs share one source pack and are hardware or execution replications, not independent data. The secondary segment-drift proxy favors aligned audio in every condition; Table 8 reports that opposing result.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Raster $\Delta$ RMSE $\downarrow$	Drift $\Delta$ $\downarrow$	Positive drift seeds
Clean	$[-0.0143,-0.0142]$	$[0.0057,0.0064]$	$4/5$
Environment-like	$[-0.0216,-0.0214]$	$[0.0076,0.0079]$	$5/5$
Music-like	$[-0.0213,-0.0210]$	$[0.0069,0.0070]$	$5/5$
Mixed	$[-0.0212,-0.0206]$	$[0.0043,0.0050]$	$4/5$

Appendix

Table 8: Five-feature-zero minus aligned-audio test differences for two lower-is-better proxies. Negative raster-reconstruction RMSE differences favor five-feature-zero, while positive drift differences favor aligned audio. The flattened-window drift statistic does not measure continuous-video quality. Ranges span four platform means computed from the same five seeds and four-video test subset; they are descriptive, not confidence intervals.

Condition	Five-feature-zero $\Delta$ RMSE $\downarrow$	Audio-only $\Delta$ RMSE $\downarrow$	Dummy $\Delta$ RMSE $\downarrow$	Shuffled $\Delta$ RMSE $\downarrow$
Clean	$[-0.0143,-0.0142]$	$[-0.0040,-0.0037]$	$[0.0165,0.0167]$	$[0.0006,0.0007]$
Environment-like	$[-0.0216,-0.0214]$	$[-0.0045,-0.0042]$	$[0.0165,0.0167]$	$[0.0011,0.0012]$
Music-like	$[-0.0213,-0.0210]$	$[-0.0046,-0.0042]$	$[0.0163,0.0166]$	$[0.0011,0.0012]$
Mixed	$[-0.0212,-0.0206]$	$[-0.0106,-0.0101]$	$[0.0108,0.0111]$	$[0.0019,0.0022]$

Appendix

Table 9: Test raster-reconstruction RMSE difference (control minus aligned audio) by condition. Each cell is the minimum–maximum of four platform-level means, each computed from five paired seed differences. These ranges are not confidence intervals; platforms share the same held-out videos and seeds. Negative values favor the named control.

Unit	$n$	Clean $\Delta$ RMSE $\downarrow$	Environment $\Delta$ RMSE $\downarrow$	Music $\Delta$ RMSE $\downarrow$	Mixed $\Delta$ RMSE $\downarrow$
1	26	$[-0.0124,-0.0123]$	$[-0.0175,-0.0172]$	$[-0.0189,-0.0188]$	$[-0.0173,-0.0169]$
2	20	$[-0.0135,-0.0127]$	$[-0.0267,-0.0258]$	$[-0.0243,-0.0237]$	$[-0.0334,-0.0326]$
3	48	$[-0.0161,-0.0158]$	$[-0.0232,-0.0231]$	$[-0.0225,-0.0222]$	$[-0.0204,-0.0199]$
4	14	$[-0.0134,-0.0132]$	$[-0.0192,-0.0191]$	$[-0.0199,-0.0195]$	$[-0.0185,-0.0177]$

Appendix

Table 10: Five-feature-zero minus aligned-audio test raster-reconstruction RMSE within each held-out source video. Unit numbers are labels for the four videos, not new sampling units;

n

counts unique windows per video, each reused across four conditions. Cells give the minimum–maximum of four five-seed platform means, not population intervals. Negative values favor five-feature-zero.

Seed	Max. epochs	Aligned MAE $\downarrow$	Constant MAE $\downarrow$	$\Delta$ MAE $\downarrow$
1701	40	0.08990	0.08652	$-0.00338$
1701	80	0.08803	0.08599	$-0.00204$
1702	40	0.08979	0.08749	$-0.00230$
1702	80	0.08874	0.08740	$-0.00133$
1703	40	0.08984	0.08719	$-0.00265$
1703	80	0.08841	0.08770	$-0.00070$

Appendix

Table 11: RAVDESS same-target sensitivity on 240 fixed targets. Maximum epochs is the stopping cap; validation selection can stop earlier.

\Delta

is constant-audio-fit MAE minus aligned-audio-fit MAE, so negative favors the constant-audio fit. The 80-epoch runs are fresh post-confirmation fits, not exact continuations of the 40-epoch trajectories.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

When Vision Speaks for Sound
May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model Evaluation Audio-Visual Understanding
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1
Intervention, Not Shared Latents: Blocking Visual Shortcuts in Audio-Video Generation
Aug 14, 2026Jian Xu, Delu Zeng, John PaisleyAudio-Video Generation Causal Inference
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models
Apr 15, 2026Ami Baid, Zihui Xue, Kristen GraumanPreference Optimization Multimodal Contrastive Learning
University of Texas at Austin, Austin, USA

May 13, 2026J/K move · Enter open · S save

When Vision Speaks for Sound

Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai+5

dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison+1

Audio-Language Model Evaluation Audio-Visual Understanding Video-Language Model Evaluation Audio Reasoning Audio Understanding Video-Language Models+5

Open paper AskPDF arXiv

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

Abstract

Multimodal Model Evaluation

Explore similar work

When Vision Speaks for Sound

Intervention, Not Shared Latents: Blocking Visual Shortcuts in Audio-Video Generation

Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models