MAV-C: A Framework for the Joint Objective Estimation of Audio-Visual Complexity in Immersive Virtual Environments

Luca Resti; Amelia Gully; Michael McLoughlin; Gavin Kearney; Alena Denisova

eess.ASSep 30, 2026

MAV-C: A Framework for the Joint Objective Estimation of Audio-Visual Complexity in Immersive Virtual Environments

Authors: Luca Resti, Amelia Gully, Michael McLoughlin, Gavin Kearney, Alena Denisova

Organizations: Department of Computer Science, University of York, UK · AudioLab, School of Physics, Engineering and Technology, University of York, UK · Department of Language and Linguistic Science, University of York, UK

Abstract

We present the Motion-Aware Audio-Visual Complexity Metric (MAV-C), a reference-free framework for the joint objective estimation of audio-visual complexity. The metric combines entropy-based audio features (temporal, spectral, and spatial) with visual features (Sobel gradient magnitude, chromatic uniqueness, and optical flow) via a parametric fusion stage, producing a continuous joint complexity score CAV (t) [0,1]. We validate MAV-C on two datasets: a controlled synthetic corpus (SYN) of stimuli with known signal characteristics and a naturalistic gameplay corpus (GAM) of 60 clips drawn from the SAFEPLAY-X dataset. On SYN, the metric exhibits strong validity: the audio score CA and visual score CV are each insensitive to changes in the opposite modality (CoV < 0.003), the joint score CAV spans [0.00,0.90] across all parameter combinations, and single-axis feature sweeps produce monotone trajectories (Spearman up to 0.995). On GAM, CV differs significantly across content categories (Kruskal-Wallis p = 0.021) while CA does not, and the two sub-scores are uncorrelated (r = 0.03), confirming they operate on independent signal dimensions. OFAT sensitivity analysis identifies a two-tier parameter hierarchy, with modality balance (wa) and visual regularization (v) as most significant tunable parameters. Full subjective calibration is planned as future work.

Figures & tables

Figure 1: MAV-C block diagram. Parallel audio and visual pipelines produce sub-scores

C_{A}(t)

and

C_{V}(t)

, fused into joint score

C_{AV}(t)\in[0,1]

. Annotated parameters are tunable.

Figure 2: One-Factor-at-a-Time (OFAT) parameter sensitivity on SYN. Each panel varies one parameter while holding all others at their default values. Panels (a)-(d):

C_{AV}

; panels (e)-(f):

C_{A}

(blue) and

C_{V}

(red). Light bands: full range across 18 conditions. Dark bands: IQR. (a)-(c) primary fusion; (d) temporal integration; (e)-(f) adaptive weight regularization.

Condition	Description
Floor	Silence with a static, low-texture canvas
Ceiling	All axes simultaneously at maximum complexity
Joint	Audio and visual complexity increase together across time
Audio-only	Audio complexity increases, visual content held at low fixed level
Independence	One modality sweeps its full range while the other is held fixed
SASE boundary	Single localized source (floor) vs. fully distributed 64-source field (ceiling); $C_{V}$ held fixed

Table 1: SYN conditions referenced in the results.

Figure 3: Distribution of per-clip mean

C_{AV}

C_{A}

, and

C_{V}

across all 60 GAM clips and all parameter combinations (6600 observations: 60 clips

\times

110 parameter combinations). Violins show full density; boxes show median and IQR; outliers beyond 1.5

\times

IQR are plotted individually.

Figure 4: Per-clip mean

C_{A}

vs.

C_{V}

across 60 GAM clips, averaged over all parameter combinations. Dashed line: linear fit (

r=0.03

p{=}0.824

). Marginal KDE curves for

C_{A}

(blue, top) and

C_{V}

(red, right). The near-zero correlation indicates that

C_{A}

and

C_{V}

vary independently on this corpus.

Explore similar work

Multimodal Confidence Modeling in Audio-Visual Quality Assessment
May 2, 2026Mayesha Maliha R. Mithila, Mylene C. Q. FariasCross-Modal Attention
Texas State University Department of Computer Science
AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection
Apr 2, 2026Damith Chamalke Senadeera, Dimitrios Kollias, Gregory SlabaughAudio-Visual Reasoning Fine-Grained Video Understanding
Queen Mary University of London, UK · Digital Environment Research Institute (DERI), London, UK
MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation
Jul 15, 2026Xiaohan Zhang, Yuqing Wen, Junlin Chen +9Audio-Video Generation Audio-Visual Consistency
Nanjing University · Kling Team · National University of Singapore +3

May 2, 2026, cs.MMJ/K move · Enter open · S save

Multimodal Confidence Modeling in Audio-Visual Quality Assessment

Mayesha Maliha R. Mithila, Mylene C. Q. Farias

Texas State University Department of Computer Science

Cross-Modal Attention

Open paper AskPDF arXiv

Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortions are often asymmetric, where one modality may be severely degraded while the other remains clean. Still, most contemporary AVQA metrics treat audio and video as equally reliable, causing confidence-unaware fusion to emphasize unreliable signals. This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework that explicitly estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention. The Audio-Visual Mixer utilizes frame-level, confidence-guided channel attention to gate fusion, modulating feature interaction between modalities so that high-confidence streams dominate while unreliable inputs are suppressed, preserving temporal degradation patterns. A multi-head visual confidence estimator turns frame-level artifact probabilities into temporally smoothed, clip-level visual confidence scores, while an audio confidence module derives confidence from speech-quality cues without requiring a clean reference. Experiments on multiple AVQA benchmarks show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores and yield more interpretable behavior under real-world asymmetric audio-visual distortions.

Abstract

Explore similar work

Multimodal Confidence Modeling in Audio-Visual Quality Assessment

AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection

MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation