cs.CVAug 26, 2026

Asymmetric Cross-Modal Fine-Grained Visual Categorization: ACF-Net and the BirdPro Benchmark

Authors: Bohan Deng, Shuo Ye, Zitong Yu

Organizations: Great Bay University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Audio-visual cross-modal Fine-Grained Visual Categorization (FGVC) aims to identify fine-grained categories by jointly leveraging visual and auditory information. However, FGVC under asymmetric cross-modal scenarios has received limited attention, where paired video and audio are not strictly synchronized and may not even correspond to the same individual or moment. Such weak and ambiguous cross-modal correspondence poses substantial challenges to effective representation learning and modality alignment. To address these issues, we propose ACF-Net, a novel optical flow-guided framework for asymmetric audio-visual fine-grained learning. ACF-Net consists of two key modules: Optical Flow-Guided Motion (OFGM) and Asymmetric CrossModal Adaptive Fusion (ACAF). OFGM captures motion-sensitive visual cues and suppresses irrelevant background interference, thereby enhancing discriminative dynamic representations in videos. ACAF estimates modality reliability under weakly matched audio-video pairs and performs uncertainty-aware adaptive fusion to improve category-level recognition robustness. To support research on asymmetric cross-modal FGVC, we further construct BirdPro, a new bird-oriented audio-visual benchmark, since existing datasets often lack large-scale category-level audio-video associations under non-strict temporal and instance correspondence. BirdPro contains 1,919 audio recordings and 11,965 videos covering 194 bird species. Extensive experiments show that ACF-Net achieves the best results compared with representative baseline methods, outperforming the strongest baselines by 2.97% and 1.92% in the fused and mismatched settings, respectively.

Figures & tables

Figure 1: Motivation of asymmetric audio-visual FGVC. (a) Unimodal video FGVC [ 13 ] suffers from fragile visual cues under pose variation, occlusion, or background clutter. (b) Synchronized audio-visual FGVC [ 10 ] leverages complementary acoustic cues under strict temporal and instance-level correspondence. (c) Asymmetric audio-visual FGVC only assumes category-level semantic consistency, where audio and video may differ in instance, time, or context.
Figure 2: Framework of ACF-Net. OFGM enhances RGB frames with optical-flow-guided motion cues, while ACAF adaptively fuses multimodal features for final classification.
MClassesTrainTest
Video19410,0541,911
Audio1941,531388
Pair1946,022470
Table 1: Statistics of BirdPro. “M” denotes the modality type.
Figure 3: Pairwise sensitivity analysis of loss weights.
Table 2: Sensitivity analysis of key hyperparameters.
MethodMAudioVideoFusionMismatch
FG-CLIP [ 30 ]V-only–70.30––
ACF (ours)V-only–71.06––
ImageBind [ 10 ]A-only47.42–––
ACF (ours)A-only57.45–––
CoC ∗ [ 16 ]A + V45.7471.0680.4370.21
MDL ∗ [ 22 ]A + V47.7371.9273.4070.43
Table 3: Comparison results. ∗ denotes our implemented baseline variant based on the cited method. The best results are highlighted in bold.
MethodAudioVideoFusion
Baseline45.9671.0682.98
w/ OFGM46.3871.9281.64
w/ ACAF57.6672.7784.89
Ours58.3072.7787.23
Table 4: Ablation study results.
Figure 4: t-SNE visualization of raw and ACF-enhanced video and audio features. ACF improves the intra-class compactness of audio features while preserving the discriminative structure of video features.
Figure 5: Attention visualization of the proposed framework. For each example, we show the input frame, attention overlay, and patch-level attention map.
More in this topic

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Explore similar work

CardsList
Jul 14, 2026

AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning

Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence before composing grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline capabilities, while sample-efficient reinforcement learning uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than increasing SFT data. Third, we introduce AVSCapBench, a benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.

Yanghai Wang, Jiahao Wang, Jiafu Tang +9

NJU-LINK Team, Nanjing University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences

Audio-Visual UnderstandingAudio-Visual AlignmentMultimodal Model Evaluation
AskPDF
Aug 29, 2026

Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.

Tianrui Hui, Shaofei Huang, Qisong Han +6

Hefei University of Technology Hefei, Anhui, China · Cardiff University Cardiff, Wales, United Kingdom · University of Macau Macau, China

Audio-Visual UnderstandingOpen-Vocabulary Semantic Segmentation
AskPDF
Oct 22, 2026

AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning

This study develops an audio-visual emotion recognition system that integrates automatic utterance segmentation with multimodal fusion. Unlike conventional methods that rely on manually annotated utterance boundaries, the proposed system automatically generates candidate segments from continuous dialogue and extracts the corresponding audio and visual features for emotion classification. To address the challenges introduced by automatic segmentation, the proposed AGF-Net integrates Audio-Guided Fusion (AGF), Multimodal Negative Learning (MNL), and Predictive Dynamic Fusion (PDF) to improve cross-modal fusion, modality balance, and robustness to varying modality quality. Experimental results show that AGF-Net achieves 64.92% Accuracy and 65.47% F1-score under clean conditions, 62.13% Accuracy and 63.08% F1-score under moderate noise, and 58.19% Accuracy and 59.41% F1-score under higher noise. Overall, the proposed framework reduces reliance on manually annotated utterance boundaries and maintains competitive performance under automatically segmented and noisy audio-visual inputs.

Chuan Han Li, Yun Chien Cheng

Department of Mechanical Engineering, College of Engineering, National Yang Ming Chiao Tung University, Hsin-Chu, Taiwan · Department of Electrical Engineering, College of Engineering, National Taiwan University, Taipei, Taiwan

Multimodal Emotion RecognitionMultimodal FusionMultimodal Learning
AskPDF