HMMF-Net: A Lightweight Hierarchical Multi-Frame Multimodal Fusion Network for Accurate Fish Feeding Intensity Assessment
Organizations: AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Vietnam
Abstract
Automated fish feeding intensity assessment (FFIA) is important for precision aquaculture and feed cost optimization. However, unimodal methods are vulnerable to acoustic noise and visual disturbances, while existing multimodal frameworks often rely on computationally intensive 3D convolutions or Transformers, limiting their use on edge devices. To address these limitations, we propose HMMF-Net, a lightweight multimodal architecture that combines early multi-frame visual encoding and multi-axis acoustic modeling through a hierarchical bidirectional messenger cross-modal fusion (BMCF) mechanism. The visual branch captures temporal variations in feeding behavior by stacking temporally distant frames as a multi-channel input to an adapted 2D EfficientNetB0 backbone. The acoustic branch employs adaptive multi-axis residual units (AMARU) to extract temporal, frequency, and local time-frequency representations from log-mel spectrograms. BMCF progressively exchanges information between the two modalities at intermediate representation levels through compact messenger tokens, reducing the computational overhead of dense cross-modal attention. Evaluated on the AV-FFIA dataset using five-fold cross-validation, HMMF-Net achieves 96.77% accuracy and 96.88% F1-score on the held-out test folds, with 3.987M parameters and 1.002 GFLOPs. Compared with the best-performing visual configuration (95.49% accuracy), HMMF-Net reduces the classification error by 28.38%. Deployment on an NVIDIA Jetson Orin Nano using TensorRT with FP32 precision maintains comparable accuracy, with an inference time of 11.54 ms per 2-second sample and a model size of 18.81 MB. These results demonstrate that HMMF-Net combines effective multimodal representation learning with computational efficiency, enabling accurate and practical FFIA on resource-constrained edge platforms.
Figures & tables
| Stage | Operator | Output size |
| Stem | convolution, , stride 2 | |
| Stage 1 | MBConv1, , | |
| Stage 2 | MBConv6, , | |
| Stage 3 | MBConv6, , | |
| Stage 4 | MBConv6, , | |
| MBConv6, , |
| Block | Operator | Repeats | Channel configuration | Output size |
| Block 1 | AMARU | 2 | ||
| average pooling (stride 2) | 1 | – | ||
| Block 2 | AMARU | 2 | ||
| average pooling (stride 2) | 1 | – |
| FFI | Description |
| Strong | Fish move freely between food items and consume all the available food |
| Medium | Fish move to take the food, but return to their original positions |
| Weak | Fish eat only pellets that fall directly in front of them but do not move to take food |
| None | Fish do not respond to food |
| Experiment | Preprocessing & augmentation | Training configurations |
| Acoustic | Preprocessing: Downsampling to 64 kHz STFT (Hann window = 2048, hop = 1024) Mel filter banks (128 bins) Augmentation: SpecAugment Frequency masking (max width = 8, stripes = 2) Time masking (max width = 64, stripes = 2) | Epochs: 400 Batch size: 256 Optimizer: Adam Learning rate: Loss: Cross-entropy |
| Visual 2D (single-frame) | Preprocessing: Resize to Augmentation: Color jitter (brightness = 0.15) Random horizontal flip ( ) Random rotation ( ) Random affine (translation = 0.2) | ImageNet pre-training Epochs: 250 Batch size: 256 Optimizer: Adam Learning rate: Loss: Cross-entropy Patience: 40 |
| Visual 2D (multi-frame) | Preprocessing: Resize to Augmentation: Color jitter (brightness = 0.15) Random horizontal flip ( ) Random rotation ( ) Random affine (translation = 0.2) | ImageNet pre-training Epochs: 250 Batch size: 256 Optimizer: Adam Learning rate: Loss: Cross-entropy Patience: 40 |
| Visual 3D (multi-frame) | Preprocessing: 20-frame sequence extraction S3D, I3D, X3D-M: resize to X3D-XS: resize to Augmentation: Color jitter (brightness = 0.15) Random horizontal flip ( ) Random rotation ( ) Random affine (translation = 0.2) | ImageNet pre-training Epochs: 40 Batch size: 16 Optimizer: Adam Learning rate: Loss: Cross-entropy |
| Multimodal | Visual preprocessing: Resize to Visual augmentation: Color jitter (brightness = 0.15) Random horizontal flip ( ) Random rotation ( ) Random affine (translation = 0.2) Acoustic preprocessing: Downsampling to 64 kHz STFT (Hann window = 2048, hop = 1024) Mel filter banks (128 bins) Acoustic augmentation: SpecAugment Frequency masking (max width = 8, stripes = 2) Time masking (max width = 64, stripes = 2) | Epochs: 400 Batch size: 256 Optimizer: Adam Learning rate: Loss: Cross-entropy Patience: 50 |
| Model | Accuracy | Precision | Recall | F1-score |
| MobileNetV2 ( Sandler et al., 2019 ) | ||||
| EfficientNetB0 ( Tan and Le, 2020 ) | ||||
| DenseNet121 ( Huang et al., 2018 ) | ||||
| ResNet18 ( He et al., 2015 ) | ||||
| ConvNeXt-Tiny ( Liu et al., 2022 ) | ||||
| Swin-Tiny ( Liu et al., 2021 ) |
| Model | Frame policy | Accuracy | Precision | Recall | F1-score | Params (M) | GFLOPs |
| MobileNetV2 | Single frame 49 (baseline) | 2.229 | 0.652 | ||||
| Multi-frame (0, 49) | |||||||
| Multi-frame (12, 49) | 2.230 | 0.674 | |||||
| Multi-frame (24, 49) | 2.230 | 0.674 | |||||
| Multi-frame (12, 24, 49) | 2.231 | 0.696 | |||||
| Multi-frame (12, 24, 37, 49) | 2.232 | 0.717 |
| Model | Accuracy | Precision | Recall | F1-score | Params (M) | GFLOPs |
| S3D ( Xie et al., 2018 ) | 7.914 | 45.231 | ||||
| I3D ( Carreira and Zisserman, 2018 ) | 27.232 | 143.341 | ||||
| X3D-M ( Feichtenhofer, 2020 ) | 2.983 | 12.655 | ||||
| X3D-XS ( Feichtenhofer, 2020 ) | 2.983 | 6.482 |
| Model | Accuracy | Precision | Recall | F1-score | Params (M) | GFLOPs |
| Proposed AMARU-Net | 0.238 | 0.193 | ||||
| MobileNetV1 ( Howard et al., 2017 ) | 4.260 | 0.397 | ||||
| MobileNetV2 ( Sandler et al., 2019 ) | 3.539 | 0.195 | ||||
| EfficientNetB0 ( Tan and Le, 2020 ) | 4.012 | 0.266 | ||||
| ResNet22 ( Kong et al., 2020 ) | 62.603 | 7.209 | ||||
| PANNs CNN14 ( Kong et al., 2020 ) | 79.682 | 10.328 |
| Model | Sampling rate (kHz) | Accuracy | Precision | Recall | F1-score |
| AMARU-Net | 48 | ||||
| 64 | |||||
| 128 | |||||
| PANNs CNN6 | 48 | ||||
| 64 | |||||
| 128 |
| Configuration | Hierarchical fusion stages (BMCF) | Accuracy (%) | Precision (%) | Recall (%) | F1-score (%) | Params (M) | GFLOPs | ||||
| Stage 1 | Stage 2 | Stage 3 | Stage 4 | Stage 5 | |||||||
| Feature concat (baseline) | 95.88 0.22 | 95.98 0.21 | 96.05 0.18 | 96.01 0.19 | 3.835 | 1.002 | |||||
| + Stage 5 | ✓ | 96.01 0.24 | 96.11 0.21 | 96.20 0.26 | 96.15 0.24 | 3.895 | 1.002 | ||||
| + Stages 4–5 | ✓ | ✓ | 96.20 0.33 | 96.30 0.33 | 96.37 0.34 | 96.33 0.33 | 3.927 | 1.002 | |||
| + Stages 3–5 | ✓ | ✓ | ✓ | 96.40 0.28 | 96.50 0.26 | 96.55 0.30 | 96.53 0.28 | 3.950 | 1.002 | ||
| + Stages 2–5 (proposed) | ✓ | ✓ | ✓ | ✓ | 96.81 0.21 | 96.89 0.19 | 96.95 0.20 | 96.92 0.20 | 3.987 | 1.002 | |
| Fusion strategy | Fusion method | Acoustic model | Visual model | Accuracy | Params (M) | GFLOPs |
| Late fusion | Probability averaging | PANNs CNN6 | EfficientNetB0 | 8.583 | 5.951 | |
| ConvNeXt-Tiny | 32.397 | 14.085 | ||||
| Swin-Tiny | 32.096 | 11.085 | ||||
| AMARU-Net | EfficientNetB0 | 4.252 | 1.042 | |||
| Intermediate fusion | Feature concatenation | PANNs CNN6 | EfficientNetB0 | 9.504 | 5.953 | |
| ConvNeXt-Tiny | 33.056 | 14.059 |
| Accuracy | Precision | Recall | F1-score |
| Data preprocessing (ms) | Inference (ms) | Model size (MB) |
| 18.81 |
| Method | Accuracy (%) | Params (M) | GFLOPs | Dataset scope | Evaluation protocol | Deployment |
| U-FFIA ( Cui et al., 2025 ) | 90.70 | 21.626 | 15.488 | Full dataset | Holdout (8:1:1, 1 run) | NR |
| ACAF-Net ( Li et al., 2025 ) | 95.40 | 11.80 | NR | Full dataset | Holdout (8:1:1, 5 runs) | NR |
| LDMF ( Sha et al., 2026 ) | 98.22 | 7.10 | 6.80 | 6,016 samples (Filtered) | Holdout (8:1:1, 1 run) | NR |
| HMKD ( Du et al., 2026 ) | 97.47 0.61 | 6.33 | 12.51 | 26,843 samples (Filtered) | Holdout (8:1:1, 3 runs) | NR |
| Proposed framework | 96.77 0.27 | 3.987 | 1.002 | Full dataset | Cross-validation (5-fold) | Jetson Orin Nano |