UniqueShip: Mitigating Data Leakage in Acoustic Ship Classification Benchmark Datasets

Connor Hashemi; Trevor Stout; Anthony Hoogs; Jason Parham

cs.SDSep 12, 2026

UniqueShip: Mitigating Data Leakage in Acoustic Ship Classification Benchmark Datasets

Authors: Connor Hashemi, Trevor Stout, Anthony Hoogs, Jason Parham

Organizations: Kitware, Inc., 1712 Route 9, Suite 300, Clifton Park, NY 12065, USA

Abstract

Underwater Acoustic Target Recognition (UATR) of ships is well-suited for machine learning, yet its progress is hindered by the lack of large, diverse, and publicly available labeled datasets. In this work, we introduce UniqueShip, a machine learning-ready benchmark dataset for UATR applications sourced from the open Ocean Networks Canada (ONC) repository. Unlike previous datasets, we explicitly control for "data leakage" between the training and evaluation sets to ensure more reliable and generalizable model evaluation that does not encourage the model to memorize individual ships. We demonstrate that typical, random data partitioning in two prominent UATR datasets leads to falsely optimistic test performance, increasing accuracy by 10-48 percentage points compared to our more careful partitioning. Ablations on UniqueShip further show that doubling the number of unique vessels improves accuracy by 2.4-2.6 percentage points, while doubling total audio duration improves only by 0.8-1.3 points, indicating that vessel diversity should drive dataset curation more than total hours. We provide baselines with convolutional and transformer backbones, and analyze how ship metadata correlates with classification performance, finding that individual vessel characteristics predict classification difficulty far better than distance to the hydrophone alone. Overall, UniqueShip contains 2,460 hours of ship-radiated audio from 4,218 unique vessels (3,437 hours including background). We publish the dataset, code, and easy-to-download splits at uniqueshipdata.org to foster further UATR research.

Figures & tables

Fig. 1: We introduce UniqueShip, a large, labeled, and publicly-available Underwater Acoustic Target Recognition (UATR) dataset sourced from the Ocean Networks Canada (ONC) repository. (a) Our dataset enables robust classification model training. Unlike (b) existing datasets that suffer from data leakage due to random sampling, (c) UniqueShip provides a larger, more diverse pool of distinct recordings and strictly partitions data by Vessel ID (MMSI) to ensure no shared vessels exist across train, test, and validation splits.

Fig. 2: Class distribution of UniqueShip; 11 vessel classes, 1 background class, 4,218 unique vessels, 2,779 days, and nearly 2.5 million 5-sec. samples.

Fig. 3: Spectrograms (top) and Mel spectrograms (bottom) from UniqueShip 5 Class subset after preprocessing for input into baseline models (Sec. IV ).

Dataset	Config.	Trans.	Accuracy	F1
DeepShip [ 1 ]	No Leakage	STFT	$0.583\pm 0.091$	$0.548\pm 0.110$
	No Leakage	Mel	$0.588\pm 0.077$	$0.563\pm 0.084$
	Data Leakage	STFT	$0.685\pm 0.030$	$0.675\pm 0.031$
	Data Leakage	Mel	$0.710\pm 0.029$	$0.700\pm 0.031$
VTUAD (4km/6km) [ 2 ]	No Leakage	STFT	$0.345\pm 0.046$	$0.207\pm 0.083$
	No Leakage	Mel	$0.416\pm 0.069$	$0.279\pm 0.092$

TABLE II: Swin model results on prominent benchmark datasets.

Fig. 4: MMSI (top) and duration (bottom) ablation studies on UniqueShip subsets using the Swin backbone. Both vessel count (top) and audio duration (bottom) scale log-linearly with accuracy, but at different rates: 2.4–2.6 points per doubling of unique MMSIs against 0.8–1.3 per doubling of duration. Differing 100% base durations account for the offset baselines.

Config.	Backbone	Trans.	Accuracy	F1
No Leakage	MobileNet	STFT	$0.579\pm 0.028$	$0.566\pm 0.033$
	MobileNet	Mel	$0.617\pm 0.012$	$0.607\pm 0.018$
	ViT	STFT	$0.620\pm 0.026$	$0.608\pm 0.034$
	ViT	Mel	$\underline{0.650}\pm\underline{0.027}$	$\underline{0.640}\pm\underline{0.035}$
	Swin	STFT	$0.624\pm 0.022$	$0.614\pm 0.023$
	Swin	Mel	$\mathbf{0.665}\pm\mathbf{0.012}$	$\mathbf{0.656}\pm\mathbf{0.017}$

TABLE III: Baseline model test results on UniqueShip subset.

Fig. 5: SHAP beeswarms of the four main continuous metadata variables per ship class. A negative SHAP value means that the metadata feature reduces the primary model’s classification accuracy, while a positive value relates to improved classification accuracy of the sample.

	Run Configuration
Metadata Features	MMSI	No MMSI	No MMSI Mundlak
Draught	0.768	0.618	0.675
Distance	0.783	0.618	0.675
Speed	0.784	0.627	0.706
Length	0.763	0.678	0.690
Length + Speed	0.785	0.725	0.746

TABLE IV: Accuracy for predicting correct ship classification based only on metadata (baseline = 0.617).

Ship Type	Length	Speed	Distance	Draught
Cargo	0.52	0.22	0.32	0.17
Passengership	0.76	0.67	0.30	0.30
Tanker	0.60	0.42	0.17	0.38
Tug	0.36	0.36	0.31	0.26

TABLE V: Mean

|

SHAP

|

Values by Ship Type and Metadata Label

Fig. 6: Workflow of codebase repository.

Fig. 7: Temporal distribution of deployments [ 6 , 7 , 8 , 9 , 10 , 11 , 12 ] 5s audio samples.

Fig. 8: Confusion matrix across all samples in the “Main 5 - Bal.” split with the top performing model in Table III (Swin model with Mel spectrograms). The results are combined across the 5 different seeds.

	Run Configuration
Metadata Features	MMSI	No MMSI	No MMSI Mundlak
Draught	0.617	0.618	0.675
Distance	0.617	0.617	0.678
Speed	0.617	0.627	0.706
Length	0.617	0.678	0.690
Length + Speed	0.617	0.725	0.746

TABLE VI: Cross-Validation Accuracy (No leak) by Metadata and Run Configuration. The baseline accuracy for the model is 0.617.

Fig. 9: Pearson correlation coefficient of ship metadata based on the ship class, where a high absolute value (max 1.0) indicates strong correlation.

Fig. 10: Distribution of 4 vessel characteristic metadata features (distance, speed, draught, and length) across the main four vessel classes.

Fig. 11: Metadata fields (minus CTD information) which do not have full coverage in dataset. Note that while “ship name” records 100% in the figure, it is missing very few labels (near 99%). CTD is not included in this table, and the current version of the dataset does not fully extract CTD over all deployments.

Explore similar work

ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognition
Oct 14, 2026Zhe Cao, Jichao Zhang, Xiao-Lei ZhangClassification Audio Representation Learning
School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, 710072, China · Research and Development Institute of Northwestern Polytechnical University in Shenzhen, Shenzhen, 518063, China
Underwater Source Detection and Classification for Signal-based Surveillance: Audio Dataset Curation and Cross-Domain Evaluation
Jun 27, 2026Quoc Thinh Vo, David K. HanTraining Data Curation Audio Classification
Department of Electrical and Computer Engineering, Drexel University, Philadelphia, PA, USA
Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings
Jan 13, 2026Hilde I. Hummel, Sandjai Bhulai, Rob D. van der Mei +1Audio Representation Learning Linear Probing
Centre of Mathematics and Computer Science, Stochastics group, Science Park 123, Amsterdam, 1098 XG, Netherlands · Vrije Universiteit, Department of Mathematics, De Boelelaan 1111, Amsterdam, 1081 HV, Netherlands · Naturalis Biodiversity Center, Darwinweg 2, Leiden, 2333 CR, Netherlands

Oct 14, 2026J/K move · Enter open · S save

ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognition

Zhe Cao, Jichao Zhang, Xiao-Lei Zhang

School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, 710072, China · Research and Development Institute of Northwestern Polytechnical University in Shenzhen, Shenzhen, 518063, China

Classification Audio Representation Learning

Open paper AskPDF arXiv

Recent advances in deep learning have significantly improved underwater acoustic vessel recognition. However, existing publicly available datasets are typically limited in scale, hindering large-scale representation learning and downstream task evaluation. To address this limitation, we introduce ONCship, a large-scale underwater acoustic dataset comprising 4521 hours of curated recordings from Ocean Networks Canada (ONC) hydrophones and Automatic Identification System (AIS) data. We evaluate the proposed dataset across three tasks: self-supervised representation learning, vessel-type classification, and MMSI-Level Vessel Classification. Experiments on DeepShip and ShipsEar demonstrate the effectiveness of ONCship for acoustic representation learning, with joint pre-training on ONCship and AudioSet achieving a classification accuracy of 62.45% on DeepShip through linear probing. Furthermore, incorporating ONCship as auxiliary training data improves downstream vessel-type classification performance. Beyond conventional vessel-type classification, we further investigate MMSI-Level Vessel Classification, treating each vessel as a distinct identity. Experimental results show that models fine-tuned with different loss functions achieve Top-1 accuracies exceeding 53% and Top-5 accuracies approaching 69%, demonstrating the feasibility of distinguishing individual vessels through MMSI-Level Vessel Classification. Overall, ONCship provides a valuable resource for underwater acoustic representation learning and vessel recognition, while its rich vessel-level metadata enables further exploration of MMSI-Level Vessel Classification.

Abstract

Explore similar work

ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognition

Underwater Source Detection and Classification for Signal-based Surveillance: Audio Dataset Curation and Cross-Domain Evaluation

Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings