ML-driven detection and reduction of ballast information in multi-modal datasets

Yaroslav Solovko

cs.LGFeb 18, 2026

ML-driven detection and reduction of ballast information in multi-modal datasets

Authors: Yaroslav Solovko

Organizations: Independent Researcher, Dublin, Ireland

Abstract

Modern datasets often contain ballast as redundant or low-utility information that increases dimensionality, storage requirements, and computational cost without contributing meaningful analytical value. This study introduces a generalized, multimodal framework for ballast detection and reduction across structured, semi-structured, unstructured, and sparse data types. Using diverse datasets, entropy, mutual information, Lasso, SHAP, PCA, topic modelling, and embedding analysis are applied to identify and eliminate ballast features. A novel Ballast Score is proposed to integrate these signals into a unified, cross-modal pruning strategy. Experimental results demonstrate that significant portions of the feature space as often exceeding 70% in sparse or semi-structured data, can be pruned with minimal or even improved classification performance, along with substantial reductions in training time and memory footprint. The framework reveals distinct ballast typologies (e.g. statistical, semantic, infrastructural), and offers practical guidance for leaner, more efficient machine learning pipelines.

Figures & tables

Fig. 1: Shannon Entropy distribution of numeric features.

Fig. 3: PCA Scatterplot of Transaction Data.

Fig. 4: Top 50 Feature Correlation Heatmap.

Metric	Before (full)	LASSO ( $C=0.01$ )	LASSO ( $C=0.1$ )	SHAP
Features used	432	105	317	69
AUC	0.9131	0.9109	0.9134	0.9147
Precision	0.8945	0.8849	0.8933	0.8920
Recall ( $isFraud$ )	0.7732	0.7768	0.7718	0.7718
F1-score ( $isFraud$ )	0.3422	0.3239	0.3393	0.3365
Training time (s)	94.15	33.93	82.79	35.72

TABLE I: Performance comparison of SHAP and LASSO feature reduction methods

Method	#Features	AUC	Accuracy	Recall_Class1	F1_Class1	TrainTimeSec	Notes
Full Dataset	432	0.91310	0.89445	0.77324	0.34216	119.54	–
PCA Reduction	23	0.99803	0.99377	0.84851	0.90579	31.70	Low interpretability, latent ballast
SHAP Pruning	69	0.91475	0.89195	0.77183	0.33651	35.72	Best trade-off: compact&performant
Sparse Removal	423	0.91341	0.89320	0.77324	0.33952	102.22	Detects trivially flat features
LASSO Pruning	105	0.91089	0.88490	0.77676	0.32394	33.93	Best linear model-aware pruning
Correlation Removal	116	0.90978	0.89325	0.77063	0.34044	20.73	Addresses multicollinearity

TABLE II: Full comparative analysis of all seven ballast reduction methods against the full dataset

Fig. 5: Feature Overlap Between SHAP and Lasso Selections.

Fig. 6: CORD-19 Ballast Pruning: AUC vs. Percentage of Features Dropped.

Fig. 7: PublayNet ballast pruning: AUC vs. percentage of features dropped.

Fig. 8: Normalized MI vs Entropy (CORD-19).

Fig. 9: Boxplot of ballast scores for retained versus dropped features in the CORD-19 dataset.

Fig. 10: Normalized mutual information versus entropy for PubLayNet features.

Fig. 11: Boxplot of Ballast Scores for retained versus dropped features in the PubLayNet dataset.

Method	Dataset	Peak Metric (Value)	Threshold at Peak	Units Kept at Peak	Notes
LDA Coherence	CORD-19	0.3510 (Coherence)	0.30	26,349	Modest coherence increase followed by degradation beyond threshold 0.30
	PubLayNet	0.1648 (Mean Coherence)	0.25-0.85	14	Stable plateau after 0.25 with no additional gain
TextRank	CORD-19	0.9922 (Avg. Coherence)	0.00	28,608	Best performance without trimming; coherence degrades monotonically
	PubLayNet	1.5391 (Mean Rank Score)	0.80-0.85	6	Sharp improvement only at very high thresholds
Embedding BERT	CORD-19	0.2653 (Semantic Similarity)	0.45	15,792	Peaks at mid-threshold, followed by gradual decline
	PubLayNet	0.9036 (Mean Similarity)	0.85	2	Activates only under aggressive pruning

TABLE III: Comparative evaluation of ballast detection techniques across datasets

Method	Ballast Type Detected	Description	Dataset Sensitivity
Entropy	Low-information tokens	Repeated headers, titles, and common layout blocks	Effective in both (stronger in PubLayNet)
Mutual Information (MI)	Statistically irrelevant text	Citations, method templates, and standard disclaimers	Effective in both (stronger in CORD-19)
TF-IDF	Lexically redundant terms	High-frequency but non-informative words	Effective in both
LDA	Topic-incoherent blocks	Clusters of incoherent or weakly defined topics (e.g., boilerplate methodological text)	Strong in CORD-19
BERT Embeddings	Semantically similar segments	Sections with low novelty or semantically repetitive content	Strong in CORD-19
TextRank	Positionally generic sentences	Frequently occurring mid-ranking or introductory statements with limited content value	Effective in both (especially textual data)

TABLE IV: Typology of ballast information in unstructured data

Fig. 12: Distribution of entropy values for product descriptions in the Amazon Fashion dataset.

Fig. 13: Visual diagnostics of feature informativeness (top) and K-Means cluster distribution over title embeddings (bottom).

Fig. 14: Comparison of storage size for the TF-IDF vectorized “ description ” field using Dense and CSR formats.

Method	Ballast Typology Detected	Affected Fields
Mutual Information (MI)	Statistical, Model-Irrelevant	price, main_cat., categories
Entropy	Statistical	description
TF-IDF	Redundant, Semantic	description, features
LDA	Semantic	description
BERT Similarity	Redundant	title
Clustering	Redundant, Noisy	title, features

TABLE V: Summary of the method–typology alignment

Fig. 15: Entropy-MI Pruning vs. Model Performance.

Fig. 16: Scatter Plot: Normalized Entropy vs. MI

Fig. 17: TF-IDF term redundancy histogram.

Fig. 18: Multi-Signal Ballast Trade-off Curve.

Step	Description	Features Remaining
Raw Dataset	Initial feature set	47
Variance Threshold	Removed near-zero variance features	46
Correlation Filter	Dropped highly correlated features ( $r>0.95$ )	42
Mutual Information	Retained features informative to the target variable	18
Entropy Filter	Confirmed relevance; no further reduction observed	18
LassoCV	Regularized linear feature selection	14

TABLE VI: Feature reduction steps and remaining feature counts

Fig. 19: SHAP feature importance bar plot.

Fig. 20: Normalized entropy vs. normalized MI scatterplot, coloured by ballast score.

Fig. 21: Trade-off curve: performance vs ballast pruning threshold.

Fig. 22: Performance stability across ballast score thresholds between 0.80-0.89.

Fig. 23: Effect of PCA-based dimensionality reduction on Random Forest performance.

Fig. 24: Variance Threshold: Metrics vs. Threshold.

Fig. 25: Variance Threshold: Features Remaining vs. Threshold

Fig. 26: Correlation Filter - Metrics vs. Threshold

Fig. 27: Correlation Filter - Features Remaining vs. Threshold.

Aspect	LightGBM	CatBoost
AUC (All Features)	$\sim$ 0.74	$\sim$ 0.80
AUC (After Pruning)	$\sim$ 0.74 (best at 21-28 features)	$\sim$ 0.80-0.81 (best at $\sim$ 22 features)
F1 Score	$\sim$ 0.77	$\sim$ 0.78
Precision / Recall	$\sim$ 0.81 / $\sim$ 0.81	$\sim$ 0.82 / $\sim$ 0.82
SHAP Feature Stability	Moderate fluct. under pruning	Stable up to $\sim$ 50% feature removal
Feature Importance Spread	Sharp drop-off after top-ranked features	More inform., smoother top-feature distrib.

TABLE VII: Comparison of SHAP-based feature selection: LightGBM vs. CatBoost

Ballast Type	Description	Indicators	Detection / Removal Methods
High-sparsity features	Features with extreme missing or zero values	Sparsity $>$ 70–90%; many nulls or default encodings	Sparsity Filter, Entropy Filter
Low-variance features	Features with nearly constant values across records	Variance near zero; no discriminative power	VarianceThreshold
Redundant features	Features highly correlated with others (linear or monotonic)	Pearson/Spearman $\rho>0.95$	Correlation Filter
Low mutual information	Features weakly associated with target labels	MI $<$ 0.03-0.05	Mutual Information (MI), Entropy–MI Scoring
Structural artifacts	System-generated identifiers or placeholders	Uniform patterns, IDs, spatial zones without predictive value	SHAP, Domain Filtering, Data Leakage Checks
Over-encoded categories	Sparse one-hot or multi-hot encodings of categorical fields	High dimensionality with sparse representation	PCA, Embedding, SHAP, Lasso

TABLE VIII: Typology of ballast information in sparse dataset

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Structured Dataset	Unstructured Dataset	Semi-Structured Dataset	Sparse Dataset
Mutual Information (MI)	Detected statistically irrelevant features with weak target signal (e.g., price, category); rejected approximately 80 features.	Effective in isolating boilerplate and citation-like structures in CORD-19; also flagged low-content layout regions in PubLayNet.	Highlighted weakly informative textual fields and categorical encodings; effective for numerical metadata.	Rejected fields with MI $<0.03$ ; highly effective for dimensionality reduction while preserving predictive signal.
Entropy	Detected flat distributions; flagged features with uniform value spread (e.g., device information).	Exposed repeated tokens and layout blocks in PubLayNet; revealed document templates in both datasets.	Identified repeated tokens in descriptions and metadata; reinforced MI-based filtering.	Detected zero-variance and ultra-sparse columns; strong synergy with MI for structure-first filtering.
SHAP	Ranked features by model relevance (LightGBM); four features retained post-processing; best interpretability–performance trade-off.	Performed poorly on text-dominant data due to limited structural grounding; weak thematic alignment.	Weak correlation with semantic methods; struggled with high-dimensional sparse textual inputs.	Final reduction stage; retained top four predictors based on SHAP values; excellent interpretability.
Lasso	Efficient linear pruning; eliminated zero-contributing fields (105 $\rightarrow$ 14 features).	Missed nonlinear ballast in free-text; underperformed compared to semantic detectors such as LDA and BERT.	Detected structurally redundant fields; insensitive to lexical overlap; effective for numerical variables.	Effective after MI and entropy filtering; removed redundant encodings and noise features.
PCA / t-SNE	Filtered latent ballast; achieved best AUC (0.998) but sacrificed interpretability (432 $\rightarrow$ 23 features).	Limited explainability for text; captured both noise and signal; effective only for embedding-based representations.	Detected over-encoded sparse fields; useful for visualizing low-dimensional semantic clusters.	Detected latent sparsity but removed interpretable signals; unsuitable for end-to-end model explanation.
Correlation Filter	Mitigated multicollinearity (432 $\rightarrow$ 116 features) with minimal performance loss.	Not applicable to raw high-dimensional text without embeddings.	Detected duplicated structured metadata fields; ineffective for free-text descriptions.	Removed redundant population statistics using Spearman $\rho>0.95$ ; foundational pipeline step.

Appendix

TABLE IX: Comparative effectiveness of ballast detection methods across data modalities

Ballast Type	Detection Methods	Description	Effective In
Statistical Ballast	Entropy, Mutual Information (MI), Variance Threshold	Low-variability or uniform features with minimal discriminative power.	Structured, Sparse, Semi-Structured
Redundancy Ballast	Correlation Filter, Lasso, SHAP, PCA, IoU	Features duplicating others either numerically or structurally, such as collinearity or templated layouts.	Structured, Sparse, Unstructured
Semantic Ballast	TF-IDF, LDA (Topic Modelling), BERT Similarity, Clustering (K-Means, DBSCAN)	Lexically or thematically non-informative components, including boilerplate text, common phrases, and repetitive segments.	Unstructured, Semi-Structured
Model-Irrelevant Ballast	SHAP, Lasso, Information Bottleneck	Features exhibiting negligible contribution to model outputs, such as near-zero coefficients or low SHAP values.	Structured, Sparse, Semi-Structured

Appendix

TABLE X: Condensed ballast typology and corresponding detection methods

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 194 papers

AI-Based On-Board Maritime Object Detection for Earth Observation Payload Data Reduction on Versal Embedded Hardware
Oct 8, 2026Thomas Goudemant, Aurélien Bobey, Omar Hlimi +1Remote Sensing Maritime Surveillance
IRT Saint Exupéry
RL-Ballast: Ship Ballast Water Path Planning and Clog Prediction via Reinforcement Learning
Jul 6, 2026Ming-Kuan Lin, Yi-Chung Lai, Ming-Hsin Chiang +2RL Control Reinforcement Learning
Department of Electrical Engineering, National Taiwan Ocean University, Keelung 20224, Taiwan · AI Research Center · Shinsoft Corporation, Taiwan +1
Data Pruning: Redundant, Problematic, and Interdependent Samples
Jun 20, 2026Leon Freese, Marthinus W. TheunissenDataset Pruning Training Data Selection
Faculty of Engineering, North-West University, South Africa · Centre for Artificial Intelligence Research (CAIR), South Africa · National Institute for Theoretical and Computational Sciences (NITheCS), South Africa

Oct 8, 2026J/K move · Enter open · S save

AI-Based On-Board Maritime Object Detection for Earth Observation Payload Data Reduction on Versal Embedded Hardware

Thomas Goudemant, Aurélien Bobey, Omar Hlimi, Marjorie Bellizzi

IRT Saint Exupéry

Remote Sensing Maritime Surveillance Efficient Object Detection

Open paper AskPDF arXiv

Very-high-resolution Earth-observation satellites acquire more data than they can store and downlink, while in maritime surveillance the vessels cover a tiny fraction of each scene. We study onboard vessel detection as a way to select what is downlinked, which reduces the data according to its content rather than coding every pixel; it is complementary to conventional onboard compression. The work follows three axes. (i) Data and algorithm: a controlled dataset is generated from 68 annotated Maxar scenes with 43 vessel classes, and a YOLOX-S detector is trained on it. (ii) Embedded deployment: the detector is quantized and deployed on the DPU of a Versal VC1902, with a limited loss of detection quality and a processing time of a few seconds per scene. (iii) Data reduction: we propose several downlink modes, from metadata only (box, class and score of each detection) to image crops around vessels, tiles holding detections, or the whole scene with a degraded background, and estimate from the measured detection errors the trade-off each offers between the vessels kept and the volume downlinked. On our dense harbor and coastal scenes, tiles keep 98% of the vessels with 29% of the scene volume, and crops 83% with 3%.

Abstract

Feature Selection

Explore similar work

AI-Based On-Board Maritime Object Detection for Earth Observation Payload Data Reduction on Versal Embedded Hardware

RL-Ballast: Ship Ballast Water Path Planning and Clog Prediction via Reinforcement Learning

Data Pruning: Redundant, Problematic, and Interdependent Samples