ML-driven detection and reduction of ballast information in multi-modal datasets
Organizations: Independent Researcher, Dublin, Ireland
Abstract
Modern datasets often contain ballast as redundant or low-utility information that increases dimensionality, storage requirements, and computational cost without contributing meaningful analytical value. This study introduces a generalized, multimodal framework for ballast detection and reduction across structured, semi-structured, unstructured, and sparse data types. Using diverse datasets, entropy, mutual information, Lasso, SHAP, PCA, topic modelling, and embedding analysis are applied to identify and eliminate ballast features. A novel Ballast Score is proposed to integrate these signals into a unified, cross-modal pruning strategy. Experimental results demonstrate that significant portions of the feature space as often exceeding 70% in sparse or semi-structured data, can be pruned with minimal or even improved classification performance, along with substantial reductions in training time and memory footprint. The framework reveals distinct ballast typologies (e.g. statistical, semantic, infrastructural), and offers practical guidance for leaner, more efficient machine learning pipelines.
Figures & tables
| Metric | Before (full) | LASSO ( ) | LASSO ( ) | SHAP |
|---|---|---|---|---|
| Features used | 432 | 105 | 317 | 69 |
| AUC | 0.9131 | 0.9109 | 0.9134 | 0.9147 |
| Precision | 0.8945 | 0.8849 | 0.8933 | 0.8920 |
| Recall ( ) | 0.7732 | 0.7768 | 0.7718 | 0.7718 |
| F1-score ( ) | 0.3422 | 0.3239 | 0.3393 | 0.3365 |
| Training time (s) | 94.15 | 33.93 | 82.79 | 35.72 |
| Method | #Features | AUC | Accuracy | Recall_Class1 | F1_Class1 | TrainTimeSec | Notes |
|---|---|---|---|---|---|---|---|
| Full Dataset | 432 | 0.91310 | 0.89445 | 0.77324 | 0.34216 | 119.54 | – |
| PCA Reduction | 23 | 0.99803 | 0.99377 | 0.84851 | 0.90579 | 31.70 | Low interpretability, latent ballast |
| SHAP Pruning | 69 | 0.91475 | 0.89195 | 0.77183 | 0.33651 | 35.72 | Best trade-off: compact&performant |
| Sparse Removal | 423 | 0.91341 | 0.89320 | 0.77324 | 0.33952 | 102.22 | Detects trivially flat features |
| LASSO Pruning | 105 | 0.91089 | 0.88490 | 0.77676 | 0.32394 | 33.93 | Best linear model-aware pruning |
| Correlation Removal | 116 | 0.90978 | 0.89325 | 0.77063 | 0.34044 | 20.73 | Addresses multicollinearity |
| Method | Dataset | Peak Metric (Value) | Threshold at Peak | Units Kept at Peak | Notes |
|---|---|---|---|---|---|
| LDA Coherence | CORD-19 | 0.3510 (Coherence) | 0.30 | 26,349 | Modest coherence increase followed by degradation beyond threshold 0.30 |
| PubLayNet | 0.1648 (Mean Coherence) | 0.25-0.85 | 14 | Stable plateau after 0.25 with no additional gain | |
| TextRank | CORD-19 | 0.9922 (Avg. Coherence) | 0.00 | 28,608 | Best performance without trimming; coherence degrades monotonically |
| PubLayNet | 1.5391 (Mean Rank Score) | 0.80-0.85 | 6 | Sharp improvement only at very high thresholds | |
| Embedding BERT | CORD-19 | 0.2653 (Semantic Similarity) | 0.45 | 15,792 | Peaks at mid-threshold, followed by gradual decline |
| PubLayNet | 0.9036 (Mean Similarity) | 0.85 | 2 | Activates only under aggressive pruning |
| Method | Ballast Type Detected | Description | Dataset Sensitivity |
|---|---|---|---|
| Entropy | Low-information tokens | Repeated headers, titles, and common layout blocks | Effective in both (stronger in PubLayNet) |
| Mutual Information (MI) | Statistically irrelevant text | Citations, method templates, and standard disclaimers | Effective in both (stronger in CORD-19) |
| TF-IDF | Lexically redundant terms | High-frequency but non-informative words | Effective in both |
| LDA | Topic-incoherent blocks | Clusters of incoherent or weakly defined topics (e.g., boilerplate methodological text) | Strong in CORD-19 |
| BERT Embeddings | Semantically similar segments | Sections with low novelty or semantically repetitive content | Strong in CORD-19 |
| TextRank | Positionally generic sentences | Frequently occurring mid-ranking or introductory statements with limited content value | Effective in both (especially textual data) |
| Method | Ballast Typology Detected | Affected Fields |
|---|---|---|
| Mutual Information (MI) | Statistical, Model-Irrelevant | price, main_cat., categories |
| Entropy | Statistical | description |
| TF-IDF | Redundant, Semantic | description, features |
| LDA | Semantic | description |
| BERT Similarity | Redundant | title |
| Clustering | Redundant, Noisy | title, features |
| Step | Description | Features Remaining |
|---|---|---|
| Raw Dataset | Initial feature set | 47 |
| Variance Threshold | Removed near-zero variance features | 46 |
| Correlation Filter | Dropped highly correlated features ( ) | 42 |
| Mutual Information | Retained features informative to the target variable | 18 |
| Entropy Filter | Confirmed relevance; no further reduction observed | 18 |
| LassoCV | Regularized linear feature selection | 14 |
| Aspect | LightGBM | CatBoost |
|---|---|---|
| AUC (All Features) | 0.74 | 0.80 |
| AUC (After Pruning) | 0.74 (best at 21-28 features) | 0.80-0.81 (best at 22 features) |
| F1 Score | 0.77 | 0.78 |
| Precision / Recall | 0.81 / 0.81 | 0.82 / 0.82 |
| SHAP Feature Stability | Moderate fluct. under pruning | Stable up to 50% feature removal |
| Feature Importance Spread | Sharp drop-off after top-ranked features | More inform., smoother top-feature distrib. |
| Ballast Type | Description | Indicators | Detection / Removal Methods |
|---|---|---|---|
| High-sparsity features | Features with extreme missing or zero values | Sparsity 70–90%; many nulls or default encodings | Sparsity Filter, Entropy Filter |
| Low-variance features | Features with nearly constant values across records | Variance near zero; no discriminative power | VarianceThreshold |
| Redundant features | Features highly correlated with others (linear or monotonic) | Pearson/Spearman | Correlation Filter |
| Low mutual information | Features weakly associated with target labels | MI 0.03-0.05 | Mutual Information (MI), Entropy–MI Scoring |
| Structural artifacts | System-generated identifiers or placeholders | Uniform patterns, IDs, spatial zones without predictive value | SHAP, Domain Filtering, Data Leakage Checks |
| Over-encoded categories | Sparse one-hot or multi-hot encodings of categorical fields | High dimensionality with sparse representation | PCA, Embedding, SHAP, Lasso |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Structured Dataset | Unstructured Dataset | Semi-Structured Dataset | Sparse Dataset |
|---|---|---|---|---|
| Mutual Information (MI) | Detected statistically irrelevant features with weak target signal (e.g., price, category); rejected approximately 80 features. | Effective in isolating boilerplate and citation-like structures in CORD-19; also flagged low-content layout regions in PubLayNet. | Highlighted weakly informative textual fields and categorical encodings; effective for numerical metadata. | Rejected fields with MI ; highly effective for dimensionality reduction while preserving predictive signal. |
| Entropy | Detected flat distributions; flagged features with uniform value spread (e.g., device information). | Exposed repeated tokens and layout blocks in PubLayNet; revealed document templates in both datasets. | Identified repeated tokens in descriptions and metadata; reinforced MI-based filtering. | Detected zero-variance and ultra-sparse columns; strong synergy with MI for structure-first filtering. |
| SHAP | Ranked features by model relevance (LightGBM); four features retained post-processing; best interpretability–performance trade-off. | Performed poorly on text-dominant data due to limited structural grounding; weak thematic alignment. | Weak correlation with semantic methods; struggled with high-dimensional sparse textual inputs. | Final reduction stage; retained top four predictors based on SHAP values; excellent interpretability. |
| Lasso | Efficient linear pruning; eliminated zero-contributing fields (105 14 features). | Missed nonlinear ballast in free-text; underperformed compared to semantic detectors such as LDA and BERT. | Detected structurally redundant fields; insensitive to lexical overlap; effective for numerical variables. | Effective after MI and entropy filtering; removed redundant encodings and noise features. |
| PCA / t-SNE | Filtered latent ballast; achieved best AUC (0.998) but sacrificed interpretability (432 23 features). | Limited explainability for text; captured both noise and signal; effective only for embedding-based representations. | Detected over-encoded sparse fields; useful for visualizing low-dimensional semantic clusters. | Detected latent sparsity but removed interpretable signals; unsuitable for end-to-end model explanation. |
| Correlation Filter | Mitigated multicollinearity (432 116 features) with minimal performance loss. | Not applicable to raw high-dimensional text without embeddings. | Detected duplicated structured metadata fields; ineffective for free-text descriptions. | Removed redundant population statistics using Spearman ; foundational pipeline step. |
| Ballast Type | Detection Methods | Description | Effective In |
|---|---|---|---|
| Statistical Ballast | Entropy, Mutual Information (MI), Variance Threshold | Low-variability or uniform features with minimal discriminative power. | Structured, Sparse, Semi-Structured |
| Redundancy Ballast | Correlation Filter, Lasso, SHAP, PCA, IoU | Features duplicating others either numerically or structurally, such as collinearity or templated layouts. | Structured, Sparse, Unstructured |
| Semantic Ballast | TF-IDF, LDA (Topic Modelling), BERT Similarity, Clustering (K-Means, DBSCAN) | Lexically or thematically non-informative components, including boilerplate text, common phrases, and repetitive segments. | Unstructured, Semi-Structured |
| Model-Irrelevant Ballast | SHAP, Lasso, Information Bottleneck | Features exhibiting negligible contribution to model outputs, such as near-zero coefficients or low SHAP values. | Structured, Sparse, Semi-Structured |
Feature Selection
Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.
See all 194 papers