Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning

Filippo Cenacchi

cs.LGMar 26, 2026

Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning

Authors: Filippo Cenacchi

Organizations: School of Computing, Macquarie University, Sydney, Australia

Abstract

Predictive systems increasingly span heterogeneous modalities such as graphs, language, and tabular records, but sparsity and efficiency remain modality-specific (graph edge or neighborhood sparsification, Transformer head or layer pruning, and separate tabular feature-selection pipelines). This fragmentation makes results hard to compare, complicates deployment, and weakens reliability analysis across end-to-end KDD pipelines. A unified sparsification primitive would make accuracy-efficiency trade-offs comparable across modalities and enable controlled reliability analysis under representation compression. We ask whether a single representation-level mechanism can yield comparable accuracy-efficiency trade-offs across modalities while preserving or improving probability calibration. We propose L0-Gated Cross-Modality Learning (L0GM), a modality-agnostic, feature-wise hard-concrete gating framework that enforces L0-style sparsity directly on learned representations. L0GM attaches hard-concrete stochastic gates to each modality's classifier-facing interface: node embeddings (GNNs), pooled sequence embeddings such as CLS (Transformers), and learned tabular embedding vectors (tabular models). This yields end-to-end trainable sparsification with an explicit control knob for the active feature fraction. To stabilize optimization and make trade-offs interpretable, we introduce an L0-annealing schedule that induces clear accuracy-sparsity Pareto frontiers. Across three public benchmarks (ogbn-products, Adult, IMDB), L0GM achieves competitive predictive performance while activating fewer representation dimensions, and it reduces Expected Calibration Error (ECE) in our evaluation. Overall, L0GM establishes a modality-agnostic, reproducible sparsification primitive that supports comparable accuracy, efficiency, and calibration trade-off analysis across heterogeneous modalities.

Figures & tables

Notation	Definition
$X$	input features (tabular: fields; graph: node features; text: tokens)
$\hat{y},\,y$	predicted probability, ground-truth label
$r$	modality interface representation (embedding / node rep / pooled text rep)
$z$	binary gate (hard-concrete relaxation in training)
$\tilde{r}=z\odot r$	gated representation ( $\odot$ : elementwise product)
$\lambda$	sparsity-control weight (higher $\lambda\Rightarrow$ fewer active dims)

Table 1. Summary of main notations.

Figure 1. L0GM overview vs. modality-specific sparsification. Unlike prior work that sparsifies different modality-specific objects, L0GM applies a single hard-concrete $L_{0}$ gate at each modality’s representation interface, controlled by one parameter $\lambda$ .

Property	Adult	IMDB	ogbn-products
Modality	Tabular	Text	Graph
Task	Income ( ${>}\$ 50$ K) cls.	Sentiment cls.	Node cls. (category)
Instances / Nodes	48,842	50,000	2,449,029
Edges	–	–	61,859,140
Classes	2	2	47
Input features	14 (mixed)	raw text	100-d node feats

Table 2. Summary of experimental datasets.

Q1: Standalone (module validity)						Q2: Integrated (system benefit)
Mod.	Method	Acc/AUC $\uparrow$	Worst (seed) $\uparrow$	ECE $\downarrow$	Rob $\mu$ $\uparrow$	Mod.	Method	Acc/AUC $\uparrow$	Worst (seed) $\uparrow$	ECE $\downarrow$	Rob $\mu$ $\uparrow$	Latency( $\mu$ s) $\downarrow$
Tab	MLP	0.8407	0.7703	0.0954	0.7926	Tab	MLP	0.8289	0.7426	0.1123	0.7768	6.83
	XGBoost	0.8705	0.7804	0.0113	0.8207		XGBoost	0.8667	0.7709	0.0146	0.8095	1.93
	LightGBM	0.8606	0.7802	0.0431	0.8178		LightGBM	0.8548	0.7684	0.0497	0.8068	2.67
	CatBoost	0.8704	0.7801	0.0067	0.8199		CatBoost	0.8659	0.7723	0.0094	0.8092	3.47
	Ours (L0GM)	0.8906	0.8702	0.0107	0.8714		Ours (L0GM)	0.8887	0.8668	0.0124	0.8696	1.24

Table 3. Best-in-column values are highlighted. Q1 measures predictive quality + calibration + robustness; Q2 adds latency. We report end-to-end forward-pass latency averaged over

R

runs after

W

warmup iterations, with fixed batch size. Worst denotes the minimum i.i.d. performance across seeds:

\mathrm{Worst}=\min_{s\in\mathcal{S}}m(s,\text{i.i.d.})

. Rob

\mu

denotes the mean performance under the robustness protocol:

\mathrm{Rob}\ \mu=\frac{1}{|\mathcal{S}|\cdot|\mathcal{P}|}\sum_{s\in\mathcal{S}}\sum_{p\in\mathcal{P}}m(s,p)

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

torch	cuda	gpu	cudnn
2.5.1+cu121	12.1	NVIDIA RTX 6000 Ada Generation	90100

Appendix

Table 4. Environment snapshot for reproducibility.

Notation	Definition
$\sigma(\cdot)$	sigmoid / activation function
$f(\cdot)$	predictor / classification head
$\theta,\Theta$	parameters (module / all parameters)
$m$	number of tabular fields
$d$	embedding dimension per field
$e_{i}$	embedding of field $i$

Appendix

Table 5. Full list of notations (supplement to Table 1 ).

Variant	Adult			IMDB			ogbn-prod
Variant	Metric $\uparrow$	Active% $\downarrow$	ECE $\downarrow$	Metric $\uparrow$	Active% $\downarrow$	ECE $\downarrow$	Metric $\uparrow$	Active% $\downarrow$	ECE $\downarrow$
Dense backbone (No Gate)	0.8729	100.0	0.0247	0.9318	100.0	0.0216	0.6893	100.0	0.1167
Static/global mask (non-conditional)	0.8696	19.7	0.0209	0.9284	21.3	0.0207	0.6768	24.6	0.1049
L0 gate (interface-aligned, default)	0.8794	12.5	0.0158	0.9356	11.8	0.0189	0.7017	15.2	0.0716
Fine-grained gate (overly granular)	0.8657	16.4	0.0296	0.9261	14.9	0.0313	0.6712	18.7	0.1124
No annealing / fixed- $\lambda$ schedule	0.8619	13.1	0.0417	0.9227	12.4	0.0398	0.6668	15.9	0.1396

Appendix

Table 6. Ablation study on L0GM design choices. We fix the backbone per modality and vary only the gating primitive and its attachment/granularity, reporting predictive quality, active fraction (%), and reliability (ECE).

Hyperparameter	Search range / candidates
Learning rate (LR)	$\{1\mathrm{e}{-4},\,3\mathrm{e}{-4},\,1\mathrm{e}{-3}\}$
Weight decay	$\{0,\,1\mathrm{e}{-6},\,1\mathrm{e}{-5},\,1\mathrm{e}{-4}\}$
Dropout	$\{0.00,\,0.05,\,0.10,\,0.15\}$
Batch size	$\{64,\,128,\,256,\,512\}$ (hardware-limited)
Gate penalty $\lambda$	log-grid over $[10^{-4},\,10^{-2}]$ (e.g., 5–9 points)
$\ell_{0}$ annealing schedule	warmup $\in[0.05,\,0.20]$ ; end-temp $\in[0.5,\,2.0]$

Appendix

Table 7. Hyperparameter search space (L0GM). Ranges are modality-agnostic; modality-specific fields are marked when applicable.

Mod.	$\lambda$	Active% $\downarrow$	Acc/AUC $\uparrow$	Worst $\uparrow$	ECE $\downarrow$
Tab	$10^{-4}$	100.0	0.890	0.870	0.018
	$3\cdot 10^{-4}$	35.0	0.889	0.868	0.012
	$10^{-3}$	15.0	0.887	0.866	0.010
	$3\cdot 10^{-3}$	7.0	0.881	0.852	0.019
Text	$10^{-4}$	100.0	0.933	0.143	0.024
	$3\cdot 10^{-4}$	30.0	0.932	0.143	0.020

Appendix

Table 8. (F1)

\lambda

sweep. Active fraction decreases monotonically as

\lambda

increases, while ECE is typically non-monotonic (often improving at moderate sparsity and degrading under extreme sparsity).

Mod.	Variant	Acc/AUC $\uparrow$	Worst $\uparrow$	ECE $\downarrow$
Tab	No annealing (fixed)	0.862	0.842	0.042
Tab	With annealing (default)	0.889	0.867	0.012
Text	No annealing (fixed)	0.923	0.158	0.040
Text	With annealing (default)	0.933	0.143	0.020
Graph	No annealing (fixed)	0.667	0.651	0.140
Graph	With annealing (default)	0.724	0.708	0.033

Appendix

Table 9. (F2) Annealing ablation. Warm-up + gradual hardening stabilizes training (improves Worst) and tends to reduce ECE compared to fixed hardness / no annealing.

Mod.	Regime	$\lambda$	Active%	Acc/AUC	ECE (no coup.)	ECE (coup.)
Tab	Moderate	$10^{-3}$	15.0	0.887	0.010	0.010
Tab	High	$3\cdot 10^{-3}$	7.0	0.881	0.019	0.014
Text	Moderate	$10^{-3}$	12.0	0.930	0.017	0.017
Text	High	$3\cdot 10^{-3}$	6.0	0.922	0.026	0.021
Graph	Moderate	$10^{-3}$	15.0	0.721	0.028	0.028
Graph	High	$3\cdot 10^{-3}$	8.0	0.712	0.045	0.036

Appendix

Table 10. (F3) Calibration coupling. Coupling has little effect at moderate sparsity (ECE already improves), but can partially recover ECE under aggressive sparsity.

Activation Sparsity

Activation sparsity studies how neural networks can use mostly inactive units to reduce computation, memory, and energy while preserving useful representations.

See all 56 papers

Apr 5, 2026

Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

Learning interpretable multimodal representations inherently relies on uncovering the conditional dependencies between heterogeneous features. However, sparse graph estimation techniques, such as Graphical Lasso (GLasso), to visual-linguistic domains is severely bottlenecked by high-dimensional noise, modality misalignment, and the confounding of shared versus category-specific topologies. In this paper, we propose Cross-Modal Graphical Lasso (CM-GLasso) that overcomes these fundamental limitations. By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space. We introduce a cross-attention distillation mechanism that condenses high-dimensional patches into explicit semantic nodes, naturally extracting spatial-aware cross-modal priors. Furthermore, we unify tailored GLasso estimation and Common-Specific Structure Learning (CSSL) into a joint objective optimized via the Alternating Direction Method of Multiplier (ADMM). This formulation guarantees the simultaneous disentanglement of invariant and class-specific precision matrices without multi-step error accumulation. Extensive experiments across eight benchmarks covering both natural and medical domains demonstrate that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.

Fei Wang, Yutong Zhang, Xiong Wang

Department of Applied Mathematics and Statistics, Stony Brook University · School of Mathematics, Sichuan University · School of Computer Science and Technology, USTC

Multimodal Learning Multimodal Graph Learning Multimodal Disentangled Representation Learning

Ask PDF

Oct 19, 2026

On Learning Expand-and-Sparsify Representations

Expand-and-sparsify (

\texttt{EaS}

) representations lift an input to a much higher dimension through a linear map and keep only the

k

largest coordinates. Existing theory builds that map at random and needs the expanded dimension to grow exponentially in the input dimension, which puts the construction out of practical reach. We ask what is gained by learning the map by gradient descent instead, and how that gain depends on how wide the expansion is. We sweep

73

OpenML classification tasks and

48

regression tasks under a protocol that holds the datasets, expanded dimensions, sparsity levels and splits fixed across every method, and we summarise each comparison by a median over datasets and a Wilcoxon signed rank test. We report four findings. First, learning the expansion helps most when the expansion is narrowest, and its advantage over a random map decays to nothing by the widest expansions we run, so a learned

\texttt{EaS}

layer can be made small. Second, with the map learned, a top-

k

representation that keeps at most a tenth of its coordinates is indistinguishable from a dense

\texttt{ReLU}

layer of the same expanded dimension at every expansion factor we test, while GELU and SiLU are not. Third, when the representation is forced to be binary the sparse code is the better of the two, and the reason is that binarization costs it nothing while it costs the dense layer accuracy. Fourth, how sparse the representation can be is governed by how wide it is: the sparsity level that is free rises as the expansion grows, so there is no fixed limit on how sparse it can be. The regression tasks reproduce the first finding and qualify the second and third.

Alireza Asghari, Parikshit Ram, Kaushik Sinha

Wichita State University · IBM Research

Topic pending

Ask PDF

Mar 12, 2026

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

Structured pruning is widely applied to compress large language models (LLMs), but its performance depends heavily on how neuron importance is estimated. Most existing methods rely on activation statistics from a single calibration set, which introduces calibration bias and degrades downstream cross-task generalization. We observe that neurons exhibit heterogeneous distribution sensitivity, ranging from maintaining relatively stable rankings across calibration datasets to showing substantially larger cross-dataset variation. Ignoring this heterogeneity, existing methods rank all neurons in shared spaces with a uniform scoring source, so calibration-specific neurons dominate the ranking and weakly-activated neurons are scored unreliably. To address this, we propose GPrune-LLM, a structured pruning framework that reduces calibration bias by measuring and exploiting the cross-distribution behavior of neurons for fair comparison. Specifically, we restructure the neuron ranking space into behavior-consistent local spaces, adapt the scoring source where the calibration signal is unreliable, and learn per-module sparsity allocation under a global budget. Experiments across multiple models and downstream tasks show that GPrune-LLM improves the generalization of its base pruning metrics, with gains most pronounced at high sparsity, and reduces dependence on the choice of importance metric.

Xiaoyun Liu, Divya Saxena, Jiannong Cao +3

Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India

LLM Pruning Structured Pruning Neural Network Pruning LLM Compression Neural Network Generalization

Ask PDF

Abstract

Activation Sparsity

Explore similar work

Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

On Learning Expand-and-Sparsify Representations

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models