Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning

Filippo Cenacchi

cs.LGMar 26, 2026

Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning

Authors: Filippo Cenacchi

Organizations: School of Computing, Macquarie University, Sydney, Australia

Abstract

Predictive systems increasingly span heterogeneous modalities such as graphs, language, and tabular records, but sparsity and efficiency remain modality-specific (graph edge or neighborhood sparsification, Transformer head or layer pruning, and separate tabular feature-selection pipelines). This fragmentation makes results hard to compare, complicates deployment, and weakens reliability analysis across end-to-end KDD pipelines. A unified sparsification primitive would make accuracy-efficiency trade-offs comparable across modalities and enable controlled reliability analysis under representation compression. We ask whether a single representation-level mechanism can yield comparable accuracy-efficiency trade-offs across modalities while preserving or improving probability calibration. We propose L0-Gated Cross-Modality Learning (L0GM), a modality-agnostic, feature-wise hard-concrete gating framework that enforces L0-style sparsity directly on learned representations. L0GM attaches hard-concrete stochastic gates to each modality's classifier-facing interface: node embeddings (GNNs), pooled sequence embeddings such as CLS (Transformers), and learned tabular embedding vectors (tabular models). This yields end-to-end trainable sparsification with an explicit control knob for the active feature fraction. To stabilize optimization and make trade-offs interpretable, we introduce an L0-annealing schedule that induces clear accuracy-sparsity Pareto frontiers. Across three public benchmarks (ogbn-products, Adult, IMDB), L0GM achieves competitive predictive performance while activating fewer representation dimensions, and it reduces Expected Calibration Error (ECE) in our evaluation. Overall, L0GM establishes a modality-agnostic, reproducible sparsification primitive that supports comparable accuracy, efficiency, and calibration trade-off analysis across heterogeneous modalities.

Figures & tables

Notation	Definition
$X$	input features (tabular: fields; graph: node features; text: tokens)
$\hat{y},\,y$	predicted probability, ground-truth label
$r$	modality interface representation (embedding / node rep / pooled text rep)
$z$	binary gate (hard-concrete relaxation in training)
$\tilde{r}=z\odot r$	gated representation ( $\odot$ : elementwise product)
$\lambda$	sparsity-control weight (higher $\lambda\Rightarrow$ fewer active dims)

Table 1. Summary of main notations.

Figure 1. L0GM overview vs. modality-specific sparsification. Unlike prior work that sparsifies different modality-specific objects, L0GM applies a single hard-concrete $L_{0}$ gate at each modality’s representation interface, controlled by one parameter $\lambda$ .

Property	Adult	IMDB	ogbn-products
Modality	Tabular	Text	Graph
Task	Income ( ${>}\$ 50$ K) cls.	Sentiment cls.	Node cls. (category)
Instances / Nodes	48,842	50,000	2,449,029
Edges	–	–	61,859,140
Classes	2	2	47
Input features	14 (mixed)	raw text	100-d node feats

Table 2. Summary of experimental datasets.

Q1: Standalone (module validity)						Q2: Integrated (system benefit)
Mod.	Method	Acc/AUC $\uparrow$	Worst (seed) $\uparrow$	ECE $\downarrow$	Rob $\mu$ $\uparrow$	Mod.	Method	Acc/AUC $\uparrow$	Worst (seed) $\uparrow$	ECE $\downarrow$	Rob $\mu$ $\uparrow$	Latency( $\mu$ s) $\downarrow$
Tab	MLP	0.8407	0.7703	0.0954	0.7926	Tab	MLP	0.8289	0.7426	0.1123	0.7768	6.83
	XGBoost	0.8705	0.7804	0.0113	0.8207		XGBoost	0.8667	0.7709	0.0146	0.8095	1.93
	LightGBM	0.8606	0.7802	0.0431	0.8178		LightGBM	0.8548	0.7684	0.0497	0.8068	2.67
	CatBoost	0.8704	0.7801	0.0067	0.8199		CatBoost	0.8659	0.7723	0.0094	0.8092	3.47
	Ours (L0GM)	0.8906	0.8702	0.0107	0.8714		Ours (L0GM)	0.8887	0.8668	0.0124	0.8696	1.24

Table 3. Best-in-column values are highlighted. Q1 measures predictive quality + calibration + robustness; Q2 adds latency. We report end-to-end forward-pass latency averaged over

R

runs after

W

warmup iterations, with fixed batch size. Worst denotes the minimum i.i.d. performance across seeds:

\mathrm{Worst}=\min_{s\in\mathcal{S}}m(s,\text{i.i.d.})

. Rob

\mu

denotes the mean performance under the robustness protocol:

\mathrm{Rob}\ \mu=\frac{1}{|\mathcal{S}|\cdot|\mathcal{P}|}\sum_{s\in\mathcal{S}}\sum_{p\in\mathcal{P}}m(s,p)

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

torch	cuda	gpu	cudnn
2.5.1+cu121	12.1	NVIDIA RTX 6000 Ada Generation	90100

Appendix

Table 4. Environment snapshot for reproducibility.

Notation	Definition
$\sigma(\cdot)$	sigmoid / activation function
$f(\cdot)$	predictor / classification head
$\theta,\Theta$	parameters (module / all parameters)
$m$	number of tabular fields
$d$	embedding dimension per field
$e_{i}$	embedding of field $i$

Appendix

Table 5. Full list of notations (supplement to Table 1 ).

Variant	Adult			IMDB			ogbn-prod
Variant	Metric $\uparrow$	Active% $\downarrow$	ECE $\downarrow$	Metric $\uparrow$	Active% $\downarrow$	ECE $\downarrow$	Metric $\uparrow$	Active% $\downarrow$	ECE $\downarrow$
Dense backbone (No Gate)	0.8729	100.0	0.0247	0.9318	100.0	0.0216	0.6893	100.0	0.1167
Static/global mask (non-conditional)	0.8696	19.7	0.0209	0.9284	21.3	0.0207	0.6768	24.6	0.1049
L0 gate (interface-aligned, default)	0.8794	12.5	0.0158	0.9356	11.8	0.0189	0.7017	15.2	0.0716
Fine-grained gate (overly granular)	0.8657	16.4	0.0296	0.9261	14.9	0.0313	0.6712	18.7	0.1124
No annealing / fixed- $\lambda$ schedule	0.8619	13.1	0.0417	0.9227	12.4	0.0398	0.6668	15.9	0.1396

Appendix

Table 6. Ablation study on L0GM design choices. We fix the backbone per modality and vary only the gating primitive and its attachment/granularity, reporting predictive quality, active fraction (%), and reliability (ECE).

Hyperparameter	Search range / candidates
Learning rate (LR)	$\{1\mathrm{e}{-4},\,3\mathrm{e}{-4},\,1\mathrm{e}{-3}\}$
Weight decay	$\{0,\,1\mathrm{e}{-6},\,1\mathrm{e}{-5},\,1\mathrm{e}{-4}\}$
Dropout	$\{0.00,\,0.05,\,0.10,\,0.15\}$
Batch size	$\{64,\,128,\,256,\,512\}$ (hardware-limited)
Gate penalty $\lambda$	log-grid over $[10^{-4},\,10^{-2}]$ (e.g., 5–9 points)
$\ell_{0}$ annealing schedule	warmup $\in[0.05,\,0.20]$ ; end-temp $\in[0.5,\,2.0]$

Appendix

Table 7. Hyperparameter search space (L0GM). Ranges are modality-agnostic; modality-specific fields are marked when applicable.

Mod.	$\lambda$	Active% $\downarrow$	Acc/AUC $\uparrow$	Worst $\uparrow$	ECE $\downarrow$
Tab	$10^{-4}$	100.0	0.890	0.870	0.018
	$3\cdot 10^{-4}$	35.0	0.889	0.868	0.012
	$10^{-3}$	15.0	0.887	0.866	0.010
	$3\cdot 10^{-3}$	7.0	0.881	0.852	0.019
Text	$10^{-4}$	100.0	0.933	0.143	0.024
	$3\cdot 10^{-4}$	30.0	0.932	0.143	0.020

Appendix

Table 8. (F1)

\lambda

sweep. Active fraction decreases monotonically as

\lambda

increases, while ECE is typically non-monotonic (often improving at moderate sparsity and degrading under extreme sparsity).

Mod.	Variant	Acc/AUC $\uparrow$	Worst $\uparrow$	ECE $\downarrow$
Tab	No annealing (fixed)	0.862	0.842	0.042
Tab	With annealing (default)	0.889	0.867	0.012
Text	No annealing (fixed)	0.923	0.158	0.040
Text	With annealing (default)	0.933	0.143	0.020
Graph	No annealing (fixed)	0.667	0.651	0.140
Graph	With annealing (default)	0.724	0.708	0.033

Appendix

Table 9. (F2) Annealing ablation. Warm-up + gradual hardening stabilizes training (improves Worst) and tends to reduce ECE compared to fixed hardness / no annealing.

Mod.	Regime	$\lambda$	Active%	Acc/AUC	ECE (no coup.)	ECE (coup.)
Tab	Moderate	$10^{-3}$	15.0	0.887	0.010	0.010
Tab	High	$3\cdot 10^{-3}$	7.0	0.881	0.019	0.014
Text	Moderate	$10^{-3}$	12.0	0.930	0.017	0.017
Text	High	$3\cdot 10^{-3}$	6.0	0.922	0.026	0.021
Graph	Moderate	$10^{-3}$	15.0	0.721	0.028	0.028
Graph	High	$3\cdot 10^{-3}$	8.0	0.712	0.045	0.036

Appendix

Table 10. (F3) Calibration coupling. Coupling has little effect at moderate sparsity (ECE already improves), but can partially recover ECE under aggressive sparsity.

Activation Sparsity

Activation sparsity studies how neural networks can use mostly inactive units to reduce computation, memory, and energy while preserving useful representations.

See all 56 papers

Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso
Apr 5, 2026Fei Wang, Yutong Zhang, Xiong WangMultimodal Learning Multimodal Graph Learning
Department of Applied Mathematics and Statistics, Stony Brook University · School of Mathematics, Sichuan University · School of Computer Science and Technology, USTC
On Learning Expand-and-Sparsify Representations
Oct 19, 2026Alireza Asghari, Parikshit Ram, Kaushik Sinha
Wichita State University · IBM Research
GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
Mar 12, 2026Xiaoyun Liu, Divya Saxena, Jiannong Cao +3LLM Pruning Structured Pruning
Department of Computing, The Hong Kong Polytechnic University, Hong Kong · School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur 342030, India

Apr 5, 2026J/K move · Enter open · S save

Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

Fei Wang, Yutong Zhang, Xiong Wang

Department of Applied Mathematics and Statistics, Stony Brook University · School of Mathematics, Sichuan University · School of Computer Science and Technology, USTC

Multimodal Learning Multimodal Graph Learning Multimodal Disentangled Representation Learning

Open paper AskPDF arXiv

Learning interpretable multimodal representations inherently relies on uncovering the conditional dependencies between heterogeneous features. However, sparse graph estimation techniques, such as Graphical Lasso (GLasso), to visual-linguistic domains is severely bottlenecked by high-dimensional noise, modality misalignment, and the confounding of shared versus category-specific topologies. In this paper, we propose Cross-Modal Graphical Lasso (CM-GLasso) that overcomes these fundamental limitations. By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space. We introduce a cross-attention distillation mechanism that condenses high-dimensional patches into explicit semantic nodes, naturally extracting spatial-aware cross-modal priors. Furthermore, we unify tailored GLasso estimation and Common-Specific Structure Learning (CSSL) into a joint objective optimized via the Alternating Direction Method of Multiplier (ADMM). This formulation guarantees the simultaneous disentanglement of invariant and class-specific precision matrices without multi-step error accumulation. Extensive experiments across eight benchmarks covering both natural and medical domains demonstrate that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.

Abstract

Activation Sparsity

Explore similar work

Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

On Learning Expand-and-Sparsify Representations

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models