Beyond the Grid: Continuous Dictionary Pursuit for Interpretable Signal Decomposition

Nick Janssen; Melanie Schaller; Bodo Rosenhahn

cs.LGNov 6, 2026

Beyond the Grid: Continuous Dictionary Pursuit for Interpretable Signal Decomposition

Authors: Nick Janssen, Melanie Schaller, Bodo Rosenhahn

Organizations: Institute of Information Processing, Leibniz University Hannover

Abstract

Deep learning models achieve high accuracy in signal processing but remain difficult to interpret. Classical matching pursuit and sparse coding offer better transparency but suffer from spectral leakage and grid mismatch when signal frequencies do not align with discrete dictionary atoms. We introduce Continuous Dictionary Pursuit (CDP), a framework for interpretable signal decomposition that learns the parameters of closed-form functions directly from data. CDP optimizes over a continuous manifold of differentiable atoms and, unlike prior continuous pursuit methods that operate on a single analytic family, searches a heterogeneous library spanning periodic, trend, transient, and discontinuous structures. The algorithm employs a greedy iterative strategy using data-driven priors and gradient-based optimization to isolate individual signal components, with discontinuous waveforms made differentiable through spectral sharpness annealing. To ensure a sparse atom set for reconstruction, the framework utilizes a stopping criterion based on a predefined atom budget and residual error convergence. Evaluations on synthetic and real-world datasets show that CDP removes the grid discretization error for atom families contained in its library, recovers generative parameters to within $1\%$ on stationary synthetic signals expressible in that library, and achieves the best sparsity-accuracy trade-off among classical, continuous-pursuit, and neural baselines in the sparse regime. Beyond decomposition, we demonstrate that the extracted atoms can serve as structured curriculum targets for training sequence models, improving forecasting accuracy over standard end-to-end training on most evaluated datasets. The method offers a glass-box alternative for signal decomposition by providing high reconstruction fidelity alongside direct physical interpretability.

Figures & tables

Figure 1: Continuous Dictionary Pursuit: The iterative greedy procedure initializes atoms from data-driven priors, identifies a candidate via truncated optimization, fully optimizes it with L-BFGS [ 30 ] , and adds it to the additive model until the atom budget is exhausted or the residual error stagnates.

Figure 2: Parametric Dictionary Library: Example parameterizations of atom types: (a) SineAtom, (b) LinearAtom, (c) ExponentialAtom, (d) GaussianAtom, (e) SquareAtom, (f) SawtoothAtom, (g) StepAtom, (h) ModulatedAtom (for example

\texttt{SineAtom}\cdot\texttt{StepAtom}

Figure 3: Parameter recovery on synthetic datasets: CDP reconstructs the true mathematical components from noisy targets (

\sigma=0.1

): (a) linear trend, (b) sine, (c) square wave, (d) off-grid frequency superposition, (e) damped sine via a modulated atom, (f) non-stationary amplitude shift. Reconstruction and prediction MSE are given per panel.

Figure 4: Pareto frontier of parameter count versus training reconstruction MSE: CDP and CDP (Modulated) achieve the optimal trade-off in the sparse regime across all baselines. The lower convex envelope excludes Analysis K-SVD, whose near-zero training residual reflects in-sample denoising rather than a parametric decomposition ( subsection A.8 ).

Dataset	NOMP	SFW	MCA	AK-SVD	CDP-sine	CDP
Electricity	0.562	0.567	0.622	0.007	0.593	0.514
ETTh1	0.418	0.421	0.594	0.005	0.486	0.427
ETTh2	0.539	0.545	0.760	0.004	0.692	0.545
ETTm1	0.420	0.422	0.575	0.002	0.488	0.426
ETTm2	0.537	0.542	0.740	0.002	0.691	0.491
Exchange	0.055	0.060	0.135	0.000	0.145	0.066

Table 1: Per-dataset training reconstruction loss at

K=5

for the continuous and structured baselines, the CDP-sine ablation, and full CDP. The best value per dataset is bold, excluding Analysis K-SVD, whose near-zero training residual reflects in-sample denoising rather than a parametric decomposition ( subsection A.8 ).

Figure 5: Signal decomposition of the PEMS04 traffic dataset: Top: the original signal overlaid with the full CDP reconstruction. Left: the individual extracted atoms, capturing daily cycles, weekly patterns, and localized rush-hour modulations. Right: their progressive summation toward the target signal.

Figure 6: Forecasting MSE for curriculum learning versus standard end-to-end training across all 13 real-world LTSF datasets (IQM over 10 seeds, whiskers show 95% bootstrap confidence intervals).

Dataset	CDP	CDP (rev.)	SSA	MSTL	OMP	Prophet	NOMP	SFW	Smooth.	Standard
Electricity	1.281	1.277	1.286	1.295	1.269	1.273	1.254	1.269	1.261	1.270
ETTh1	2.128	2.244	2.272	2.285	2.216	2.199	2.222	2.241	2.283	2.345
ETTh2	1.409	1.675	1.566	1.449	1.503	1.496	1.497	1.397	1.492	1.593
ETTm1	2.006	2.105	2.162	2.158	2.100	2.177	2.065	2.136	2.183	2.052
ETTm2	1.233	1.272	1.233	1.218	1.227	1.254	1.220	1.285	1.236	1.228
Exchange	2.488	2.340	2.265	2.260	2.062	2.335	2.323	$9.8{\times}10^{4}$	1.824	2.248

Table 2: Curriculum evaluation on all datasets. Out-of-sample forecast MSE of the S4D model for every curriculum target (IQM over 10 seeds). The best value per dataset is bold.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Details	Default
Input projection	Conv1d( $1\to d_{\text{model}}$ , kernel 1)	$d_{\text{model}}=128$
S4D blocks	$n_{\text{layers}}$ stacked S4DBlocks	$n_{\text{layers}}=4$
State dimension	Per-block S4D state size	$d_{\text{state}}=64$
Local conv	Depthwise Conv1d(kernel 3, padding 1)	per block
Output gate	GLU via Conv1d( $d_{\text{model}}\to 2d_{\text{model}}$ )	per block
Dropout	Applied after GLU	$p=0.1$

Appendix

Table 3: AtomS4DModel architecture. Default hyperparameters used in all curriculum learning experiments.

Hyperparameter	Description	Default
$T$	Total training iterations	1000
$K$	Total number of atoms	10
$\rho$	Fraction of iterations on real data	0.1
$\alpha$	Curriculum transition smoothness	0.2

Appendix

Table 4: Curriculum learning hyperparameters used in all experiments.

Figure 7: Pareto frontier of model complexity versus out-of-sample forecasting error. The x-axis indicates the parameter count on a logarithmic scale. The y-axis shows the out-of-sample forecasting MSE. CDP and CDP (Modulated) maintain their performance at the sparse end of the frontier, achieving competitive forecasting accuracy with far fewer parameters than deep learning baselines. The continuous and structured baselines (NOMP, SFW, MCA, Analysis K-SVD) are included, with Analysis K-SVD lying outside the displayed range (median test error above

28

, see subsection A.8 ).

Figure 8: Mean absolute SHAP value per atom index in greedy selection order, averaged over all experiments and datasets (log scale). Shading denotes 95% bootstrap confidence intervals. SFW is excluded from the figure because its per-atom attributions diverge at larger budgets (see text).

Method	Counted parameters
CDP / CDP (Mod.)	Every continuously optimized scalar: $3$ – $5$ atom-specific parameters plus one bias per atom
OMP	Frequency index and linear coefficient per selected dictionary atom
NOMP	Refined continuous frequency plus complex amplitude per sinusoid
SFW	Continuous atom position plus weight per atom
MCA	Selected linear coefficients over the grid-discretized sub-dictionaries
Analysis K-SVD	All entries of the learned analysis operator rows

Appendix

Table 5: Parameter-count definitions used in all Pareto analyses. Each method’s count comprises exactly the scalars required to reproduce its output.

Figure 9: Training and inference cost versus model complexity. (a) Fitting time in seconds (log–log scale). (b) Evaluation (inference) time in seconds. Shading denotes 95% bootstrap confidence intervals over datasets.

Method	$K{=}1$	$K{=}2$	$K{=}3$	$K{=}5$	$K{=}10$	$K{=}20$	$K{=}30$	$K{=}40$
CDP	12	20	29	63	129	223	329	417
NOMP	22	44	66	125	441	462	695	1722
SFW	46	91	141	216	451	858	1174	1879
MCA	1	1	1	2	1	1	1	1
Analysis K-SVD	95	86	60	73	76	46	30	14

Appendix

Table 6: Median fitting time in seconds per atom budget

K

for CDP and the continuous and structured baselines, taken over 13 datasets and 10 seeds.

Hyperparameter	Value	Description
scout_batch_size	10	Number of candidate atoms evaluated in parallel during truncated optimization
scout_epochs	5	Number of L-BFGS steps per candidate during the truncated scouting phase
global_epochs	100	Number of L-BFGS steps during full optimization of the selected atom
num_global_optims	5	Number of joint global refinement passes over all active atoms
n_atoms	{1,2,3,5,8,10,15,20,30,40}	Atom budgets evaluated for the Pareto frontier
atom_types (CDP)	All except ModulatedAtom	Atom families available for greedy selection

Appendix

Table 7: CDP hyperparameters used in all experiments. CDP uses all atom types excluding the ModulatedAtom, while CDP (Modulated) includes it.

Figure 10: Electricity. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 11: ETTh1. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 12: ETTh2. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 13: ETTm1. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 14: ETTm2. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 15: Exchange. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 16: PEMS03. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 17: PEMS04. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 18: PEMS07. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 19: PEMS08. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 20: Solar. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 21: Traffic. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Figure 22: Weather. Reconstruction (a) and forecasting (b) MSE versus parameter count.

Time Series Decomposition

Time series decomposition separates evolving signals into components such as trend, seasonality, cycles, and residual variation.

See all 21 papers

Sep 12, 2025

Multi-Dictionary Learning for Low Rank Sparse Coding

Sparse dictionary coding represents signals as linear combinations of a few dictionary atoms. It has been applied to images, time series, graph signals and multi-way spatio-temporal data by jointly employing temporal and spatial dictionaries. Data-agnostic analytical dictionaries, such as the discrete Fourier transform, wavelets and graph Fourier, have seen wide adoption due to efficient implementations and good practical performance. On the other hand, dictionaries learned from data offer sparser and more accurate solutions but require learning of both the dictionaries and the coding coefficients. This becomes especially challenging for multi-dictionary scenarios since encoding coefficients correspond to all atom combinations from the dictionaries. To address this challenge, we propose a low-rank coding model for 2-dictionary scenarios and study its data complexity. Namely, we establish upper and lower bounds on the number of samples needed to learn dictionaries that generalize to unseen samples from the same distribution. We propose an alternating convex optimization solution, called AODL, which employs alternating optimization between the sparse coding matrices and the learned dictionaries. We demonstrate its quality for data reconstruction and missing value imputation in both synthetic and real-world datasets. For a fixed reconstruction quality, AODL learns up to

90\%

sparser solutions compared to non-low-rank and analytical (fixed) dictionary baselines. In addition, the learned dictionaries reveal interpretable insights into patterns from training samples.

Boya Ma, Abram Magner, Maxwell McNeil +1

University at Albany – SUNY, Computer Science, Albany, NY, USA

Representation Learning Sample Complexity Dictionary Learning Sparse Coding

Ask PDF

Jan 11, 2026

Variational decomposition autoencoding improves disentanglement of latent representations

Understanding the structure of complex, nonstationary, high-dimensional time-evolving signals is a central challenge in scientific data analysis. In many domains, such as speech and biomedical signal processing, the ability to learn disentangled and interpretable representations is critical for uncovering latent generative mechanisms. Traditional approaches to unsupervised representation learning, including variational autoencoders (VAEs), often struggle to capture the temporal and spectral diversity inherent in such data. Here we introduce variational decomposition autoencoding (VDA), a framework that incorporates a strong structural bias toward signal decomposition into variational representation learning. VDA is instantiated through variational decomposition autoencoders (DecVAEs), i.e., encoder-only neural networks that perform latent autoencoding at the level of distributions to learn multiple latent subspaces aligned with time-frequency characteristics. We evaluate DecVAEs on simulated data and three publicly available scientific datasets, spanning speech recognition, dysarthric speech evaluation, and emotional speech classification. DecVAEs improve on VAE-based methods in disentanglement quality and robustness, and their latent subspaces become factor-specific when the generative factors have distinct spectral signatures. These findings position decomposition-aware architectures as potential tools for extracting structured representations from speech, and potentially from other dynamic signals with factors exhibiting distinct spectral signatures, with applications in clinical diagnostics, human-computer interaction, and adaptive neurotechnologies.

Ioannis N. Ziogas, Aamna Al Shehhi, Ahsan H. Khandoker +1

Dept. of Biomedical Engineering, Khalifa University of Science and Technology, Abu Dhabi, UAE · School of Electrical and Computer Engineering, Aristotle University of Thessaloniki, Thessaloniki, Greece

Disentangled Representation Learning Variational Autoencoders Time Series Representation Learning Audio Representation Learning

Ask PDF

Oct 5, 2026

Inference and learning in sparse autoencoders as natural gradient flow

Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify inference and dictionary learning as natural-gradient flows on a shared variational free energy. We instantiate this framework as BeFOND, an encoder-free sparse coding model with closed-form inference and learning dynamics. We show how recurrent explaining away reduces interference between overlapping features, while Fisher preconditioning can compensate for the slow learning of rare features. On synthetic data, BeFOND improves dictionary recovery and rare-feature detection, with a growing advantage over amortized baselines as superposition increases. On language-model activations, it improves single-feature concept detection and selective intervention, outperforming pretrained reference SAEs with substantially less training data. Its feature quality continues to improve with dictionary width, whereas the evaluated baselines largely plateau. Together, these results show how improving inference and learning within a unified probabilistic framework can make better use of data and dictionary capacity to interpret and intervene on neural representations.

Hadi Vafaii, Tejas Rao, David Chanin +6

Astera Institute · UC Berkeley · UCL +2

Gradient Descent Sparse Autoencoders Mechanistic Interpretability Dictionary Learning Sparse Coding

Ask PDF

Abstract

Time Series Decomposition

Explore similar work

Multi-Dictionary Learning for Low Rank Sparse Coding

Variational decomposition autoencoding improves disentanglement of latent representations

Inference and learning in sparse autoencoders as natural gradient flow