MICA: Multivariate Infini Compressive Attention for Time Series Forecasting

Willa Potosnak; Nina Żukowska; Michał Wiliński; Dan Howarth; Ignacy Stępka; Mononito Goswami; Artur Dubrawski

cs.LGApr 7, 2026

MICA: Multivariate Infini Compressive Attention for Time Series Forecasting

Authors: Willa Potosnak, Nina Żukowska, Michał Wiliński, Dan Howarth, Ignacy Stępka, Mononito Goswami, Artur Dubrawski

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA · Amazon, Seattle, WA, USA

Abstract

Multivariate forecasting with Transformers faces a core scalability challenge: modeling cross-channel dependencies via attention compounds attention's quadratic sequence complexity with quadratic channel scaling, making full cross-channel attention impractical for high-dimensional time series. We propose Multivariate Infini Compressive Attention (MICA), an architectural design to extend channel-independent Transformers to channel-dependent forecasting. By adapting efficient attention techniques from the sequence dimension to the channel dimension, MICA adds a cross-channel attention mechanism to channel-independent backbones that scales linearly with channel count and context length. We evaluate channel-independent Transformer architectures with and without MICA across multiple forecasting benchmarks. MICA reduces forecast error over its channel-independent counterparts by 5.4% on average and up to 25.4% on individual datasets, highlighting the importance of explicit cross-channel modeling. Moreover, models with MICA rank first among deep multivariate Transformer and MLP baselines. MICA models also scale more efficiently with respect to both channel count and context length than Transformer baselines that compute attention across both the temporal and channel dimensions, establishing compressive attention as a practical solution for scalable multivariate forecasting.

Figures & tables

Figure 1: MICA consists of local attention to capture temporal dependencies and global linear attention to capture cross-channel information, combined via a mixing gate. Attention outputs are shown as patch (token)

\times

d

matrices, where darker cells indicate stronger weighted values.

Method	Architecture Type	Cross-Channel Method	Application Stage	Complexity
Decorrelation ( Kessy et al., 2018 ; Zheng & Sun, 2024 )	Universal	PCA	Preprocessing	$\mathcal{O}(PC^{2}+C^{3})$
Decorrelation ( Popescu, 2003 )	Universal	FastICA	Preprocessing	$\mathcal{O}(PC^{2})$
MLP -Multivariate ( Rosenblatt, 1958 )	MLP	Dense Layer Mixing	Model Encoder	$\mathcal{O}(PC)$
TSMixer ( Chen et al., 2023 )	MLP	Channel-wise Linear Projection	Model Encoder	$\mathcal{O}(PC)$
TinyTimeMixers ( Ekambaram et al., 2024 )	MLP	Channel-wise Linear Projection	Model Encoder	$\mathcal{O}(PC)$
TimeMixer ( Wang et al., 2024b )	MLP	Multi-scale Mixing	Model Encoder	$\mathcal{O}(P^{2}C)$

Table 1: Comparison of multivariate time series forecasting methods by cross-channel processing technique and complexity. Complexity refers to how computational cost scales with the context length (

P

tokens/patches) and number of channels

C

. Complexities assume constant hidden dimensions and number of layers.

Figure 2: MICA is an attention-based architectural design proposed to model both local patterns and global cross-channel interactions. Q , K , and V denote the query, key and value matrices and

N

denotes the number of attention heads. MICA consists of three complementary components: (1) a scaled dot-product attention module that models detailed temporal relationships locally within individual time series, (2) a linear attention module that efficiently aggregates information globally across channels, and (3) a learnable attention mixing gate that adaptively balances local and global information in a computationally efficient manner. MICA variants, such as the non-linear MLP gate, are discussed in Appendix D.1 .

Panel of Table 2 — Table 2: Forecasting MAE averaged over 5 random seeds with standard deviation in parentheses. Methods without standard deviation have deterministic solutions. MICA results correspond to the MLP-Query Gate. Best results are shown in bold . Second best results are underlined . Blue results indicate lower forecast error of MICA compared with the univariate model counterpart. The average rank across datasets for deep learning models is presented at the bottom of the table, with the best result in bold and second best underlined .

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 299 papers

Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism
Mar 19, 2026Lei Gao, Hengda Bao, Jingfei Fang +3Multivariate Time Series Forecasting Sparse Attention
Department of Machine Learning, SF Express, Shanghai, China · Department of Management, Zhejiang University of Technology, Hangzhou, China · Department of Management, Zhejiang University, Hangzhou, China
UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration
Mar 6, 2026Xingsheng Chen, Xianpei Mu, Deyu Yi +6Multivariate Time Series Forecasting Long-Term Time Series Forecasting
School of Computing and Data Science, The University of Hong Kong, Hong Kong, China · School of Information Engineering, Beijing Institute of Graphic Communication, Beijing, China · Innovation Engineering College, Macau University of Science and Technology, Macau, China +1
XCTFormer: Leveraging Cross-Channel and Cross-Time Dependencies for Enhanced Time-Series Analysis
May 18, 2026Israel Zexer, Omri AzencotMultivariate Time Series Forecasting Time Series Imputation
The Stein Faculty of Computer and Information Science Ben-Gurion University of the Negev

Mar 19, 2026J/K move · Enter open · S save

Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism

Lei Gao, Hengda Bao, Jingfei Fang, Guangzheng Wu+2

Department of Machine Learning, SF Express, Shanghai, China · Department of Management, Zhejiang University of Technology, Hangzhou, China · Department of Management, Zhejiang University, Hangzhou, China

Multivariate Time Series Forecasting Sparse Attention

Open paper AskPDF arXiv

The task of multi-channel time series forecasting is ubiquitous in numerous fields such as finance, supply chain management, and energy planning. It is critical to effectively capture complex dynamic dependencies within and between channels for accurate predictions. However, traditional method paid few attentions on learning the interaction among channels. This paper proposes Linear-Network (Li-Net), a novel architecture designed for multi-channel time series forecasting that captures the linear and non-linear dependencies among channels. Li-Net dynamically compresses representations across sequence and channel dimensions, processes the information through a configurable non-linear module and subsequently reconstructs the forecasts. Moreover, Li-Net integrates a sparse Top-K Softmax attention mechanism within a multi-scale projection framework to address these challenges. A core innovation is its ability to seamlessly incorporate and fuse multi-modal embeddings, guiding the sparse attention process to focus on the most informative time steps and feature channels. Through the experiment results on multiple real-world benchmark datasets demonstrate that Li-Net achieves competitive performance compared to state-of-the-art baseline methods. Furthermore, Li-Net provides a superior balance between prediction accuracy and computational burden, exhibiting significantly lower memory usage and faster inference times. Detailed ablation studies and parameter sensitivity analyses validate the effectiveness of each key component in our proposed architecture. Keywords: Multivariate Time Series Forecasting, Sparse Attention Mechanism, Multimodal Information Fusion, Non-linear relationship

Abstract

Multivariate Time Series Forecasting

Explore similar work

Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism

UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration

XCTFormer: Leveraging Cross-Channel and Cross-Time Dependencies for Enhanced Time-Series Analysis