Multi-view learning primarily aims to fuse multiple features to describe data comprehensively. Most prior studies implicitly assume that different views share similar dimensions. In practice, however, severe dimensional disparities often exist among different views, leading to the unbalanced multi-view learning issue. For example, in emotion recognition tasks, video frames often reach dimensions of 106, while physiological signals comprise only 101 dimensions. Existing methods typically face two main challenges for this problem: (1) They often bias towards high-dimensional data, overlooking the low-dimensional views. (2) They struggle to effectively align representations under extreme dimensional imbalance, which introduces severe redundancy into the low-dimensional ones. To address these issues, we propose the Adaptive Multi-view Sparsity Learning (AdaMuS) framework. First, to prevent ignoring the information of low-dimensional views, we construct view-specific encoders to map them into a unified dimensional space. Given that mapping low-dimensional data to a high-dimensional space often causes severe overfitting, we design a parameter-free pruning method to adaptively remove redundant parameters in the encoders. Furthermore, we propose a sparse fusion paradigm that flexibly suppresses redundant dimensions and effectively aligns each view. Additionally, to learn representations with stronger generalization, we propose a self-supervised learning paradigm that obtains supervision information by constructing similarity graphs. Extensive evaluations on a synthetic toy dataset and seven real-world benchmarks demonstrate that AdaMuS consistently achieves superior performance and exhibits strong generalization across both classification and semantic segmentation tasks.
Figures & tables
Fig. 1 : Ubiquitous Dimensional Imbalance in Real-world Multi-view Applications. We illustrate four representative scenarios—(a) Medical Diagnosis, (b) Financial Analysis, (c) Emotion Recognition, and (d) Monitoring—where extreme dimensionality disparities (e.g., ≈106 vs. ≈101 ) between views are prevalent. This fundamental imbalance challenges conventional fusion methods, which often bias towards high-dimensional features.
Fig. 2 : Illustration of the proposed AdaMuS framework. AdaMuS establishes view-specific encoders f(v)(⋅) to process unbalanced multi-view data. These encoders are structurally optimized by the Principal Neuron Analysis (PNA) module to learn aligned representations {znv}v=1V with a unified dimension. Specifically, a Multi-view Sparse Batch Normalization (MSBN) layer is proposed to explicitly integrate these features via a sparse fusion paradigm. Finally, we train AdaMuS in a self-supervised manner to guide the representation learning.
Fig. 3 : Learning curves of a DNN classifier on the low-dimensional view of the CUB dataset. The model projects the 30-dimensional input into a high-dimensional space (structure: 30–1024–512–10). The increasing gap between training and validation performance demonstrates that expanding low-dimensional data with deep networks leads to overfitting.
Fig. 4 : PNA workflow: (1) Compute covariance matrix from layer outputs; (2) Extract eigenvalue distribution; (3) Measure dissimilarity to Uniform/Dirac baselines; (4) Adjust pruning rate with view imbalance.
Fig. 5 : Construction process of the multi-view toy example.
Fig. 6 : t-SNE visualization of the learned representations on the toy example.
Fig. 7 : Impact of Dimensional Compression. Comparison between appropriate compression ( 3000→1024 ) and over-compression ( 3000→128 ). The gray gap highlights the significant information loss caused by aggressive compression, verifying the need for a high-dimensional subspace.
Fig. 8 : Validation of the PNA pruning module. We compare the learning dynamics of (a) PNA-enhanced DNN vs. (b) Standard DNN during dimensional expansion ( 10→512→1024 ). The Standard DNN suffers from overfitting, whereas PNA ensures stable convergence by pruning redundant connections.
Fig. 9 : Verification of sparse alignment. (Left) Ground-truth mapping matrix W2T , where red dashed boxes indicate irrelevant dimensions. (Right) The learned scaling factor γ effectively suppresses these dimensions while activating the informative ones.
Dataset
Size
Category
Dimensions
Λ
UCI
1600
10
6/47/240
2.88
CUB
480
10
30/1024
3.22
ORL
400
40
30/3304/6750
2.54
MSRCV1
210
7
20/100/210/256/512/1302
2.41
Mfeat
2000
10
240/216/76/64/47/6
2.22
100leaves
1600
100
5/64/64
2.14
TABLE I : DATASET STATISTICS. Λ INDICATES THE UNBALANCED DEGREE. NYUV2 IS EMPLOYED FOR SEGMENTATION.
Datasets
Metric
Methods
Concat
DCCAE
CGCN
BMGC
ICMVC
CANDY
GUMRL
PDMF
UIMC
SURER
STCMC-UR
UMDL
AdaMuS
UCI
ACC
71.56±3.98
74.62±1.19
70.00±1.96
73.45±1.68
85.70±1.18
69.18±2.05
86.09±0.08
85.20±1.09
76.88±2.01
77.90±1.12
72.24±1.39
87.51±0.32
91.49±1.08
NMI
72.39±2.11
79.77±1.11
78.01±0.87
73.33±1.57
83.13±1.05
67.23±2.14
78.66±0.91
75.84±1.44
67.15±2.33
75.57±1.76
73.27±0.60
79.08±1.02
86.93±0.97
ARI
62.28±3.15
71.97±0.78
57.71±2.96
63.64±2.83
75.14±1.31
57.61±1.75
69.63±0.26
71.51±1.06
60.44±3.11
67.06±2.17
62.46±1.10
74.23±0.14
82.85±0.90
CUB
ACC
64.23±2.91
69.71±1.83
57.08±2.42
68.96±1.95
62.53±2.40
51.48±2.90
61.73±2.16
70.47±3.71
70.11±1.17
69.60±1.02
55.22±2.55
80.12±1.64
81.65±1.74
NMI
60.85±1.69
67.34±2.54
69.29±1.34
57.46±2.85
54.07±1.76
64.75±2.31
53.05±1.93
66.83±1.71
65.82±1.07
72.29±0.99
63.55±2.63
73.43±2.74
77.21±0.45
TABLE II : Clustering performance comparison on different datasets (mean result ± standard deviation). AdaMuS is our proposed method. The best and the second best results are highlighted by boldface and underlined , respectively.
Datasets
Metric
Methods
PDMF
GUMRL
SURER
RCML
UMDL
AdaMuS (Ours)
UCI
ACC
94.43 ± 0.15
96.88 ± 0.09
96.00 ± 0.25
96.67 ± 0.72
97.25 ± 0.17
98.50 ± 0.03
F1
96.03 ± 0.19
96.78 ± 0.10
95.87 ± 0.32
96.63 ± 0.74
97.00 ± 0.17
98.50 ± 0.03
CUB
ACC
87.29 ± 0.69
85.42 ± 1.16
89.17 ± 0.24
88.75 ± 2.64
87.99 ± 0.40
90.00 ± 0.31
F1
87.17 ± 0.58
85.50 ± 0.84
89.09 ± 0.33
88.20 ± 2.26
87.56 ± 0.50
89.57 ± 0.17
ORL
ACC
69.50 ± 4.03
92.12 ± 0.27
94.00 ± 0.21
96.62 ± 0.72
92.25 ± 0.43
97.25 ± 0.02
TABLE III : Classification comparison across benchmarks (Mean ± Std). Bold and underlined values denote the highest and second-highest scores, respectively.
Fig. 10 : Efficiency comparison between UMDL and AdaMuS. The chart displays the ratios of parameters and FLOPs required by AdaMuS relative to UMDL (blue and orange bars, respectively).
Fig. 11 : t-SNE visualization of original data and comprehensive representations on CUB dataset. (a) and (b) are original features, respectively. (c) is the result of AdaMuS-RS, which removes the MSBN layer. (d) is the result of AdaMuS.
Datasets
Method
Metrics
LG
LS
LP
ACC (%)
NMI (%)
ARI (%)
MSRCV1
✓
-
-
87.67 ± 1.59
80.72 ± 1.84
76.24 ± 1.80
✓
✓
-
88.71 ± 1.47
81.19 ± 1.33
76.66 ± 1.72
✓
-
✓
92.86 ± 0.92
88.28 ± 1.05
83.21 ± 1.13
✓
✓
✓
95.05 ± 0.77
93.07 ± 0.76
91.75 ± 0.60
ORL
✓
-
-
70.27 ± 2.34
84.21 ± 1.18
66.04 ± 1.83
TABLE IV : Ablation study. LG is the foundational graph-based contrastive learning module. LS denotes the sparsity fusion constraint, and LP represents the PNA pruning module. Best results are in bold .
Fig. 12 : Segmentation results of our AdaMuS-SEG and baseline UNET on the NYUv2 dataset.
Fig. 13 : Hyper-parameter analysis on the MSRCV1 dataset.
Fig. 14 : Convergence analysis on the MSRCV1 dataset.
The high-dimensional features extracted from large-scale unlabeled data via various pretrained models with diverse architectures are referred to as heterogeneous multiview data. Most existing unsupervised transfer learning methods fail to faithfully recover intrinsic subspace structures when exploiting complementary information across multiple views. Therefore, a fundamental challenge involves constructing sparse similarity graphs that preserve these underlying subspace structures for achieving semantic alignment across heterogeneous views. In this paper, we propose a sparse attention graph learning (SAGL) method that learns subspace-preserving sparse attention graphs from heterogeneous multiview data. Specifically, we introduce a bilinear attention factorization scheme to capture asymmetric similarities among the high-dimensional features, which breaks the symmetry bottleneck that is inherent in the traditional representation learning techniques. A dynamic sparsity gating mechanism then predicts a feature-specific compression factor for adaptively controlling the topological contributions of neighbors. Furthermore, we employ a structured sparse projection via α-entmax to generate subspace-preserving sparse attention graphs for individual views. SAGL leverages these view-specific graphs to conduct sparse information aggregation, yielding discriminative representations for multiview learning tasks. In addition, we provide a rigorous theoretical analysis that bridges differentiable sparse attention and probability simplex constraints. Extensive experiments conducted on multiple benchmark datasets demonstrate that SAGL consistently outperforms the state-of-the-art unsupervised transfer learning approaches.
Jie Chen, Yuanbiao Gou, Chuanbin Liu +2
College of Computer Science Sichuan University Chengdu, China · School of Economics and Management China University of Petroleum (Beijing) Beijing, China · Law School Sichuan University Chengdu, China +1
Urban region representation learning encodes heterogeneous urban signals, such as mobility flows, points of interest, and land-use patterns, into unified embeddings for downstream tasks, including check-in prediction, crime forecasting, and service demand estimation. Existing approaches rely on globally shared fusion and joint optimization, implicitly assuming uniform modality importance and naturally balanced multi-objective training. However, urban data is inherently heterogeneous. Our empirical analysis reveals two structural issues: (i) modality utility varies markedly across regions, causing static fusion to overemphasize signals from certain modalities; and (ii) heterogeneous objectives exhibit persistent gradient imbalance, causing representational bias and unstable training. We present PRISM, a self-adjusting framework for adaptive multi-view learning. PRISM adapts sparse Mixture-of-Experts (MoE) routing to learn region-specific view allocations, while a Proportional-Integral-Derivative (PID)-inspired regulation mechanism mitigates gradient conflicts during optimization. Experiments on three large-scale urban datasets show that PRISM achieves more competitive performance than strong multi-view baselines, while providing more stable optimization behavior and more balanced region representations.
Sean Bin Yang, Zongyi Xu, Jilin Hu +5
Aalborg University, Denmark · Chongqing University of Posts and Telecommunications, China · East China Normal University, China
Multi-View Clustering (MVC) has gained significant attention for its ability to leverage complementary information across diverse views. However, existing deep MVC methods often struggle with view-distribution entanglement during cross-view fusion, which hampers the quality of the shared latent space and leads to suboptimal Figures. To address this issue, we propose the Generalized Multi-view Auto-Encoder (GMAE), a framework designed to preserve cross-view complementarity through disentangled representation learning. Specifically, GMAE employs dual-path autoencoders to decouple source features into view-specific and view-common embeddings, facilitating the discovery of clearer clustering structures. We further construct cross-view adversarial discriminators to guide view-specific encoders in capturing more discriminative features. By strategically modulating mutual information, GMAE effectively aligns distributions and prevents representation collapse, ensuring the generation of robust, non-trivial embeddings. Comprehensive experiments on 13 benchmark datasets demonstrate that GMAE consistently outperforms state-of-the-art methods in both complete and incomplete MVC tasks. Our code implementation is available at the repository: https://github.com/obananas/GMAE.
Xin Zou, Ruimeng Liu, Chang Tang +4
School of Computing and Information Technology, University of Wollongong, NSW, 2500, Australia · School of Software Engineering, Huazhong University of Science and Technology, Wuhan 430074, China