Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

Eeshaan Jain; Linus Bleistein; Bart Deplancke; Charlotte Bunne

cs.LGOct 2, 2026

Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

Authors: Eeshaan Jain, Linus Bleistein, Bart Deplancke, Charlotte Bunne

Organizations: School of Computer and Communication Sciences, EPFL · School of Life Sciences, EPFL

Abstract

Recent progress in multimodal, high-dimensional learning has enabled foundation models to process heterogeneous, large-scale data. However, at test time, acquiring all features or modalities can be prohibitively costly and often redundant. Sequentially selecting informative modalities is therefore critical, yet challenging when the downstream task or prediction target is unknown. To this end, we introduce ECHO- $k$ , a task-agnostic and self-supervised learning principle for modality acquisition: we use a deep model's internal pretrained representations (e.g., from a foundation model) as proxy targets that summarize cross-modal information. We provide theoretical guarantees in a stylized linear setting that motivate a reinforcement learning (RL) policy for sequential modality selection. Across task-agnostic and label-free acquisition baselines, ECHO- $k$ consistently improves budgeted downstream performance across diverse foundation-model backends. Our method provides a principled route to cost-aware test-time deployment, with implications for any multimodal system where measurements are expensive or time-constrained, and downstream tasks unknown a priori.

Figures & tables

Figure 1:

\textsc{echo}\textrm{-}k

learns an RL policy to iteratively selects views to approximate the full latent representation learned by any pretrained model, without any knowledge of downstream tasks. Once learned, the policy can be deployed for budgeted feature acquisition.

Figure 2: Expected true loss gap curve between the oracle and the test-time optimal, random, and prior-based (

\mathbb{E}[\bm{E}_{j}]

) policies.

Figure 3: a. Results of $\textsc{echo}\textrm{-}k$ on acquiring geospatial modalities to predict soil pH regression on MMEarth-Bench ( $d=12$ ) b. $\textsc{echo}\textrm{-}k$ can operate on single-cell RNA-seq foundation models to sequentially acquire informative genes that support broad set of downstream tasks. We further present comparison of $\textsc{echo}\textrm{-}k$ to baselines, evaluated on a cell type annotation task on hPancreas (with $d=200$ ).

Figure 4: Results of $\textsc{echo}\textrm{-}k$ on the NSCLC dataset. a. Overview of the experimental setting: given high-dimensional multiplex imaging data, we study how foundation models trained on such data can be leveraged to sequentially acquire informative markers for downstream annotation and discovery tasks. b. Comparison of $\textsc{echo}\textrm{-}k$ with baseline methods on the cancer subtyping task. c. Comparison of $\textsc{echo}\textrm{-}k$ with baseline methods on the cancer grading task. d. Scatter plot illustrating the accuracy trade-off between cancer subtyping and grading when using the same set of views.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Meaning
Representation-acquisition setup
$n$	Number of i.i.d. samples.
$K$	Number of possible views in the general acquisition problem.
$\bm{m}_{k}^{i}$	The $k$ -th view of sample $i$ . A view may be a modality or a jointly acquired feature group.
$\mathcal{M}^{k}$	Space of possible values for view $k$ .
$\Phi_{k}:\mathcal{M}^{k}\to\mathbb{R}^{d_{k}}$	View-specific encoder for view $k$ .

Appendix

Table 5

Figure 5: Overview of the Latent Gumbel Correlation Model. a. Distribution of $\log(\bm{x}_{j})$ for 3 equiprobable latent states $Z$ with support $\{-1,0,1\}$ . b. Correlation of the features under the gradient correlation family described in § G.1.6 .

Name	Model Driven?	Selection Criterion	Adaptive?	Source
Oracle	✓	1-step Oracle knowledge	✓	—-
GIR	✓	Average contribution to latent state	✗	—-
Random	✗	Uniform distribution over unseen features	✗	—-
Laplacian Score †	✗	Locality preservation	✗	He et al. (2005)
spec - $\varphi_{3}$ †	✗	Graph-based ranking	✗	Zhao and Liu (2007)
UDFS †	✗	Local discriminative structure	✗	Yang et al. (2011)

Appendix

Table 6: Summary description of implemented baselines. Methods marked with

(\dagger)

rely on unsupervised feature selection and have oracle access to all features at ranking time. Methods marked with a

(\star)

are only deployed on the cancer grading and typing datasets, since they rely on a protein sequence foundation model ( Lin et al., 2023 ) . Methods marked with

(\ddagger)

are label-free adaptive baselines that select instance-specific feature sets through continuous updates of their selection policies.

Parameter	MNIST/Fashion-MNIST	MiniBooNE	hPancreas	NSCLC	MMEarth-Bench	Synthetic
Acquisition budget $T$	10	7	25	10	5	varies
Step threshold $t_{0}$	$T/2$	$T/2$	$T/2$	$T/2$	$T/2$	$T/2$
Gain factor $\lambda$	1.5	1.5	1.5	1.5	1.5	1.5
Discount factor $\gamma$	0.98	0.98	0.99	0.98	0.98	0.98
Training steps	300k	300k	500k	300k	200k	300k

Appendix

Table 7: Hyperparameters used for

\textsc{echo}\textrm{-}k

across datasets. The remaining hyperparameters follow the defaults in Stable-Baselines3 .

Figure 6: Downstream performance of random, prior-based (

\mathbb{E}[\bm{E}_{j}]

), optimal-test-time, and oracle policies on a random isotropic regression task

Figure 7: Results of $\textsc{echo}\textrm{-}k$ across all baselines for the MNIST (top-left), Fashion-MNIST (top-right), hPancreas (bottom-left), and MiniBooNE (bottom-right) datasets.

Method	10%	20%	40%	60%
DIME	$79.97$	$84.23$	$86.87$	$87.07$
GDFS	$80.23$	$84.73$	$87.50$	$87.80$
SEFA	$78.67$	$84.70$	$87.73$	$88.13$
GSMRL	$75.73$	$83.27$	$87.37$	$87.77$
EDDI	$72.30$	$76.70$	$83.50$	$85.07$
$\textsc{echo}\textrm{-}k$	$80.03$	$81.70$	$86.87$	$87.60$

Appendix

Table 9: Comparison with supervised acquisition on MiniBooNE. Accuracy (%) at different acquisition budgets. All methods except

\textsc{echo}\textrm{-}k

use downstream labels. denotes best and second best performance.

Figure 8: a. Overview of the task and acquisition behavior of echo - $k$ : we illustrate how $\textsc{echo}\textrm{-}k$ sequentially acquires pixels in an unsupervised manner, progressively concentrating the (unobserved) downstream class probability mass. b. Distribution of acquisition costs among the first ten selected pixels; the cost-aware policy selects lower-cost pixels early. c . Accuracy as a function of cumulative acquisition cost; cost-aware training improves performance in the low-budget regime, while both policies converge at larger budgets.

Figure 9: Distribution of $\bm{B}^{\top}\bm{B}-\bm{I}_{d}$ for 4 different DinoV2 models of increasing capacity.

Figure 10: Variation of $\|\bm{B}^{\top}\bm{B}-\bm{I}_{d}\|_{\text{op}}$ for varying values of $d$ on trained autoencoder models and variation of the empirical estimation of $C$ with $d$ .

Figure 11: Analysis of the top-15 markers and their frequency, as chosen by $\textsc{echo}\textrm{-}k$ jointly for the cancer subtyping and grading task on the NSCLC spatial proteomics dataset.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 231 papers

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion
Oct 5, 2026Payal Mohapatra, Haodong Yang, Yueyuan Sui +3Efficient Multimodal Inference Multimodal Classification
Arm Inc., USA · Northwestern University, USA
QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities
Sep 28, 2026Hanyin Cheng, Linfeng Wang, Zhengbo Qu +7Multivariate Time Series Forecasting Multimodal Foundation Models
East China Normal University · Huawei Technologies Co., Ltd.
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
May 20, 2026Herman Bergström, Aditya Mehrotra, Rahul G. KrishnanMultimodal Embedding Multimodal Classification
Chalmers University of Technology and University of Gothenburg · Vector Institute · University of Toronto

Oct 5, 2026J/K move · Enter open · S save

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

Payal Mohapatra, Haodong Yang, Yueyuan Sui, Stephen Xia+2

Arm Inc., USA · Northwestern University, USA

Efficient Multimodal Inference Multimodal Classification Multimodal Fusion

Open paper AskPDF arXiv

Multimodal systems often encode every available input, even when a subset suffices for prediction. Adaptive acquisition can reduce this cost by using predictions from incrementally fused evidence to decide which modality to encode next and when to stop. However, sequential fusion makes these predictions order-dependent, so decisions based on them may need to distinguish factorially many histories of the same acquired set. We introduce SemARC, which couples a Sequential Modality Aggregator (SeMA) with an Adaptive Runtime Controller (ARC) and uses acquired evidence to select each modality before its encoder runs. SeMA executes only selected encoder and fusion branches, updates a fixed-size state, and predicts after each acquisition without recomputing earlier branches. We supervise every acquisition prefix under randomized modality subsets and orders to encourage consistent predictions across acquisition orders. ARC combines a set-dependent marginal-utility prior with residual fitted-Q learning to select the next available modality or stop, without inspecting unacquired inputs or retaining acquisition order. Across six multimodal classification datasets and eleven baselines, SemARC achieves 3.2% higher macro-F1 and 61.4% lower total inference GFLOPs on average relative to each dataset's most accurate baseline. End-to-end latency falls by 44.0% across GPU and CPU and by 47.2% on Android INT8 relative to the fastest measured baseline, on average. Under varying runtime modality missingness, SemARC still skips available modalities, matching or exceeding the best baseline macro-F1 in 21 of 24 conditions with 14.8% lower total GFLOPs on average. SemARC thus offers a practical path toward efficient multimodal inference across heterogeneous devices.

Abstract

Efficient Multimodal Inference

Explore similar work

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities

Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach