HIFI: A Highly Efficient Framework for Statistical Inference with Foundation Models

Po-Wei Chen; Shan Yu; Chunlin Li

stat.MLOct 25, 2026

HIFI: A Highly Efficient Framework for Statistical Inference with Foundation Models

Authors: Po-Wei Chen, Shan Yu, Chunlin Li

Organizations: Department of Statistics, University of Virginia

Abstract

Foundation models provide powerful representations for unstructured data, such as text and images, and parameter-efficient fine-tuning (PEFT) makes their adaptation to downstream tasks increasingly practical. For neural estimators that lack a tractable sampling distribution, resampling-based procedures offer a flexible route to statistical inference, but they often require hundreds of model fits, making them computationally prohibitive at foundation-model scale. We introduce HIFI, a two-stage fine-tuning framework for statistical inference on parameters such as conditional means. Stage 1 adapts a foundation model once on a larger sample, capturing the complex structure of the data and controlling estimation bias through a single fit. Stage 2 applies a resampling inference procedure to a smaller independent sample, concurrently fitting multiple lightweight PEFT estimators to the residual to quantify estimation variability without relearning the full task. HIFI accommodates a broad class of resampling procedures, including ensemble subsampling and the nonparametric bootstrap. Across vision and text experiments, HIFI scales to pretrained models ranging from tens of millions to more than ten billion parameters while providing accurate point estimates and well-calibrated confidence intervals across diverse architectures, and reduces training compute by up to 91.9% relative to direct resampling. An application to gene-expression inference illustrates the practical value of the framework for uncertainty-aware scientific inference.

Figures & tables

Figure 1: Overview of HIFI. Stage 1 performs a single task-adaptation fit on $\mathcal{D}_{1}$ , yielding an adapted backbone $\widehat{F}_{1}$ and predictor $\widehat{\theta}_{1}$ . Holding these components fixed, Stage 2 fits $B$ estimator-specific residual PEFT modules using different sampling weights on the independent sample $\mathcal{D}_{2}$ . The residual estimates are then combined to obtain a point estimate and pointwise confidence interval for $\mu_{0}(x)$ .

			Point Estimation		Statistical Inference
Model	Params	Method	MAB	MAE	Cov. 90%	Cov. 95%	95% CI Length
		HIFI -ESM	0.018 (0.02)	0.041 (0.01)	0.908 (0.08)	0.937 (0.06)	0.282 (0.02)
		HIFI -Bootstrap	0.017 (0.02)	0.047 (0.01)	0.888 (0.07)	0.938 (0.05)	0.231 (0.02)
		HulC	0.028 (0.03)	0.040 (0.02)	0.867 (0.11)	0.914 (0.10)	0.206 (0.03)
ViT-B/32	87.5M	Bayesian-PEFT	0.054 (0.05)	0.060 (0.05)	0.702 (0.33)	0.764 (0.32)	0.183 (0.04)
		HIFI -ESM	0.011 (0.01)	0.044 (0.01)	0.933 (0.04)	0.955 (0.03)	0.319 (0.04)

Table 1: Statistical inference with HIFI and external baselines across pretrained vision models.

			Point Estimation		Statistical Inference
Model	Params	Method	MAB	MAE	Cov. 90%	Cov. 95%	95% CI Length
Qwen3- Embedding-8B	7.6B	HIFI -ESM	0.014 (0.01)	0.060 (0.01)	0.899 (0.03)	0.928 (0.03)	0.374 (0.06)
		HIFI -Bootstrap	0.015 (0.01)	0.064 (0.02)	0.885 (0.04)	0.937 (0.03)	0.316 (0.07)
		HulC	0.075 (0.05)	0.097 (0.04)	0.849 (0.10)	0.889 (0.09)	0.418 (0.06)
		Bayesian-PEFT	0.085 (0.06)	0.102 (0.05)	0.830 (0.22)	0.890 (0.18)	0.426 (0.05)
XLM-R XXL	10.7B	HIFI -ESM	0.014 (0.01)	0.051 (0.01)	0.899 (0.04)	0.926 (0.03)	0.361 (0.08)

Table 2: Statistical inference with HIFI and external baselines across pretrained language models.

Figure 2: Effect of Stage 1 task adaptation on MAB, 95% coverage, and CI length.

Figure 3: HIFI versus direct single-stage resampling under the same total data budget. HIFI training-compute savings measured in EFLOPs.

Figure 4: Gene-expression inference on Xenium tissue Section 2. (a) HIFI with ESM versus iStar across 313 genes by RMSE, SSIM, and PCC, with the iStar point estimate for TACSTD2 . (b) HIFI point estimate and 95% pointwise confidence interval for TACSTD2 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Examples of the synthetic vision data. Rows correspond to circles, squares, and equilateral triangles, respectively. Foreground and background colors, object positions, and polygon orientations vary across images.

Language	Model input $X$	$T(X)$ (min)	$\mu_{0}(X)$	$Y$
English	Kate walked into the terminal 15 minutes before the scheduled departure.	15	0.0962	0
English	Zack arrived at the airport 49 minutes before his flight.	49	0.3422	1
English	His flight DL8517 was assigned to gate B17. Henry reached the gate 1 hour and 16 minutes ahead of takeoff.	76	0.6472	0
English	American listed the flight as AA2789 from Terminal A. Frank reached the airport with 1 hour and 50 minutes left until the scheduled flight.	110	0.8996	1
Chinese	航班预定起飞前16分钟，周强已经抵达机场。 English translation: Zhou Qiang had already arrived at the airport 16 minutes before the scheduled departure.	16	0.1004	0
Chinese	航站楼里人特别多。她的航班是长荣BR6967。距起飞还剩50分钟时，林雪走进了候机厅。 English translation: The terminal was especially crowded. Her flight was EVA Air BR6967. With 50 minutes remaining before takeoff, Lin Xue entered the departure lounge.	50	0.3528	0

Appendix

Table 3: Illustrative observations from the bilingual text DGP at representative arrival lead times. English translations of the Chinese inputs are included only for readability.

Setting	ViT-B/32	DINOv2-B/14	Swin-B
Pooling	CLS + mean patch (concatenated)	CLS + mean patch (concatenated)	Global average
Prediction head	Two-layer MLP; 64 hidden units; GELU
Training loss	Mean squared error
Stage 1
Training / validation sample size	8,000 / 500
Blocks with LoRA	All 12	Last 4	All 24

Appendix

Table 4: Architecture, adaptation, and training settings for the vision experiments.

Setting	ViT-B/32	DINOv2-B/14	Swin-B
HulC
Training / validation size	9,000 / 700
Adapted blocks	Last 6	Last 2	Final stage (2 blocks)
LoRA configuration	Q and V; rank 8; scaling 16; dropout 0.05
Learning rate	$10^{-4}$	$5\times 10^{-5}$	$10^{-4}$
Training	AdamW; batch size 8; weight decay $10^{-4}$

Appendix

Table 5: Settings for the external vision comparators.

Setting	Qwen3-Embedding-8B	XLM-R XXL	NV-Embed-v2
Pooling	Last token	CLS token	Native latent attention
Maximum sequence length	96 tokens	96 tokens	112 tokens
Input prompt	Instruct: Given a passenger flight-status description, estimate the probability that the passenger makes the flight. Query: <text>
Prediction head	Two-layer MLP; 128 hidden units; GELU
Training loss	Binary cross-entropy with logits
LoRA configuration	Q and V projections; rank 16; scaling 32; dropout 0.05

Appendix

Table 6: Architecture, adaptation, and training settings for the text experiments.

Setting	Qwen3-Embedding-8B	XLM-R XXL	NV-Embed-v2
HulC
Training / validation size	9,000 / 700
Adapted blocks	Last 6	Last 4	Last 4
LoRA configuration	Q and V; rank 16; scaling 32; dropout 0.05
Learning rate	$4\times 10^{-5}$	$2\times 10^{-4}$	$3\times 10^{-4}$
Training	AdamW; batch size 8; weight decay 0

Appendix

Table 7: Settings for the external text comparators.

Figure 6: Target-wise empirical coverage of nominal 95% pointwise confidence intervals for the vision regression task. Each point represents one fixed evaluation target and gives the proportion of replications in which the corresponding interval contains

\mu_{0}(x)

. The dashed horizontal line marks the nominal 95% coverage level.

Figure 7: Target-wise mean lengths of nominal 95% pointwise confidence intervals for the vision regression task. Each point represents one evaluation target and reports the confidence-interval length averaged across replications. Boxplots summarize the distribution of mean interval lengths across targets.

Figure 8: For visualization, 50 targets are selected from the full target panel by stratified random sampling across shape classes and true-mean bins. Columns correspond to the three vision backbones, and rows correspond to HIFI-ESM, HIFI-Bootstrap, HulC, and Bayesian-PEFT. Points show the mean estimated conditional mean for each target, and vertical segments show the mean lower and upper endpoints of the corresponding 95% intervals. The dashed diagonal line represents perfect agreement with the true conditional mean

\mu_{0}(x)

Figure 9: Target-wise empirical coverage of nominal 95% pointwise confidence intervals for the text probability-prediction task. Panels correspond to the three language backbones, and within each panel results are shown for HIFI-ESM, HIFI-Bootstrap, HulC, and Bayesian-PEFT. Each point represents one fixed evaluation target and gives the proportion of replications in which the corresponding interval contains the true conditional probability

\mu_{0}(x)

. Boxplots summarize the distribution across targets, and the dashed horizontal line marks the nominal 95% coverage level.

Figure 10: Target-wise mean lengths of nominal 95% pointwise confidence intervals for the text probability-prediction task. Each point represents one evaluation target and reports the confidence-interval length averaged across 100 replications. Boxplots summarize the distribution of mean interval lengths across targets.

Figure 11: Target-level probability estimation and uncertainty for 50 evaluation targets selected by stratified random sampling across languages and true-probability bins. Columns correspond to the three language backbones, and rows correspond to HIFI-ESM, HIFI-Bootstrap, HulC, and Bayesian-PEFT. Points show the estimated conditional probability averaged across replications for each target, and vertical segments show the corresponding mean lower and upper endpoints of the 95% intervals. The dashed diagonal line represents perfect agreement with the true conditional probability

\mu_{0}(x)

Figure 12: Effect of Stage 1 task-adaptation sample size on 90% coverage and CI length. The dotted line marks the nominal 90% coverage level.

Figure 13: Target-wise 95% coverage (top) and mean 95% confidence-interval length (bottom) for ESM and bootstrap across the single-stage baseline and HIFI with

n_{1}\in\{2{,}000,4{,}500,8{,}500\}

. Each point represents one of the 424 evaluation targets, with coverage and interval length computed across 100 replications.

Figure 14: Target-wise point estimates and mean endpoints of 95% confidence intervals for 50 representative targets, selected by stratified random sampling across languages and true-probability bins. Columns correspond to ESM and bootstrap; rows show the single-stage baseline and HIFI with

n_{1}\in\{2{,}000,4{,}500,8{,}500\}

. Points denote estimates averaged over 100 replications, and vertical segments connect the mean lower and upper confidence-interval endpoints. The dashed line is the identity line.

Figure 15: HIFI versus direct single-stage resampling under the same total data budget at the 90% confidence level.

Figure 16: Target-wise empirical coverage of nominal 95% confidence intervals for HIFI and direct single-stage resampling across backbones and inference methods. Each point represents one fixed evaluation target.

Figure 17: Target-wise lengths of nominal 95% confidence intervals for HIFI and direct single-stage resampling across backbones and inference methods.

Backbone	Method	Procedure	GPU	Training EFLOPs/rep	Wall-clock time/rep
ViT-B/32	ESM	Single-stage	A40	0.2934	2 h 44 min
	ESM	HIFI	RTX 3090	0.0239	15 min
	Bootstrap	Single-stage	A40	0.4685	2 h 46 min
	Bootstrap	HIFI	RTX 3090	0.0668	32 min
DINOv2-B/14	ESM	Single-stage	A40	0.3174	1 h 48 min
	ESM	HIFI	RTX 3090	0.0649	37 min

Appendix

Table 8: Training compute and wall-clock time for HIFI and single-stage configurations across tasks, backbones, and inference methods.

Shared setting	Specification
Input features	192 HViT-4K + 384 HViT-256 + 3 RGB + 2 coordinates
MLP layer dimensions	$581\to 256\to 256\to 256\to 256\to 313$
Hidden activation	Leaky ReLU with negative slope 0.1
Trainable MLP layers	All hidden layers and the output layer
LoRA targets	QKV and attention output projections
LoRA parameters	Rank 8; scaling 16; dropout 0

Appendix

Table 9: HIFI settings for the Xenium experiment. Batch sizes and resampling sample sizes count pseudo-Visium spots.

Figure 18: Point-prediction comparison between HIFI with bootstrap and iStar at

16\,\mu\mathrm{m}

resolution. Each point represents one of the 313 genes, and dashed lines indicate equality. Lower RMSE and higher SSIM and PCC indicate better agreement with observed expression.

Figure 19: Additional ESM interval maps for KRT7 and GATA3 . Columns show observed expression, the lower endpoint, the point estimate, and the upper endpoint of the nominal 95% pointwise interval for mean expression. The corresponding TACSTD2 example appears in Figure 4 .

Figure 20: Bootstrap interval maps for KRT7 , GATA3 , and TACSTD2 . Columns show observed expression, the lower percentile endpoint, the bootstrap mean, and the upper percentile endpoint. The endpoints correspond to nominal 95% pointwise intervals. Each gene uses the same display mapping as its ESM panels.

Efficient Inference

Efficient inference reduces the compute, memory, latency, or energy needed to produce accurate predictions, posterior estimates, and decisions.

See all 114 papers

Can Tabular Foundation Models Amortize Statistical Inference?
Sep 27, 2026Kai Ye, Shijin Gong, Hongyi Zhou +2Amortized Inference Confidence Region Estimation
Department of Statistics, London School of Economics and Political Science, London, UK · School of Management, University of Science and Technology of China, Hefei, China · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China +1
Streaming Hierarchical Inference with Tabular Foundation Models
Sep 7, 2026Vitor Crista, Afonso Lourenço, Diogo Martinho +1Tabular Foundation Models Efficient Inference
GECAD, Polytechnic of Porto, Portugal
Context-Adaptive Inference: A Unified Statistical and Foundation-Model View
Jul 25, 2026Yue Yao, Caleb N. Ellington, Jingyun Jia +9Adaptive Inference Meta-Learning
Department of Statistics, University of Wisconsin-Madison · Computational Biology Department, Carnegie Mellon University · Department of Computer Science, Yale University +5

Sep 27, 2026J/K move · Enter open · S save

Can Tabular Foundation Models Amortize Statistical Inference?

Kai Ye, Shijin Gong, Hongyi Zhou, Valentina Zangirolami+1

Department of Statistics, London School of Economics and Political Science, London, UK · School of Management, University of Science and Technology of China, Hefei, China · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China+1

Amortized Inference Confidence Region Estimation Efficient Inference Tabular Foundation Models

Open paper AskPDF arXiv

For decades, statistical inference has largely been developed one problem at a time. Given a scientific target, such as a treatment effect or a regression function, statisticians design a problem-specific estimator together with a procedure for quantifying its uncertainty. This paper proposes a different paradigm. We focus on a classical problem in statistical inference, confidence interval construction, and develop TabCon, an amortized inference system built on a tabular foundation model that produces confidence intervals for new datasets through a simple forward pass. The key methodological ingredients of TabCon are a sparse mixture-of-experts architecture and reinforcement-learning-based post-training that calibrate the resulting confidence intervals to a desired coverage level. Across a wide range of benchmark datasets, TabCon attains near-nominal coverage while producing short confidence intervals. At inference time, it also offers considerably greater computational efficiency, running 50 times faster than the classical bootstrap procedure, even when the latter uses only 50 bootstrap samples.

Abstract

Efficient Inference

Explore similar work

Can Tabular Foundation Models Amortize Statistical Inference?

Streaming Hierarchical Inference with Tabular Foundation Models

Context-Adaptive Inference: A Unified Statistical and Foundation-Model View