Generative Score Inference for Multimodal Data

Xinyu Tian; Xiaotong Shen

stat.MLMar 27, 2026

Generative Score Inference for Multimodal Data

Authors: Xinyu Tian, Xiaotong Shen

Organizations: School of Statistics, University of Minnesota

Abstract

Accurate uncertainty quantification is crucial for making reliable decisions in various supervised learning scenarios, particularly when dealing with complex, multimodal data such as images and text. Current approaches often face notable limitations, including rigid assumptions and limited generalizability, constraining their effectiveness across diverse supervised learning tasks. To overcome these limitations, we introduce Generative Score Inference (GSI), a flexible inference framework capable of constructing statistically valid and informative prediction and confidence sets across a wide range of multimodal learning problems. GSI utilizes synthetic samples generated by deep generative models to approximate conditional score distributions, facilitating precise uncertainty quantification without imposing restrictive assumptions about the data or tasks. We empirically validate GSI's capabilities through two representative scenarios: hallucination detection in large language models and uncertainty estimation in image captioning. Our method achieves state-of-the-art performance in hallucination detection and robust predictive uncertainty in image captioning, and its performance is positively influenced by the quality of the underlying generative model. These findings underscore the potential of GSI as a versatile inference framework, significantly enhancing uncertainty quantification and trustworthiness in multimodal learning.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of (a) the QA system and (b) the image captioning model.

	MEPS–20			Bio			Kin8nm			Naval			Blog
	$C_{\text{marg}}$	$L_{\text{avg}}$	$C_{G}$	$C_{\text{marg}}$	$L_{\text{avg}}$	$C_{G}$	$C_{\text{marg}}$	$L_{\text{avg}}$	$C_{G}$	$C_{\text{marg}}$	$L_{\text{avg}}$	$C_{G}$	$C_{\text{marg}}$	$L_{\text{avg}}$	$C_{G}$
CP	0.90	1.24	0.15	0.90	2.42	0.85	0.90	2.17	0.83	0.89	1.31	0.78	0.89	1.89	0.57
CUQR	0.89	1.21	0.76	0.90	2.40	0.88	0.89	2.19	0.85	0.86	1.26	0.85	0.87	1.82	0.67
GSI	0.91	0.98	0.80	0.89	2.19	0.88	0.90	1.82	0.85	0.89	0.96	0.85	0.90	1.66	0.80

Table 1: Performance on five benchmarks at the nominal

90\%

level, summarized by marginal coverage (

C_{\text{marg}}

), average interval length (

L_{\text{avg}}

), and worst-case subgroup coverage (

C_{G}

). Higher coverage and shorter intervals indicate better performance. Baseline results for CP and CUQR are reproduced from ( Alaa, Hussain and Sontag, 2023 ) . The boldface marks the best result in each metric.

Figure 3: Type I error and power comparison for SE ( Farquhar et al., 2024 ) , CA ( Gui, Jin and Ren, 2024 ) , and GSI across varying $\alpha\in(0,1)$ . Smaller $\alpha$ indicates higher confidence $(1-\alpha)$ . The diagonal line $y=x$ represents ideal Type I error control.

Question	Reference Answer	Generated answer	Score	SE	CA	GSI
Who is on blink 182 album cover?	The cover artwork for Enema of the State features porn star Janine Lindemulder famously clad in a nurse uniform.	The band members are on the Blink-182 album cover.	0.883	0.028	0.007	0.117*
		The Blink-182 album cover features the band members as children.		( $\leq$ 0.143)	( $\leq$ 0.102)	(¿0.078)
		The album cover features the faces of the band members.
What is a popular people meter?	A people meter is an audience measurement tool used to measure the viewing habits of TV and cable audiences.	A popular people meter is a device used to track television viewership and audience preferences.	0.418	0.028	0.0291	0.045
		A popular people meter is an electronic device used to measure TV ratings and viewer preferences.		( $\leq$ 0.143)	( $\leq$ 0.102)	( $\leq$ 0.078)
		A popular people meter is a device used in television ratings to measure the viewing habits.

Table 2: Illustrative examples of generated answers and their testing results for given questions in the Q&A task. The ’*’ indicates a rejection, i.e. the probability

P(z=1)

given in the method larger than the threshold

\alpha_{*}

, which is derived from the validation set under the significance level

\alpha=0.05

Figure 4: FDR and power for CA ( Gui, Jin and Ren, 2024 ) and GSI across target FDR levels; the line $y=x$ represents perfect FDR control.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Search Space
hidden_dim	{64, 128, 256, 512, 1024}
time_embed_dim	{64, 128, 256, 512}
layers	Integer in [3, 15]
dropout_p	Float in [0.0, 0.05]
learning_rate	Log-uniform in $[10^{-5},10^{-4}]$
noise_steps	Integer in [300, 1000]

Appendix

Table 3: Hyperparameter search space for diffusion model training.

Parameter	Search Space
learning_rate	{0.1, 0.2}
max_depth	{3, 5}
min_child_weight	{3, 5}
subsample	{0.7, 1.0}
colsample_bytree	{0.7, 1.0}
n_estimators	{100, 200}

Appendix

Table 4: Grid search space for XGBoost hyperparameters.

Method	Additional tuning parameters
CVAE	latent_dim $\in\{4,8,16\}$ , beta_kl $\in\{0.001,0.01,0.05\}$ .
CGAN	latent_dim $\in\{4,8,16\}$ , recon_weight $\in\{5,10,20\}$ .
CFlow	No additional method-specific parameter beyond the shared parameters.

Appendix

Table 5: Generator-specific tuning parameters. The shared parameters hidden_dim , layers , dropout_p , learning_rate , and batch_size use the same search spaces as those in Table 3 .

Dataset	Method	$C_{\text{marg}}$	$C_{G}$	$L_{\text{avg}}$	Train time (s)	Inference time (s)
Meps_20	Conditional Diffusion	0.91	0.80	0.98	214.66	230.03
	Conditional VAE	0.88	0.60	0.83	20.71	0.05
	Conditional GAN	0.77	0.00	0.55	42.06	0.06
	Conditional flow	0.91	0.40	0.96	17.78	0.11
Bio	Conditional Diffusion	0.89	0.88	2.19	203.29	106.51
	Conditional VAE	0.90	0.88	2.38	140.92	0.16

Appendix

Table 6: Additional generator comparison across all five tabular datasets.

Task	Dataset	Method	Train time (s)	Inference time (s)
Tabular Prediction	Meps_20	GSI	214.66	230.03
		CP	–	–
		CUQR	0.02	4.56
	Bio	GSI	203.29	106.51
		CP	–	–
		CUQR	0.03	4.71

Appendix

Table 7: Training and inference time comparison across tabular, Question&Answer, and Image Captioning tasks. For the tabular tasks, we report the diffusion-based GSI timing together with the mean training and test time of the new TCP and CP runs. Entries smaller than 0.01 seconds are replaced with “–” and treated as negligible.

LLM Uncertainty Estimation

Large language model uncertainty estimation measures when outputs may be wrong, ambiguous, or unreliable to guide safer decisions.

See all 399 papers

Unified Multimodal Uncertain Inference
Apr 9, 2026Dengjia Zhang, Alexander Martin, William Jurayj +3Probability Calibration Multimodal Reasoning
Johns Hopkins University · Human Language Technology Center of Excellence
Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Feb 27, 2026Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin +1LLM Uncertainty Estimation Multimodal Large Language Models
Department of Computer Science, National University of Singapore
Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models
Sep 1, 2026Soroush Seifi, Vaggelis Dorovatas, Lin Li +2Uncertainty Quantification Multimodal Large Language Models
Toyota Motor Europe · University of Oxford

Apr 9, 2026J/K move · Enter open · S save

Unified Multimodal Uncertain Inference

Dengjia Zhang, Alexander Martin, William Jurayj, Kenton Murray+2

Johns Hopkins University · Human Language Technology Center of Excellence

Probability Calibration Multimodal Reasoning Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

We introduce Unified Multimodal Uncertain Inference (UMUI), a multimodal inference task spanning text, audio, and video, where models must produce calibrated probability estimates of hypotheses conditioned on a premise in any modality or combination. While uncertain inference has been explored in text, extension to other modalities has been limited to single-modality binary entailment judgments, leaving no framework for fine-grained probabilistic reasoning in or across other modalities. To address this, we curate a human-annotated evaluation set with scalar probability judgments across audio, visual, and audiovisual settings, and additionally evaluate on existing text and audio benchmarks. We introduce CLUE (Calibrated Latent Uncertainty Estimation), which combines self-consistent teacher calibration and distribution-based confidence probing to produce calibrated predictions. We demonstrate that our 3B-parameter model achieves equivalent or stronger performance than zero-shot baselines up to 32B parameters across all modalities.

Abstract

LLM Uncertainty Estimation

Explore similar work

Unified Multimodal Uncertain Inference

Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume

Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models