ALICE: In-context, Zero-shot, Mutual Information Estimation

Giulio Franzese; Simone Rossi; Pietro Michiardi

cs.LGSep 28, 2026

ALICE: In-context, Zero-shot, Mutual Information Estimation

Authors: Giulio Franzese, Simone Rossi, Pietro Michiardi

Organizations: EURECOM, Sophia Antipolis, France

Abstract

Estimating mutual information (MI) from samples is a central objective in a variety of scientific fields. Modern neural estimators are accurate in the large-data regime, but they fall short when data is scarce, and each must be fit anew for every distribution under study. Current estimators are moreover tied to specific data types. These constraints limit their adoption in many applications where per-distribution training is impractical and sample sizes are small. We present ALICE, a foundation model that removes per-distribution training, while achieving competitive estimation accuracy. Trained exclusively on a broad family of synthetic distributions, ALICE acts as an in-context estimator of rectified-flow velocity fields: conditioned on samples of an unseen distribution, it estimates that distribution's velocity field without any explicit training. MI is then obtained through a fixed identity that integrates the squared difference between the joint and conditional fields. We validate ALICE on a standard, challenging benchmark and apply it in three domains, biology, genetics, and neuroscience, whose data the model has never seen. For the first time, we show that a single model closes the gap with neural estimators trained separately for each distribution, while natively supporting different data dimensionality and sample cardinality, enabling zero-shot MI analysis across scientific domains.

Figures & tables

Figure 2 : Category-wise MAE on the

40

-task Czyż et al. (2023) benchmark at matched data budgets. Rows show budgets of

1

5

k, and

10

k samples, and columns group tasks by base family or transformation. Upward triangles mark values above the plotted range. Stars mark the lowest MAE.

Figure 3 : MAE against inference FLOPs per task and seed on the

40

-task Czyż benchmark; labels show Alice total budgets (

64

queries).

Figure 4 : In-context analysis of the NF-

\mathcal{K}

B dose channel. (a) Median nuclear NF-

\mathcal{K}

B response per TNF-

\alpha

dose, with the interquartile band at the two extreme doses. (b) Capacity of each single frame and (c) of the prefix of frames from minute

0

, with the envelope over seeds. The pairwise discrimination matrices are in Appendix G .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7 : One forward pass of Alice . Clean context values determine the relation graph, whose gated edges $A_{ij}$ condition every graph layer (dashed). Context and query cells pass through the same shared projection and input graph layer; $K$ learned latents per coordinate read the encoded context twice through cross-attention, and $L$ blocks alternate self-attention among one coordinate’s latents with graph attention across coordinates. A query decodes in one pass, cross-attending to the latents for the global summary and to the encoded context for local detail, and a shared scalar head produces one output per coordinate; two head evaluations, at times $t$ and $0$ , form the velocity through the boundary parameterization. The context side, left on the figure, is encoded once per context and cached; every interaction with the $n$ context samples is linear in $n$ .

Panel of Figure 8 — Figure 8 : Alice relation-graph construction and coordinate mixing. Node and block colors identify coordinates, and each two-color block denotes a source-target pair $j\to i$ . Top: shared feature maps $\ell,r$ aggregate paired clean-context rows into signed, gated relation weights $A_{ij}^{(h)}$ . Bottom: for the green target $i$ , each source representation $\mathbf{r}_{j}^{(h)}$ is weighted by its incoming edge, and the center module sums and normalizes these contributions to produce $u_{i}^{(h)}$ for the residual update.

Panel of Figure 9 — Figure 9 : Induced context bottleneck. The learned bank $U$ is broadcast across coordinates. For each coordinate, the first cross-attention layer uses $U$ as queries and that coordinate’s context tokens as keys and values; the second uses the updated latents as queries and the same context tokens as keys and values, producing $K$ context-specific latent vectors.

Config name	Alice Small	Alice Base
Model hidden dimension	$384$	$768$
Layers	$5$	$6$
Attention heads	$8$	$12$
Feed-forward dimension	$1536$	$3072$
Time frequencies	$16$	$16$
Dropout	$0.1$	$0.1$

Appendix

Table 1: The two Alice variants, with model configuration values and parameter counts for each preset.

Foundation models
Estimator	$1$ k	$5$ k	$10$ k
ALICE Base	$0.092\pm 0.004$	$0.063\pm 0.001$	$0.060\pm 0.001$
ALICE Small	$0.101\pm 0.005$	$0.103\pm 0.002$	$0.104\pm 0.001$
InfoAtlas	$0.253\pm 0.004$	$0.271\pm 0.005$	$0.276\pm 0.001$
Neural estimators
MINDE– c	$0.353$	$0.070$	$0.065$

Appendix

Table 2 : Mean absolute error in nats over the

40

tasks of the Czyż benchmark at matched budgets of

1

5

k, and

10

k samples per task. Alice and InfoAtlas condition on a context of

N-64

samples from that budget, and their cells give the mean and sample standard deviation over eight independent context draws; neural estimators are trained per distribution on that number of samples and classic estimators are fit on it, both as five-seed means. “

>10

” marks a diverged estimator.

Model	$N$	All ( $40$ )	dims $\leq 10$ ( $33$ )	dims $50/100$ ( $7$ )
ALICE Base	$1000$	$0.092\pm 0.004$	$0.079\pm 0.004$	$0.153\pm 0.008$
	$5000$	$0.063\pm 0.001$	$0.057\pm 0.002$	$0.090\pm 0.004$
	$10000$	$0.060\pm 0.001$	$0.054\pm 0.001$	$0.086\pm 0.002$
ALICE Small	$1000$	$0.101\pm 0.005$	$0.081\pm 0.005$	$0.197\pm 0.017$
	$5000$	$0.103\pm 0.002$	$0.077\pm 0.001$	$0.228\pm 0.009$
	$10000$	$0.104\pm 0.001$	$0.074\pm 0.002$	$0.241\pm 0.006$

Appendix

Table 3 : Czyż benchmark accuracy by joint width against the sample budget

N

for both Alice sizes and InfoAtlas. “dims

\leq 10

” aggregates the

33

tasks of joint width at most

10

and “dims

50/100

” the

7

wider ones. Cells give the mean and sample standard deviation of the per-draw MAE over eight independent context draws. The Alice training window is

2048

samples, so the rows at

5000

and

10000

are context extrapolation.

Foundation models
GT	0.22	0.43	0.29	0.45	0.41	0.41	0.41	1.02	1.02	1.02	1.02	0.29	1.02	1.29	1.02	0.41	1.02	0.59	1.62	0.41	1.02	1.02	1.02	1.02	1.02	1.02	1.02	1.02	1.02	0.22	0.43	0.19	0.29	0.18	0.45	0.30	0.41	1.71	0.33	0.41
ALICE Base	0.26	0.42	0.30	0.44	0.42	0.42	0.42	1.03	1.07	1.09	1.06	0.29	1.03	1.44	1.07	0.43	1.10	0.64	1.85	0.42	1.03	1.08	1.09	0.84	0.59	0.58	1.00	0.96	1.01	0.26	0.42	0.21	0.30	0.19	0.45	0.29	0.48	1.63	0.31	0.42
ALICE Small	0.23	0.46	0.29	0.37	0.49	0.49	0.48	1.27	1.10	1.06	1.15	0.34	1.26	1.62	1.10	0.48	1.05	0.71	2.06	0.48	1.26	1.11	1.06	0.89	0.69	0.55	1.09	1.03	1.01	0.23	0.45	0.20	0.30	0.17	0.38	0.23	0.55	1.85	0.35	0.48
InfoAtlas	0.18	0.33	0.22	0.34	0.40	0.39	0.40	0.02	0.98	0.97	0.96	0.28	0.02	0.46	0.98	0.39	0.96	0.56	0.46	0.39	0.02	0.98	0.96	0.02	0.49	0.61	0.02	0.87	0.88	0.04	0.07	0.15	0.22	0.14	0.34	0.23	0.10	1.10	0.17	0.40
Neural estimators
MINDE– c	0.19	0.39	0.26	0.41	0.40	0.40	0.40	0.97	0.98	0.96	1.01	0.29	1.04	1.20	0.99	0.43	0.99	0.61	1.51	0.39	1.01	0.93	0.92	0.89	0.81	0.79	1.01	0.91	0.91	0.10	0.17	0.17	0.23	0.17	0.39	0.28	0.25	1.62	0.29	0.40

Appendix

Table 4 : Per-task MI estimates on the Czyż et al. [2023] suite against ground truth (GT), in nats, with every estimator at a budget of

10

k samples per task. Cell shading encodes the signed bias of the estimate: blue for over-estimation, red for under-estimation, with saturation growing linearly up to a bias of

0.6

nats. Competitor rows report five-seed means. The Alice rows use zero-shot in-context estimation with a frozen model, and the Alice and InfoAtlas rows are means over eight independent context draws. Abbreviations: Mn multinormal, St Student-t, Nm normal, Hc half-cube, Sp spiral.

Figure 10 : Six-panel analysis of the NF- $\mathcal{K}$ B dose channel. (A–C) As in Figure 4 . (D–F) Pairwise probability of correct discrimination between doses: from the single frame at minute $21$ (D), from the trajectory (E), and the gain from dynamics (F, E minus D). The filled fraction of each circle and its color both encode the value, from chance ( $0.5$ ) to certain discrimination ( $1$ ) in D and E, and from $0$ to $0.25$ in F.

$k$	1	2	3	5	10	20	50	100
change	$-0.02$	$+0.07$	$+0.07$	$+0.07$	$+0.06$	$+0.05$	$+0.01$	$-0.03$
non-change	$+0.47$	$+0.38$	$+0.33$	$+0.25$	$+0.15$	$+0.10$	$+0.08$	$+0.07$

Appendix

Table 5: Within-session autocorrelation of the six-area vector at lag

k

(in flashes), mean over areas and sessions,

100

150

ms window.

Figure 11 : Complement of Figure 6 :

\Omega

-info of the six visual areas in the five

50

ms windows after a flash, one estimate per session. Top left: familiar-image sessions, one thin line per mouse and flash type, group medians in bold. Top right: difference between the novel and the familiar session of each mouse for non-change flashes. Bottom: difference between change and non-change flashes within each session, for familiar-image (left) and novel-image (right) sessions. Difference panels show the median, the interquartile band, and a signed-rank test per window (

*

p<0.05

**

p<0.01

***

p<0.001

). The plotted values are stored with the figure.

sessions	flash type	$0$ – $50$	$50$ – $100$	$100$ – $150$	$150$ – $200$	$200$ – $250$ ms
familiar ( $n=31$ )	change	$0.34$ [ $0.25$ , $0.47$ ]	$0.24$ [ $0.18$ , $0.32$ ]	$0.41$ [ $0.29$ , $0.59$ ]	$0.38$ [ $0.24$ , $0.56$ ]	$0.22$ [ $0.19$ , $0.33$ ]
familiar ( $n=36$ )	non-change	$0.34$ [ $0.22$ , $0.48$ ]	$0.25$ [ $0.18$ , $0.37$ ]	$0.41$ [ $0.25$ , $0.51$ ]	$0.69$ [ $0.50$ , $0.85$ ]	$0.35$ [ $0.28$ , $0.53$ ]
novel ( $n=32$ )	change	$0.16$ [ $0.12$ , $0.24$ ]	$0.83$ [ $0.69$ , $0.96$ ]	$1.08$ [ $0.88$ , $1.27$ ]	$0.66$ [ $0.51$ , $0.79$ ]	$0.37$ [ $0.24$ , $0.47$ ]
novel ( $n=36$ )	non-change	$0.17$ [ $0.11$ , $0.26$ ]	$0.55$ [ $0.45$ , $0.73$ ]	$0.60$ [ $0.48$ , $0.75$ ]	$0.49$ [ $0.39$ , $0.73$ ]	$0.28$ [ $0.22$ , $0.41$ ]

Appendix

Table 6: Median over sessions of the per-session

\Omega

-info (nats) of the six areas, context of

128

rows drawn from the same session, by window after flash onset. Interquartile range in brackets.

contrast	group	$0$ – $50$	$50$ – $100$	$100$ – $150$	$150$ – $200$	$200$ – $250$ ms
change $-$ non-change, within session	familiar ( $n=31$ )	$-0.01$ ( $0.8$ , $48\%$ )	$+0.01$ ( $0.8$ , $52\%$ )	$-0.00$ ( $0.8$ , $48\%$ )	$-0.26$ ( $8\cdot 10^{-6}$ , $10\%$ )	$-0.10$ ( $2\cdot 10^{-6}$ , $6\%$ )
change $-$ non-change, within session	novel ( $n=32$ )	$+0.01$ ( $0.9$ , $56\%$ )	$+0.26$ ( $3\cdot 10^{-5}$ , $75\%$ )	$+0.42$ ( $3\cdot 10^{-9}$ , $97\%$ )	$+0.10$ ( $3\cdot 10^{-3}$ , $81\%$ )	$+0.06$ ( $5\cdot 10^{-3}$ , $72\%$ )
novel $-$ familiar, within mouse	change ( $n=27$ )	$-0.16$ ( $8\cdot 10^{-4}$ , $15\%$ )	$+0.51$ ( $3\cdot 10^{-8}$ , $96\%$ )	$+0.56$ ( $7\cdot 10^{-8}$ , $93\%$ )	$+0.22$ ( $7\cdot 10^{-4}$ , $81\%$ )	$+0.11$ ( $1\cdot 10^{-3}$ , $81\%$ )
novel $-$ familiar, within mouse	non-change ( $n=36$ )	$-0.13$ ( $8\cdot 10^{-4}$ , $22\%$ )	$+0.30$ ( $2\cdot 10^{-8}$ , $86\%$ )	$+0.21$ ( $2\cdot 10^{-5}$ , $78\%$ )	$-0.15$ ( $0.01$ , $31\%$ )	$-0.07$ ( $0.09$ , $33\%$ )

Appendix

Table 7: Paired contrasts of the per-session estimates (nats): median difference, two-sided Wilcoxon signed-rank

p

-value, and fraction of positive differences, by window.

window (ms)	mouse	experience	flash type	seed	remainder
$0$ – $50$	$41\%$	$10\%$	$0\%$	$20\%$	$29\%$
$50$ – $100$	$21\%$	$40\%$	$3\%$	$13\%$	$24\%$
$100$ – $150$	$38\%$	$23\%$	$8\%$	$10\%$	$22\%$
$150$ – $200$	$54\%$	$0\%$	$1\%$	$14\%$	$31\%$
$200$ – $250$	$55\%$	$0\%$	$1\%$	$14\%$	$29\%$

Appendix

Table 8: Variance shares of the per-session estimates (all

63

sessions with both flash types, five context draws each) by window: sequential sums of squares of mouse identity, experience level, and flash type; the seed share is the variability of the estimator across context draws; the remainder holds interactions.

Explore similar work

May 29, 2026cs.LG

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas, a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving

100\times

speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis.

Zhengyang Hu, Yanzhi Chen, Hanxiang Ren +5

The University of Hong Kong · University of Cambridge · Microsoft +2

Mutual Information Temporal Dependence Foundation Model Atlas

Ask PDF

May 20, 2026cs.CL

PromptNCE: Conditional Probabilities and PMI Using Only LLMs and Contrastive Estimation Prompts

Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings. We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities. We construct a benchmark from three publicly available human-annotated datasets with ground-truth PMI, and evaluate five information-theoretic prompting-based estimators. Our main method, PromptNCE, frames conditional probability estimation as a contrastive task and augments the candidate set with an explicit OTHER category. The OTHER category allows the model to assign probability mass outside the candidate set, avoiding the closed-set normalization of standard contrastive prompts. PromptNCE gives the best conditional probability estimates on all three datasets. For full PMI, we find that estimating label base rates is the primary bottleneck on two of the three datasets, with the best methods reaching Spearman correlation up to 0.78. We also present a case study in computer science education showing how these estimators can be used to score student knowledge summaries in a low-data setting. We release our code and prompts.

Juliette Woodrow, Chris Piech

Department of Computer Science Stanford University

Mutual Information Prompt Engineering Contrastive Learning

Ask PDF

Jul 3, 2026cs.LG

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored. We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective that subsumes existing benchmarks as special cases. Within this framework, we propose two complementary families of tests: a copula-first family that systematically varies ground-truth MI, dimensionality, and marginal complexity using synthetic and flow-based transformations; and a marginals-first family that couples real-world image data with controlled dependency structures, extending the classic same-class-pairing paradigm. We use this suite to extensively evaluate three classes of estimators: non-parametric, discriminative, and generative. Contrary to prevailing assumptions, our results indicate that there is no universal winner: each category can systematically outperform all other estimators under specific setups. By analyzing these cases, we identify fundamental estimation barriers and propose new tests that more effectively stress these specific limitations. We share the open source code at https://github.com/VanessB/mutinfo.

Alberto Foresti, Ivan Butakov, Alexander Tolmachev +3

Department of Data Science EURECOM 450 Route des Chappes, 06410 Biot, France · Applied AI Institute Presnenskaya naberezhnaya, 12, 121205 Moscow, Russia · Moscow Independent Research Institute of Artificial Intelligence Bolshaya Cheremushkinskaya Street, 20, Building 4, Premises 3/1, 117218 Moscow, Russia

Mutual Information Gaussian Copula Structural Complexity

Ask PDF