Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means

Raphaël Razafindralambo; Rémy Sun; Frédéric Precioso; Damien Garreau; Pierre-Alexandre Mattei

stat.MLMar 4, 2026

Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means

Authors: Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Damien Garreau, Pierre-Alexandre Mattei

Organizations: Université Côte d’Azur, Inria, CNRS, I3S/LJAD, Maasai, Nice, France · Julius-Maximilians-Universität Würzburg, Institute for Computer Science / CAIDAS, Würzburg, Germany

Abstract

Density aggregation is a central problem in machine learning, for instance when combining predictions from a Deep Ensemble. The choice of aggregation remains an open question with two commonly proposed approaches being linear pooling (probability averaging) and geometric pooling (logit averaging). In this work, we address this question by studying the normalized generalized mean of order $r \in \mathbb{R} \cup \{-\infty,+\infty\}$ through the lens of log-likelihood, the standard evaluation criterion in machine learning. This provides a unifying aggregation formalism and shows different optimal configurations for different situations. We show that the regime $r \in [0,1]$ is the only range ensuring systematic improvements relative to individual distributions, thereby providing a principled justification for the reliability and widespread practical use of linear ( $r=1$ ) and geometric ( $r=0$ ) pooling. In contrast, we show that aggregation rules with $r \notin [0,1]$ may fail to provide consistent gains with explicit counterexamples. Finally, we corroborate our theoretical findings with empirical evaluations using Deep Ensembles on image and text classification benchmarks.

Figures & tables

Figure 1: Only intermediate powers

r

, particularly

r\in[0,1]

, yield consistent NLL improvements, whereas extreme values can degrade performance. Top: aggregated densities

\overset{\circ}{p}_{2}

obtained from two Gaussian experts

p^{(1)}

and

p^{(2)}

for various

r

. Small

r

concentrates mass in a single region, while large positive

r

preserves the bimodal structure of the experts. Bottom: NLL evaluated on samples

y\sim\mathcal{N}(0,4^{2})

(purple ticks indicate test samples). In this setting, negative

r

(min-type behavior) underperforms the average individual NLL (dashed line), whereas positive

r

achieves the lowest values. Our theory ( Theorem 3.1 ) identifies

r\in[0,1]

as the provably reliable interval. See Appendix B for different behaviors.

Panel of Figure 2 — Figure 2: Visual illustration of Theorem 3.2 . Aggregation fails at points where densities strongly differ when $r<0$ , and at consensus points when $r>1$ . We look at the location of the gap in Equation 20 across $r$ regimes. “Avg log” denotes the average of individual log-likelihoods (right-hand side of Equation 20 ). Left (a): We consider two gaussians $\mathcal{N}(\pm 2.5,1)$ . Center (b): For $r<0$ , the inequality of Theorem 3.1 is satisfied at $x=2.5$ . We observe that the gap is not confined and amplifies when $x$ increases. Right (c): For $r\geq 1$ , the AM ( $r=1$ ) globally dominates the average log-density, while all $r>1$ exhibit a very localized (but non-punctual) negative effect around $x=0$ .

Panel of Figure 3 — Figure 3: Illustration of the cross-entropy behavior of the normalized power mean likelihood $\overset{\circ}{p}_{k,r}$ (via cross-entropy) in three classification settings. All plots exhibit a U-shaped performance curve: extreme aggregation amplifies model disagreement, whereas optimal values lie in $[0,1]$ . Consistent with Theorem 3.1 , the regime $r\in[0,1]$ remains reliably below the individual model uncertainty band. By contrast, negative orders ( $r<0$ ) perform poorly, likely due to disagreements on dominant classes, as discussed in Section 3.3 .

Panel of Figure 4 — Figure 4: Illustration of the cross-entropy behavior of the normalized power mean likelihood $\overset{\circ}{p}_{k,r}$ , focusing on values of $r$ around $[0,1]$ (zoomed view of Figure 3 ). On MedMNIST (b) and IMDb (c), the optimal likelihood lies within the reliability interval $[0,1]$ ( Theorem 3.1 ), whereas on CIFAR-100 (a) it lies slightly beyond it ( $\approx 1.4$ ). This shows that while the interval $[0,1]$ provides a stable and reliable regime, mild optimism outside it can still be beneficial in practice.

Figure 5: Illustration of the cross-entropy behavior of the normalized power mean likelihood

\overset{\circ}{p}_{k,r}

on CIFAR-100 in a controlled near-consensus regime. The trend contrasts with Figure 3 : extreme optimistic aggregation becomes harmful and the maximum operator performs worst.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Effect of decreasing

\sigma

on aggregation behavior. From top to bottom: large (

\sigma=3

), moderate (

\sigma=2

), small (

\sigma=1

), and very small

\sigma

(

\sigma=0.2

). The U-shaped behavior first sharpens, then all

r

outperform the individual baseline for intermediate

\sigma

, before the trend reverses for very small

\sigma

Figure 7: Effect of

\mu_{1}

and

\mu_{2}

on aggregation performance. From top to bottom: merged experts, moderately separated experts, and highly separated experts. When

\mu_{1}

and

\mu_{2}

are close, disagreement regions are frequent, which explains the poor performance of min-like aggregations. As the means separate, this effect diminishes as test samples concentrate on the modes of the generalized mean.

Ensemble Learning

Ensemble learning combines multiple predictors to improve accuracy, robustness, calibration, or uncertainty estimates.

See all 272 papers

Geometric Mean Pooling for Equal-Weight Multiplicative Coarse-Graining
Sep 18, 2026Ang-Kun Wu, Fangdi Wen, Jingtao Zhang
Department of Physics and Astronomy, University of Tennessee, Knoxville Knoxville, Tennessee 37996, USA · Department of Physics and Astronomy, Rutgers University New Brunswick, New Jersey 08901, USA · Google Mountain View, CA 94043, USA
A Distributional Optimisation Perspective on Combining Models in Deep Learning
Sep 21, 2026Congye Wang, Yan Lin, Zheyang Shen +2Ensemble Learning Deep Ensembles
Newcastle University, UK
Optimal Rates for Agentic Networked Information Aggregation
Sep 4, 2026MohammadHossein Bateni, Zahra Hadizadeh, MohammadTaghi Hajiaghayi +2Statistical Learning Theory Decentralized Learning
Google Research · University of California, Irvine · University of Maryland

Sep 18, 2026J/K move · Enter open · S save

Geometric Mean Pooling for Equal-Weight Multiplicative Coarse-Graining

Ang-Kun Wu, Fangdi Wen, Jingtao Zhang

Department of Physics and Astronomy, University of Tennessee, Knoxville Knoxville, Tennessee 37996, USA · Department of Physics and Astronomy, Rutgers University New Brunswick, New Jersey 08901, USA · Google Mountain View, CA 94043, USA

Open paper AskPDF arXiv

As an alternative to the additive and extremal biases of average and max pooling, we introduce Geometric Mean Pooling (GMP), a signed pooling operator that combines the product of feature signs with the geometric mean of feature magnitudes. Motivated by local-to-global composition in quantum many-body physics, GMP retains both joint sign information and a characteristic multiplicative scale without introducing learnable pooling parameters. We show that non-overlapping hierarchical GMP preserves the corresponding global multiplicative statistic and evaluate it on synthetic sequence tasks, iterative coarse-graining, image classification, and molecular lipophilicity regression. On the synthetic tasks, GMP recovers product-based signals more accurately than average and max pooling and maintains predictive performance under the tested levels of multiplicative input noise. On image and molecular data, however, its effectiveness depends on the representation, target parameterization, and placement of local and global pooling. These results position GMP as a complementary, regime-dependent inductive bias for tasks in which equal-weight multiplicative composition is plausible, rather than as a universal replacement for standard pooling operators.

Abstract

Ensemble Learning

Explore similar work

Geometric Mean Pooling for Equal-Weight Multiplicative Coarse-Graining

A Distributional Optimisation Perspective on Combining Models in Deep Learning

Optimal Rates for Agentic Networked Information Aggregation