cs.LGNov 9, 2026

When Are Predictions Enough? An Evaluation Protocol for Frozen Expert Composition

Authors: Alessandro Pereira, Keith J Ransom, Lewis Mitchell

Organizations: Australian Institute for Machine Learning (AIML), Adelaide University · School of Computer Science and Information Technology, Adelaide University · Adelaide Data Science Centre (ADSC), School of Mathematical Sciences, Adelaide University

Abstract

Practitioners often combine independently trained models while keeping their weights frozen. These experts are commonly combined through their final predictions, although their internal features may retain useful information that those predictions omit. We ask when combining features improves on combining predictions, and how to make that comparison fair. We introduce a reusable evaluation protocol, apply it to 21 frozen deepfake detectors from DF40 (Yan et al., 2024), and audit how evaluation choices change the conclusions. The protocol uses strong prediction baselines, compares simple and larger feature combiners, varies architecture and training data separately, and reports results by held-out generator with train/test overlap controls. Under leave-one-generator-out evaluation, logistic regression on the features of six architecture-diverse experts improves macro AUROC by 0.081 over the best tested prediction combiner in each fold, with gains on seven of eight generators. The larger network shows no clear improvement over the linear model (approximately 3.5M versus 10K parameters), suggesting that access to richer information matters more than additional combiner capacity on this pool. Features may help when they retain information that the predictions omit and that transfers to new generators. Predictions may be enough when the strongest prediction baseline already performs well, while neither approach reliably compensates for poor expert transfer. Weakening prediction baselines on the overlap-affected standard split inflates the apparent feature advantage by up to 0.172 AUROC. Overlap-aware evaluation also changes the estimated gain, though the contrast does not isolate overlap. Results by generator and expert pool show where the advantage holds, weakens, or reverses. Without these controls, the same benchmark can support different conclusions.

Explore similar work

CardsList
  1. Honest Fusion Evaluation for Cross-Domain Deepfake Detection: Leakage Audits and Tail Robustness

    Nov 11, 2026Luke Collins, Chang-Tsun Li, Duc Thanh Nguyenscore fusionDeepfake Detection

  2. R2^2M: Real-Aware Residual Model Merging for Robust and Generalizable Deepfake Detection

    Sep 29, 2025Jinhee Park, Guisik Kim, Choongsang Cho +1Domain GeneralizationDeepfake Detection