cs.CRNov 9, 2026

Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing

Authors: Andrew Trask, Sol Messing, Vai Pahwa, Pegah Maham, Rene Kolga, Austin Frantz, Andrew Tash, Kate Thomas, 

Organizations: Google · OpenMined · AVERI · Singapore AISI · 

Abstract

AI safety evaluations often involve sensitive prompts and proprietary models. Evaluators want to keep their evaluation prompts secret from model owners to prevent purposeful or incidental data leakage, ensuring longer usefulness of their benchmark. Model owners in turn wish to keep their proprietary model weights and inference code within their own secure infrastructure to protect their IP. We present a way forward for "double-blind evals", which provide cryptographic guarantees that an evaluator's prompts are not shared with the model owner, while simultaneously maintaining the confidentiality of the model weights and inference code. To accomplish this, we use secure enclaves that encrypt data in RAM at the hardware level. In a world-first pilot, we test this approach by evaluating a proprietary Google DeepMind model (Gemini 2.5 Flash Lite) against a selection of the private AILuminate benchmark corpus, using an NVIDIA H100 secure enclave on Google Cloud Platform, and using OpenMined's PySyft to maintain data and model privacy. We believe this effort marks the first use of this technology to provide mutual secrecy for one organization's frontier-class proprietary model weights and another organization's closed benchmark.

Figures & tables

Explore similar work

CardsList
  1. Making AI-Assisted Grant Evaluation Auditable without Exposing the Model

    Apr 28, 2026Kemal BicakciLLM AuditingPrivacy-Preserving ML

  2. Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI

    Mar 8, 2026Heng Jin, Chaoyu Zhang, Hexuan Yu +4Verifiable Outsourced ComputationLLM Auditing

  3. Evaluation Awareness Shifts from Format to Context with Model Scale

    Aug 20, 2026Navraj Singh, Maheep ChaudharyEvaluation Awareness in Language ModelsLLM Interpretability