LLM Olympiad: Why Model Evaluation Needs a Sealed Exam

Jan Christian Blaise Cruz; Alham Fikri Aji

cs.AIMar 24, 2026

LLM Olympiad: Why Model Evaluation Needs a Sealed Exam

Authors: Jan Christian Blaise Cruz, Alham Fikri Aji

Organizations: MBZUAI

Abstract

Benchmarks and leaderboards are how NLP most often communicates progress, but in the LLM era they are increasingly easy to misread. Scores can reflect benchmark-chasing, hidden evaluation choices, or accidental exposure to test content -- not just broad capability. Closed benchmarks delay some of these issues, but reduce transparency and make it harder for the community to learn from results. We argue for a complementary practice: an Olympiad-style evaluation event where problems are sealed until evaluation, submissions are frozen in advance, and all entries run through one standardized harness. After scoring, the full task set and evaluation code are released so results can be reproduced and audited. This design aims to make strong performance harder to ``manufacture'' and easier to trust.

Figures & tables

Figure 1: Comparison of evaluation formats across three dimensions. Open benchmarks are fully transparent but expose tasks to optimization and leakage. Shared tasks hide test labels but evaluate against a known target under non-standardized protocols. The proposed Olympiad format seals tasks until evaluation, enforces a standardized harness, and releases all artifacts afterward — combining the strengths of existing formats while mitigating their core weaknesses.

Figure 2: End-to-end flow of the proposed LLM Olympiad. Task authors propose problems via an open call; organizers curate and seal the task bundle before any model submission is evaluated. Model and system submitters prepare broadly without knowledge of task content and freeze their submissions in advance. Evaluation is run centrally under a standardized harness, with scores released to submitters and all artifacts — tasks, scoring code, and harness — released publicly afterward for community audit and reuse.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions
Aug 31, 2026Guangxiang Zhao, Qilong Shi, Xusen Xiao +13LLM Evaluation Scientific Reasoning
1Qiyuan Tech · 2Tsinghua University · 3The University of Hong Kong +1
What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks
Sep 15, 2026Chao WangLLM Evaluation Benchmark Design
Independent Researcher · Independent Researcher.

Aug 7, 2026J/K move · Enter open · S save

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

Sahil Pardasani, Madhusudan Singh

Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.

LLM Evaluation LLM-as-a-Judge LLM Auditing Language Model Bias Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable. Unverified claims that DeepSeek R1 outperformed OpenAI's o1 contributed to market panic on January 27, 2025, when Nvidia lost USD589 billion in market value. Yet vendor benchmarks often depend on an honor system. Academic reassessments and independent leaderboards have found undisclosed changes to proprietary models, contaminated training data, and selective reporting. LLM-as-a-judge methods scale evaluation by reducing human review. Studies, however, suggest that judges may show identity-aware bias, scoring an answer according to its source model rather than its quality. This bias has not been fully measured or corrected across politically sensitive, reasoning-intensive, and preference-based tasks. We examine this problem using seven verifier models: GPT-OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large3, and Sarvam M. They score anonymous and identity-disclosed responses from three primary models on 58 factual, reasoning, political, and preference-based questions. Identity disclosure slightly raises scores for factual questions, moderately affects stress-reasoning tasks, and causes large changes for geopolitically sensitive topics. Notable results include GLM5.1 (+7.00 points, p = 0.0249) and Llama 3.3 70B (+1.56 points, p = 0.00). We also introduce a blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger. In Phase 1, each judge records a one-way hash of its score and a secret salt before candidate identities are revealed. In Phase 2, the identity and raw score are disclosed and verified on-chain. This creates a tamper-evident audit trail that separates blind evaluation from post-hoc claims and reduces the verification burden on independent researchers and leaderboard operators.

Abstract

LLM Evaluation

Explore similar work

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions

What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks