cs.LGJan 24, 2025

Humanity's Last Exam

Authors: Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, 

Organizations: Michael Pokorny · Quinn · Tony · Quod

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Figures & tables

Figure 1: Compared against the saturation of some existing benchmarks, Humanity’s Last Exam accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities. The sources for our evaluation metrics are detailed in Section C.6 . We further evaluate more frontier models on HLE in Table 1 .
Figure 2: Samples of the diverse and challenging questions submitted to Humanity’s Last Exam .
Figure 3: HLE consists of 2,5002{,}500 exam questions in over a hundred subjects, grouped into high level categories here. We provide a more detailed list of subjects in Section B.4 .
Figure 4: Dataset creation pipeline. We accept questions that make frontier LLMs fail, then iteratively refine them with the help of expert peer reviewers. Each question is then manually approved by organizers or expert reviewers trained by organizers. A private held-out set is kept in addition to the public set to assess model overfitting and gaming on the public benchmark.
Pre-Release ModelsAccuracy (%) ↑\uparrowCalibration Error (%) ↓\downarrow
GPT-4o2.72.78989
Grok 23.03.08787
Claude 3.5 Sonnet4.14.18484
Gemini 1.5 Pro4.64.68888
Gemini 2.0 Flash Thinking6.66.68282
o18.08.08383
Table 1: Accuracy and RMS calibration error of different models on HLE , demonstrating low accuracy and high calibration error across all models, indicative of hallucination. ∗ Model is not multi-modal, evaluated on text-only subset. We report text-only results on all models in Section C.2 and accuracy by category in Section C.3 .
Figure 5: Average completion token counts of reasoning models tested, including both reasoning and output tokens. We also plot average token counts for non-reasoning models in Section C.4 .
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ModelAccuracy (%) ↑\uparrowCalibration Error (%) ↓\downarrow
GPT-4o2.32.38888
Grok 23.23.28989
Claude 3.5 Sonnet4.34.38383
Gemini 1.5 Pro4.64.68787
Gemini 2.0 Flash Thinking6.66.68282
o17.87.88484
Appendix
Table 2: Accuracy and RMS calibration error of models from Table 1 on the text-only questions of HLE .
 Text-Only
ModelMathBio/MedPhysicsCS/AIHumanitiesChemistryEngineeringOther
GPT-4o2.32.35.05.01.51.50.90.92.62.62.02.01.61.62.32.3
Grok 23.23.25.45.44.54.53.63.61.01.01.01.04.84.81.11.1
Claude 3.5 Sonnet3.83.85.95.94.54.52.22.26.76.75.05.09.79.72.92.9
Gemini 1.5 Pro5.35.35.45.42.02.04.04.03.63.66.06.03.23.23.43.4
Gemini 2.0 Flash Thinking8.18.17.77.74.54.54.94.96.26.25.05.04.84.82.92.9
Appendix
Table 3: Category-wise breakdown of model performance on HLE .
Figure 6: Average output token counts of non-reasoning models.
ModelVersion
GPT-4ogpt-4o-2024-11-20
Grok 2grok-2-latest
Claude 3.5 Sonnetclaude-3-5-sonnet-20241022
Gemini 1.5 Progemini-1.5-pro-002
Gemini 2.0 Flash Thinkinggemini-2.0-flash-thinking-exp-01-21 ∗
o1o1-2024-12-17
Appendix
Table 4: Evaluated model versions. All models use temperature 0.0 when configurable and not otherwise stated. o3-mini and o1 models only support temperature 1.0. ∗ The first version of the paper along with Figure 5 used the now deprecated 12-19 model with temperature 0.0. The new model is sampled at temperature 0.7.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList
Jul 29, 2026

Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

Humanity's Last Exam (HLE) is widely used to evaluate frontier language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE (J=428J = 428 items) and apply psychometric methods to assess both the dimensionality of the benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find convergent evidence that HLE measures a single general reasoning factor: McDonald's ωh=0.998ω_h = 0.998, domain labels explain only 3.5% of item response variance, within- and between-domain residual correlations are nearly identical (Cohen's d=0.016d = 0.016), and domain-specific ability estimates are near-redundant with the total score (r≥0.81r \geq 0.81). A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above θ=0θ= 0, where frontier models sit. These findings suggest that HLE's domain subscores do not warrant distinct capability interpretations and that the benchmark's ability to discriminate among the strongest models is limited.

Mayank Sharma, Savira Nadela, Tyler Matteson

Stanford University

LLM EvaluationPsychometric Validation
AskPDF
Oct 14, 2026

OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score

We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai

Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang +3

Twinkle AI · APMIC · Open Taiwan Benchmark +4

LLM EvaluationBenchmark ValidityLLM Reliability
AskPDF
May 26, 2026

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.

Luca Giordano, Simon Razniewski

ScaDS.AI Dresden/Leipzig & TU Dresden, Germany

LLM EvaluationFactual Knowledge in Language Models
AskPDF