cs.LGJan 24, 2025

Humanity's Last Exam

Authors: Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, 

Organizations: Michael Pokorny · Quinn · Tony · Quod

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Figures & tables

Figure 1: Compared against the saturation of some existing benchmarks, Humanity’s Last Exam accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities. The sources for our evaluation metrics are detailed in Section C.6 . We further evaluate more frontier models on HLE in Table 1 .
Figure 2: Samples of the diverse and challenging questions submitted to Humanity’s Last Exam .
Figure 3: HLE consists of 2,5002{,}500 exam questions in over a hundred subjects, grouped into high level categories here. We provide a more detailed list of subjects in Section B.4 .
Figure 4: Dataset creation pipeline. We accept questions that make frontier LLMs fail, then iteratively refine them with the help of expert peer reviewers. Each question is then manually approved by organizers or expert reviewers trained by organizers. A private held-out set is kept in addition to the public set to assess model overfitting and gaming on the public benchmark.
Pre-Release ModelsAccuracy (%) ↑\uparrowCalibration Error (%) ↓\downarrow
GPT-4o2.72.78989
Grok 23.03.08787
Claude 3.5 Sonnet4.14.18484
Gemini 1.5 Pro4.64.68888
Gemini 2.0 Flash Thinking6.66.68282
o18.08.08383
Table 1: Accuracy and RMS calibration error of different models on HLE , demonstrating low accuracy and high calibration error across all models, indicative of hallucination. ∗ Model is not multi-modal, evaluated on text-only subset. We report text-only results on all models in Section C.2 and accuracy by category in Section C.3 .
Figure 5: Average completion token counts of reasoning models tested, including both reasoning and output tokens. We also plot average token counts for non-reasoning models in Section C.4 .
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ModelAccuracy (%) ↑\uparrowCalibration Error (%) ↓\downarrow
GPT-4o2.32.38888
Grok 23.23.28989
Claude 3.5 Sonnet4.34.38383
Gemini 1.5 Pro4.64.68787
Gemini 2.0 Flash Thinking6.66.68282
o17.87.88484
Appendix
Table 2: Accuracy and RMS calibration error of models from Table 1 on the text-only questions of HLE .
 Text-Only
ModelMathBio/MedPhysicsCS/AIHumanitiesChemistryEngineeringOther
GPT-4o2.32.35.05.01.51.50.90.92.62.62.02.01.61.62.32.3
Grok 23.23.25.45.44.54.53.63.61.01.01.01.04.84.81.11.1
Claude 3.5 Sonnet3.83.85.95.94.54.52.22.26.76.75.05.09.79.72.92.9
Gemini 1.5 Pro5.35.35.45.42.02.04.04.03.63.66.06.03.23.23.43.4
Gemini 2.0 Flash Thinking8.18.17.77.74.54.54.94.96.26.25.05.04.84.82.92.9
Appendix
Table 3: Category-wise breakdown of model performance on HLE .
Figure 6: Average output token counts of non-reasoning models.
ModelVersion
GPT-4ogpt-4o-2024-11-20
Grok 2grok-2-latest
Claude 3.5 Sonnetclaude-3-5-sonnet-20241022
Gemini 1.5 Progemini-1.5-pro-002
Gemini 2.0 Flash Thinkinggemini-2.0-flash-thinking-exp-01-21 ∗
o1o1-2024-12-17
Appendix
Table 4: Evaluated model versions. All models use temperature 0.0 when configurable and not otherwise stated. o3-mini and o1 models only support temperature 1.0. ∗ The first version of the paper along with Figure 5 used the now deprecated 12-19 model with temperature 0.0. The new model is sampled at temperature 0.7.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList

  1. Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

    Jul 29, 2026Mayank Sharma, Savira Nadela, Tyler MattesonLLM EvaluationPsychometric Validation

    Stanford University

  2. OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score

    Oct 14, 2026Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang +3LLM EvaluationBenchmark Validity

    Twinkle AI · APMIC · Open Taiwan Benchmark +4

  3. Beyond Questions: Evaluating the Stated Knowledge of LLMs

    May 26, 2026Luca Giordano, Simon RazniewskiLLM EvaluationFactual Knowledge in Language Models

    ScaDS.AI Dresden/Leipzig & TU Dresden, Germany