Artificial superintelligence (ASI) requires AI to move beyond mastering existing knowledge toward exploring the unknown, creating new knowledge, and turning new ideas into verifiable results. However, the capabilities of today's AI systems are still largely built on learning, compressing, and applying existing human knowledge. Accordingly, existing benchmarks primarily test whether AI can produce correct answers based on learned knowledge, or whether it can complete tasks under extensive human guidance. We therefore introduce ASI-Bench, the first benchmark to jointly evaluate AI systems' capabilities of innovative exploration and autonomous scientific execution across general research domains, and the first to progressively withdraw human methodological guidance within the same research project to test how far AI can proceed on its own. Built by over 40 experts with the cost of 31,000+ human hours, ASI-Bench contains 60 project-level research tasks across 11 scientific domains and progressively reduces methodological guidance to test whether AI can independently select methods, conduct research, and produce verifiable results. All tasks undergo expert review, AI-assisted auditing, sandbox execution, and scorer validation. Across 18 state-of-the-art agent--model configurations, the average score drops from 50.91 with full methodological guidance to 29.10 with only the method specified and 26.62 when agents must determine the method themselves. This sharp decline shows that current systems remain heavily dependent on human guidance and are still far from autonomously conducting end-to-end, project-level scientific research. ASI-Bench is open to the world. We invite researchers and builders everywhere to contribute new tasks, challenge the limits of today's AI, and help accelerate humanity's collective path toward artificial superintelligence at https://asibench.apexin.ai/submit.
Figures & tables
Figure 1: Overview of ASI-Bench. Left: B3 performance across agents. Right: scores from B1 to B4, where B1 provides full methods, B2 only the method name, B3 only the research goal and data, and B4 further adds distractors.
Figure 2: Performance comparison across HLE, SWE-bench, Terminal-Bench, and ASI-Bench.
Benchmark
Task Setting
Capabilities toward Autonomous Research
Cross-domain Generality
Method Autonomy
End-to-end Research
Guidance Gradient
Humanity’s Last Exam [ 5 ]
Academic QA
✓
–
–
–
Terminal-Bench [ 7 ]
Terminal tasks
–
△
△
–
ScienceAgentBench [ 8 ]
Scientific analysis
△
△
△
–
PaperBench [ 9 ]
Research replication
–
–
✓
–
MLE-Bench [ 10 ]
ML engineering
–
△
✓
–
Table 1: Comparison of representative benchmarks for evaluating capabilities toward artificial superintelligence.
Figure 3: Representative project-level tasks in ASI-Bench across physics, astronomy, electrical engineering, and computer science.
Harness
Backbone Model
Scientific Score
Overall
Diagnostic Metrics
B1
B2
B3
B4
B2 − B1
B3 − B1
B4 − B3
GPT-5.5 (xhigh)
57.57
35.28
29.29
30.46
38.15
−22.29
−28.27
+1.16
± std.
±5.74
±2.23
±1.57
±1.55
±1.25
±7.37
±5.26
±3.10
GPT-5.6 Sol (xhigh)
62.75
42.96
40.86
40.74
46.83
−19.79
−21.89
−0.12
± std.
±3.05
±1.59
±2.35
±1.77
±1.10
±1.48
±5.27
±3.66
GPT-5.6 Sol (ultra)
71.78
49.57
51.60
50.41
55.84
−22.21
−20.18
−1.20
Table 2: Main results on 60 project-level tasks without external tool access. Scores are macro-averaged over tasks and, unless otherwise specified, over three independent runs. Gray rows report ± one sample standard deviation across independent runs. The last three columns report B2 − B1, B3 − B1, and B4 − B3, respectively, with differences computed independently for each run before aggregation. Bold values indicate the column-wise maximum result.
Figure 4: Computational cost under different levels of methodological guidance and cost–performance trade-offs across Agent × Model combinations. (a–b) Average per-task token consumption and execution time under B1–B4. (c) Relationship between per-run monetary cost and B3 scientific score across evaluated systems. Colors denote backbone models, while marker shapes denote agent harnesses.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: ASI-Bench task contribution portal. The online submission workspace provides a 15-step Guided Flow for constructing and validating new tasks. Representative interfaces show the major stages of task authoring, including scientific problem definition, B1–B4 prompt construction, evaluation design with gates and scorers, runtime configuration, task-file preparation, and local testing before review.
Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However, evaluating this ability is difficult: (1) how to verify whether a genuinely new hypothesis holds, and (2) how to determine whether a system has discovered it through exploration or merely recalled related knowledge from pre-training data. To this end, we introduce ExplorationBench, which turns the wicked problem of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds: their rules are executable, so every answer can be checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. The benchmark contains two sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks). Each sandbox provides a flawed manual, task-specific environmental feedback, and a dedicated tool-call schema. Systems use these resources to explore the sandbox, then solve held-out tasks. We evaluate 10 AI systems and find that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains. ExplorationBench represents a step towards AI systems that can acquire and apply genuinely new knowledge through exploration in unknown environments.
AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.
Autonomous AI research agents aim to speed up scientific discovery by automating the research pipeline from hypothesis generation to peer review. However, existing benchmarks rarely test a fundamental bottleneck whether LLMs can judge the methodological viability of a research idea before investing time and computational resources. Without a reliable ``first-gate'' filter, autonomous agents risk scaling flawed methodology rather than accelerating meaningful science. We introduce SoundnessBench, a curated benchmark of 1,099 real research proposals, with source verified extraction, reviewer-derived proxy labels, and proposal-only expert judgments on a subset of 327 proposals. Across 12 frontier LLMs, our evaluation reveals a pervasive optimism bias: standard prompting yields a 74.0% mean false-positive rate. Aggressive prompting reduces false approvals but substantially increases false rejections. Additional analyses show that optimism persists under controls for public corpus contamination, and surface features, as well as in literature-grounded agent evaluation. These results suggest a limitation in judging scientific soundness, indicating that current LLMs are not yet reliable as standalone gatekeepers for scientific rigor.