cs.CLMar 11, 2026

SciNav: A General Agent Framework for Scientific Coding Tasks

Authors: Tianshu Zhang, Huan Sun

Organizations: The Ohio State University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomous science agents built on large language models (LLMs) are increasingly used to generate hypotheses, design experiments, and produce reports. However, prior work mainly targets open-ended scientific problems with subjective outputs that are difficult to evaluate. Scientific coding benchmarks, by contrast, provide executable outputs for objective assessment. Existing approaches remain engineering-driven pipelines, revealing the need for structured, end-to-end science agent frameworks for scientific coding tasks. We address this gap by focusing on scientific coding tasks, where evaluation can be made rigorously, and introducing an agent framework SciNav (Scientific Navigator) that enables more effective solution exploration. Our framework is designed to operate under constrained search budgets, moving beyond reliance on pre-defined success metrics and prolonged search cycles. Inspired by findings that comparative judgments often reveal finer-grained quality differences and therefore provide greater discriminative power than absolute scoring, our framework leverages pairwise relative judgments within a tree search process to select top-K promising solution branches, prune low-potential ones, and progressively narrow down the solution candidates on the selected branches guided by relative comparisons. We demonstrate our agent's effectiveness across different types of tasks on two benchmarks. Experiments show that SciNav significantly outperforms direct prompting and prior agents like OpenHands and Self-Debug across different base models, task types, and difficulty levels, and exceeds different frontier comparators such as random selection and LLM absolute scoring. These results confirm the strength of our agent design and highlight the effectiveness of relative judgment-guided top-K search for high-quality scientific coding, marking a step toward more practical science agents.

Figures & tables

Figure 1: Illustration of Top-K Comparative Tree Search (TKCTS). Left: the search tree expands candidate solutions from an initial set, with relative comparisons (grey dashed arrows) guiding which branches and solutions to retain and explore. Right: comparison between absolute scoring, which LLM assigns pointwise scores to individual solutions, and relative judgment, which LLM evaluates solution pairs and provides sharper, more reliable distinctions. Relative judgments guide the search toward higher-quality solutions under constrained budgets.
Science AgentsScopeMulti-Plan?Retrieval?Self-Improve SelectionArtifact EvaluatedAutomatic Evaluation
ResearchAgent ( Baek et al., 2024 )Full-cycle, open-endedNYNeurIPS reviewIdea + Exp.NeurIPS review
AgentLab ( Schmidgall et al., 2025 )Full-cycle, open-endedNYLLM absolute scoreExp. + Paper qualityNeurIPS review
CodeScientist ( Jansen et al., 2025 )Full-cycle, open-endedNN-Paper + CodeAccept/Reject Hyp.
AI Scientist-V2 ( Yamada et al., 2025 )Full-cycle, open-endedYNTask-defined metricPaperNeurIPS review
AlphaEvolve ( Novikov et al., 2025 )Full-cycle, open-endedNNTask-defined metricCode Exec.Task-specific
SciMaster ( Chai et al., 2025 )QA with tool use-Y-Text answerAccuracy
Table 1: A comparison of existing science agents with SciNav , in terms of their focus, research artifacts being evaluated, and automatic evaluations. ‘Full-cycle’ refers to the stages from literature review, ideation, experimentation to report writing. ‘Self-Improve Selection’ refers to the evaluation strategy used to choose solutions from the candidate pool for further refinement. Unlike prior agents that rely on task-defined success metrics, our agent is designed for practical scenarios where such criteria are not available at run time. ‘Y’ indicates yes; ‘N’ indicates no; ‘–’ indicates not applicable.
Base ModelAgentsSRVERCost ↓\downarrow
GPT-4o (2024-05-13)Direct Prompting7.5042.20.011
OpenHands13.162.81.093
Self-Debug14.771.20.057
SciNav (Ours)16.166.00.512
GPT-4o (2024-11-20)Self-Debug15.067.00.030
SciNav (Ours)18.669.90.342
Table 2: Mean performances of each agent on ScienceAgentBench ( Chen et al., 2025 ) . Among the reported metrics, SR (Success Rate) is the most important as it directly reflects task success. VER (Valid Execution Rate) indicates whether a program can be executed without errors and is closely related to the number of debugging steps. Note that we run SciNav with 3 debug steps, while Self-Debug is run with 10 debug steps, which may occasionally allow Self-Debug to achieve higher VER than SciNav .
Figure 2: Performance on DA-Code ( Huang et al., 2024 ) : (a) by task type, (b) by task difficulty. Model: GPT-4o (2024-11-20).
ModelsFrontier ComparatorSRVER
GPT-4o (2024-11-20)Random Selection15.264.7
LLM-Absolute16.269.1
Relative Judgments18.669.9
GPT-4o (2024-11-20)Rubric-Absolute (w/ GT)21.174.5
Table 3: The effect of different frontier comparators on our agent. “w/ GT" means using ground truth program information in the frontier comparator. We include Rubric-Absolute only as a reference, since it is impractical in real-world settings.
Num of Init.Use Self-Improve?Avg # of SuccessfulAvg # ofSuccess Rate
SolutionsInit. SolutionsSuccessful Nodes
1No0.240.4040.5
2No0.400.5740.5
5No0.981.1445.2
5Yes1.172.6957.1
Table 4: Component ablation study on SciNav . Model: GPT-4o (2024-11-20). The experiment is conducted on 40 tasks of ScienceAgentBench, each of which has been successfully solved at least once by either a baseline agent or SciNav .
Figure 3: Error analysis on 20 randomly selected unsuccessful tasks of ScienceAgentBench.
Figure 4: Agent performance and cost across different values of K∈{1,2,3,4,5}K\in\{1,2,3,4,5\} .
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Base ModelAgentsSRVERCost ↓\downarrow
GPT-4o (2024-05-13)Direct Prompting7.50 (0.5)42.2 (1.6)0.011 (0.000)
OpenHands13.1 (2.6)62.8 (2.9)1.093 (0.071)
Self-Debug14.7 (3.2)71.2 (1.2)0.057 (0.001)
SciNav (Ours)16.1 (1.2)66.0 (3.5)0.512 (0.009)
GPT-4o (2024-11-20)Self-Debug15.0 (4.8)67.0 (7.4)0.030 (0.010)
SciNav (Ours)18.6 (3.3)69.9 (0.6)0.342 (0.008)
Appendix
Table 5: Mean performances of each agent and standard deviations on ScienceAgentBench ( Chen et al., 2025 ) . Among the reported metrics, SR (Success Rate) is the most important as it directly reflects task success. VER (Valid Execution Rate) indicates whether a program can be executed without errors and is closely related to the number of debugging steps. Note that we run SciNav with 3 debug steps, while Self-Debug is run with 10 debug steps, which may occasionally allow Self-Debug to achieve higher VER than SciNav .
ModelsSRVER
Solution Generation/RefinementJudge/Verify
GPT-4oGPT-4o18.669.9
mixture of GPT-4o and Claude-3.7GPT-4o19.663.7
Claude-3.7GPT-4o18.672.5
Appendix
Table 6: Model selection of solution generation/refinement and judgment/verification effect on our agent on ScienceAgentBench ( Chen et al., 2025 ) .
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
More in this topic

Scientific Code Generation

Scientific code generation develops systems that turn scientific questions, papers, and workflows into executable, scientifically valid programs.

See all 52 papers

Explore similar work

CardsList
Oct 23, 2026

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.

Tianyun Zhong, Wei Wang, Jialin Li +17

University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4

Scientific Agent BenchmarksLLM Agent EvaluationAI Agent BenchmarksAI for ScienceProcess-Based Agent Evaluation
AskPDF
Jun 23, 2026

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench

Yuru Wang, Lejun Cheng, Yuxin Zuo +14

Horizon Research, Frontis.AI Tsinghua University

AI for ScienceCoding AgentsAI Agent EvaluationAI Agent BenchmarksSynthetic Benchmark GenerationScientific Agent Benchmarks
AskPDF
Oct 4, 2026

AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities. We address this challenge by investigating whether scientific-agent benchmarks can be automatically generated and iteratively adapted as agent capabilities evolve. We introduce AutoSciBench, a framework that represents each task as a high-level concept specifying the scientific domain, data modality, and required reasoning approach, together with a low-level recipe specifying how the question, environment, and ground-truth answer are constructed and verified. Agents attempt to solve each task, producing solver trajectories and corresponding judge feedback which AutoSciBench uses to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories further guides new concept generation, allowing lessons from earlier task refinement to inform subsequent benchmark construction. Starting from existing benchmarks, we evaluate AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science, respectively, while generated tasks receive higher average quality ratings across all three domains, suggesting that scientific-agent evaluation can adapt as agent capabilities advance.

Dongki Kim, Namkyeong Lee, Surag Nair +7

Genentech · KAIST

AI for ScienceAI Agent EvaluationAI Agent BenchmarksSynthetic Benchmark GenerationScientific Agent Benchmarks
AskPDF