Autonomous science agents built on large language models (LLMs) are increasingly used to generate hypotheses, design experiments, and produce reports. However, prior work mainly targets open-ended scientific problems with subjective outputs that are difficult to evaluate. Scientific coding benchmarks, by contrast, provide executable outputs for objective assessment. Existing approaches remain engineering-driven pipelines, revealing the need for structured, end-to-end science agent frameworks for scientific coding tasks. We address this gap by focusing on scientific coding tasks, where evaluation can be made rigorously, and introducing an agent framework SciNav (Scientific Navigator) that enables more effective solution exploration. Our framework is designed to operate under constrained search budgets, moving beyond reliance on pre-defined success metrics and prolonged search cycles. Inspired by findings that comparative judgments often reveal finer-grained quality differences and therefore provide greater discriminative power than absolute scoring, our framework leverages pairwise relative judgments within a tree search process to select top-K promising solution branches, prune low-potential ones, and progressively narrow down the solution candidates on the selected branches guided by relative comparisons. We demonstrate our agent's effectiveness across different types of tasks on two benchmarks. Experiments show that SciNav significantly outperforms direct prompting and prior agents like OpenHands and Self-Debug across different base models, task types, and difficulty levels, and exceeds different frontier comparators such as random selection and LLM absolute scoring. These results confirm the strength of our agent design and highlight the effectiveness of relative judgment-guided top-K search for high-quality scientific coding, marking a step toward more practical science agents.
Figures & tables
Figure 1: Illustration of Top-K Comparative Tree Search (TKCTS). Left: the search tree expands candidate solutions from an initial set, with relative comparisons (grey dashed arrows) guiding which branches and solutions to retain and explore. Right: comparison between absolute scoring, which LLM assigns pointwise scores to individual solutions, and relative judgment, which LLM evaluates solution pairs and provides sharper, more reliable distinctions. Relative judgments guide the search toward higher-quality solutions under constrained budgets.
Science Agents
Scope
Multi-Plan?
Retrieval?
Self-Improve Selection
Artifact Evaluated
Automatic Evaluation
ResearchAgent ( Baek et al., 2024 )
Full-cycle, open-ended
N
Y
NeurIPS review
Idea + Exp.
NeurIPS review
AgentLab ( Schmidgall et al., 2025 )
Full-cycle, open-ended
N
Y
LLM absolute score
Exp. + Paper quality
NeurIPS review
CodeScientist ( Jansen et al., 2025 )
Full-cycle, open-ended
N
N
-
Paper + Code
Accept/Reject Hyp.
AI Scientist-V2 ( Yamada et al., 2025 )
Full-cycle, open-ended
Y
N
Task-defined metric
Paper
NeurIPS review
AlphaEvolve ( Novikov et al., 2025 )
Full-cycle, open-ended
N
N
Task-defined metric
Code Exec.
Task-specific
SciMaster ( Chai et al., 2025 )
QA with tool use
-
Y
-
Text answer
Accuracy
Table 1: A comparison of existing science agents with SciNav , in terms of their focus, research artifacts being evaluated, and automatic evaluations. ‘Full-cycle’ refers to the stages from literature review, ideation, experimentation to report writing. ‘Self-Improve Selection’ refers to the evaluation strategy used to choose solutions from the candidate pool for further refinement. Unlike prior agents that rely on task-defined success metrics, our agent is designed for practical scenarios where such criteria are not available at run time. ‘Y’ indicates yes; ‘N’ indicates no; ‘–’ indicates not applicable.
Base Model
Agents
SR
VER
Cost ↓
GPT-4o (2024-05-13)
Direct Prompting
7.50
42.2
0.011
OpenHands
13.1
62.8
1.093
Self-Debug
14.7
71.2
0.057
SciNav (Ours)
16.1
66.0
0.512
GPT-4o (2024-11-20)
Self-Debug
15.0
67.0
0.030
SciNav (Ours)
18.6
69.9
0.342
Table 2: Mean performances of each agent on ScienceAgentBench ( Chen et al., 2025 ) . Among the reported metrics, SR (Success Rate) is the most important as it directly reflects task success. VER (Valid Execution Rate) indicates whether a program can be executed without errors and is closely related to the number of debugging steps. Note that we run SciNav with 3 debug steps, while Self-Debug is run with 10 debug steps, which may occasionally allow Self-Debug to achieve higher VER than SciNav .
Figure 2: Performance on DA-Code ( Huang et al., 2024 ) : (a) by task type, (b) by task difficulty. Model: GPT-4o (2024-11-20).
Models
Frontier Comparator
SR
VER
GPT-4o (2024-11-20)
Random Selection
15.2
64.7
LLM-Absolute
16.2
69.1
Relative Judgments
18.6
69.9
GPT-4o (2024-11-20)
Rubric-Absolute (w/ GT)
21.1
74.5
Table 3: The effect of different frontier comparators on our agent. “w/ GT" means using ground truth program information in the frontier comparator. We include Rubric-Absolute only as a reference, since it is impractical in real-world settings.
Num of Init.
Use Self-Improve?
Avg # of Successful
Avg # of
Success Rate
Solutions
Init. Solutions
Successful Nodes
1
No
0.24
0.40
40.5
2
No
0.40
0.57
40.5
5
No
0.98
1.14
45.2
5
Yes
1.17
2.69
57.1
Table 4: Component ablation study on SciNav . Model: GPT-4o (2024-11-20). The experiment is conducted on 40 tasks of ScienceAgentBench, each of which has been successfully solved at least once by either a baseline agent or SciNav .
Figure 3: Error analysis on 20 randomly selected unsuccessful tasks of ScienceAgentBench.
Figure 4: Agent performance and cost across different values of K∈{1,2,3,4,5} .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Base Model
Agents
SR
VER
Cost ↓
GPT-4o (2024-05-13)
Direct Prompting
7.50 (0.5)
42.2 (1.6)
0.011 (0.000)
OpenHands
13.1 (2.6)
62.8 (2.9)
1.093 (0.071)
Self-Debug
14.7 (3.2)
71.2 (1.2)
0.057 (0.001)
SciNav (Ours)
16.1 (1.2)
66.0 (3.5)
0.512 (0.009)
GPT-4o (2024-11-20)
Self-Debug
15.0 (4.8)
67.0 (7.4)
0.030 (0.010)
SciNav (Ours)
18.6 (3.3)
69.9 (0.6)
0.342 (0.008)
Appendix
Table 5: Mean performances of each agent and standard deviations on ScienceAgentBench ( Chen et al., 2025 ) . Among the reported metrics, SR (Success Rate) is the most important as it directly reflects task success. VER (Valid Execution Rate) indicates whether a program can be executed without errors and is closely related to the number of debugging steps. Note that we run SciNav with 3 debug steps, while Self-Debug is run with 10 debug steps, which may occasionally allow Self-Debug to achieve higher VER than SciNav .
Models
SR
VER
Solution Generation/Refinement
Judge/Verify
GPT-4o
GPT-4o
18.6
69.9
mixture of GPT-4o and Claude-3.7
GPT-4o
19.6
63.7
Claude-3.7
GPT-4o
18.6
72.5
Appendix
Table 6: Model selection of solution generation/refinement and judgment/verification effect on our agent on ScienceAgentBench ( Chen et al., 2025 ) .