Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

Timothy Kassis

cs.AISep 7, 2026

Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

Authors: Timothy Kassis

Organizations: K-Dense

Abstract

Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific Agents, an open-source corpus of 503 profession-specific AGENTS.md profiles, with Gemini 3.8 Flash via OpenRouter in the Pi agent harness. We compare matched profiles with four controls: a minimal baseline ("You are a helpful assistant"), the profile's opening role sentence, a generic scientific rigor guide, and a profile from an unrelated domain. Across nine text-based science benchmarks (4,531 sampled questions, 100 matched profiles), 4,488 items completed all five conditions after API-error retries, scored with automated, rule-based grading. The average profile-baseline accuracy difference is -0.6 percentage points (95% bootstrap interval [-1.5, +0.2] across fixed tasks), and no benchmark shows a statistically clear improvement. Matched profiles produced 1.5-2.3 times as many output tokens and cost 2.2-4.5 times more per successful call. On 60 tool-using BioMysteryBench bioinformatics problems (three runs each for baseline and profile), mean solve rates were 46.7% with the profile and 56.7% at baseline, a difference of -10.0 percentage points (95% interval [-16.7, -3.3]) driven by more frequent token- and time-limit stops under the profile. Longer prompts had one unexpected operational advantage: on SuperGPQA, frequent provider API drops left the short baseline with a correct first-pass answer on only 54.0% of items, against 71.6% with the profile. Generic and mismatched prompts were about as reliable, so this gain comes from prompt length or formatting rather than domain expertise. For the tested model and tasks, loading full profession profiles by default does not improve accuracy and costs considerably more; whether selective retrieval of profile sections or open-ended scientific tasks would change this remains to be tested.

Figures & tables

Benchmark	Selection	$n$	Grading
SuperGPQA ( M-A-P Team et al., 2025 )	Hard MCQ; 40 subfields	1,200	Letter
HLE-MC ( Phan et al., 2025 )	Text-only science MCQ	390	Letter
MedXpertQA ( Zuo et al., 2025 )	Text; body-system strata	400	Letter
ChemBench ( Mirza et al., 2024 )	MCQ and numeric; 8 topics	489	Letter / number
SciKnowEval v2 ( Feng et al., 2024 )	L2–L4; 4 domains	600	Letter / true–false
ClimaQA Gold ( Manivannan et al., 2024 )	MCQ and cloze	452	Letter / term

Table 1: Fixed text samples and deterministic grading. These are selected subsets, not necessarily the full benchmark or its official evaluation protocol. Counts precede provider-failure exclusions.

Figure 1: Primary text comparison on all-five complete items. Columns give the retained item count and baseline (Base) and matched-profile accuracies; points show profile minus baseline, with unadjusted 95% bootstrap intervals and numerical values at right. Positive differences favor the profile. Intervals resample subfield, subject, body-system, topic, or task clusters for the first five rows and items for ClimaQA and BioProBench (Table 26 ). Secondary contrasts appear in Figure 10 .

Figure 2: BioMysteryBench. (a) Gray points: run-specific profile–baseline differences, without intervals. Blue diamond: the planned three-run mean and its 95% interval, resampling 60 paired problem-level means (10,000 draws). The mean reuses the three runs; it is not a fourth independent estimate. (b) Mutually exclusive scoring/stopping categories, grouped by run, with 60 problems per row; generic has run 1 only. Numbers label solved cases (blue), time-limit stops (hatched tan), and token-limit stops (orange). Hatched pale blue denotes only correct answers whose credit was removed by the lookup rule; other flagged runs may already be unsolved. Right: mean cumulative input, cache-read, and output tokens over turns, rounded to thousands. Categories are not independent failure causes or a mediation analysis.

Figure 3: Descriptive resource ratios, calculated from unrounded means over the retained successful response per all-five complete text item. These are ratios of means, not means of per-item ratios or inferential intervals. (a) Output tokens include reasoning; (b) estimated list-price cost prices input, cached input, and output. The line at 1 marks baseline use. Condition-specific vertical offsets separate nearby points without changing their ratios. Agentic resources use a different metric and appear in Figure 2 .

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Corpus coverage and evaluated exposure. (a) Blue bars show unique matched text profiles evaluated; full bar lengths show all profiles in each catalog domain. Labels give evaluated/total counts, not item-weighted exposure. (b) One dot per profile, with vertical jitter for visibility and black ticks for domain medians; 1 KiB = 1,024 bytes. The tail above 30 KiB contains 7 of 503 profiles.

Section	Intended content
Mindset and first principles	Laws, models, and assumptions used to reason about the field
How you frame a problem	Problem taxonomy, initial questions, and misleading explanations
How you work	Typical workflow and order of operations
Tools, instruments, and software	Tools, selection criteria, and version dependencies
Data, resources, and literature	Databases, protocols, literature, and other resources
Rigor and critical thinking	Controls, statistics, confounders, and error models

Appendix

Table 2: The canonical profile template. A heading-based scan identifies all ten sections in 485 of 503 files. Median counts are 11 second-level headings and 115 bullet points.

Domain	Profiles	Median size (KiB)	Median sources
Biology & Life Sciences	108	19.7	55
Engineering	90	19.4	52
Earth, Environmental & Atmospheric Science	58	17.6	52
Medicine & Clinical Science	58	17.8	52
Chemistry	37	16.7	52
Physics	30	19.3	52

Appendix

Table 3: Profiles per domain, using the repository’s index. Source counts are catalog metadata, not independently verified citations.

Condition	System prompt
baseline	You are a helpful assistant.
persona (bioinformatician)	You are an experienced bioinformatician.
persona (organic-chemist)	You are an experienced organic chemist specializing in synthetic and methods chemistry.
persona (neurologist)	You are an experienced neurologist spanning vascular neurology, epilepsy, movement disorders, neuroimmunology, neuromuscular disease, headache, dementia, and neurocritical care.
persona (climate-scientist)	You are an experienced climate scientist spanning physical climate, paleoclimate, detection and attribution, and Earth system model evaluation.

Appendix

Table 4: The baseline system prompt and four persona examples. Persona uses only the profile’s first sentence.

Task	Items	Profiles	Spec.	Gen.	Most-used profile (items; fallback)
SuperGPQA	1193	40	1193	0	acoustics-physicist (30)
HLE-MC	385	49	195	190	pure-mathematician (75)
MedXpertQA	400	7	165	235	physician-scientist (180)
ChemBench	485	7	420	65	physical-chemist (130)
SciKnowEval	588	15	177	411	experimental-physicist (112)
ClimaQA	448	1	448	0	climate-scientist (448)

Appendix

Table 5: Matched-profile exposure on all-five complete text items. Profiles counts distinct slugs per task; Spec., Gen., and Keyw. count item assignments. The final column gives the most-used profile and its item count; BioProBench also reports assignments through the default keyword fallback. There are 100 distinct matched profiles across tasks, not the sum of the per-task counts. BioMysteryBench uses one matched profile.

Task	Sampled rows	Distinct rendered prompts
SuperGPQA	1200	1200
HLE-MC	390	390
MedXpertQA	400	400
ChemBench	489	489
SciKnowEval	600	600
ClimaQA	452	452

Appendix

Table 6: Post-plan exact rendered-prompt duplication audit. Identical strings are hashed before any response analysis; all sampled rows are retained. Distinct IDs do not establish independent questions. This audit is not a shared-source or semantic-duplication check.

Subfield	Matched profile	Mismatch partner	Match
Aeronautical and Astronautical Science and Technology	aeronautical-engineer	hydrologist	specialist
Mass Transport and Separation Process in Chemical Engineering	separation-processes-engineer	mathematical-analyst	specialist
Elements of Chemical Reaction Engineering	reaction-engineering-specialist	low-temperature-physicist	specialist
Geotechnical Engineering	geotechnical-engineer	poultry-scientist	specialist
Data Structures	algorithms-researcher	industrial-engineer	specialist
Formal Languages	theoretical-computer-scientist	embedded-systems-engineer	specialist

Appendix

Table 7: SuperGPQA mappings (30 items sampled per subfield).

Subject	Matched profile	Mismatch partner	Match
Medicine	physician-scientist	condensed-matter-physicist	general
Genetics	geneticist	welding-joining-engineer	specialist
Biology	molecular-biologist	radiologist	general
Ecology	ecologist	pavement-engineer	specialist
Neuroscience	neuroscientist	physical-chemist	specialist
Biochemistry	biochemist	medical-geneticist	specialist

Appendix

Table 8: HLE multiple-choice subject mappings.

Body system	Matched profile	Mismatch partner	Match
Cardiovascular	cardiologist	earthquake-engineer	specialist
Nervous	neurologist	quantum-computing-scientist	specialist
Endocrine	endocrinologist	surface-engineering-specialist	specialist
Integumentary	dermatologist	brain-computer-interface-engineer	specialist
Lymphatic	hematologist	regenerative-medicine-scientist	specialist
Digestive	physician-scientist	condensed-matter-physicist	general

Appendix

Table 9: MedXpertQA body-system mappings.

Topic	Matched profile	Mismatch partner	Match
analytical_chemistry	analytical-chemist	agricultural-engineer	specialist
general_chemistry	physical-chemist	neuroscientist	general
inorganic_chemistry	inorganic-chemist	transportation-engineer	specialist
materials_science	materials-chemist	entomologist	specialist
organic_chemistry	organic-chemist	surface-physicist	specialist
physical_chemistry	physical-chemist	neuroscientist	specialist

Appendix

Table 10: ChemBench topic mappings.

Domain / task	Matched profile	Mismatch partner	Match
Biology / L2_General	molecular-biologist	radiologist	general
Biology / L2_Biology	molecular-biologist	radiologist	general
Biology / protein_function_prediction	protein-engineer	low-temperature-physicist	specialist
Biology / proteotoxicity_prediction	protein-engineer	low-temperature-physicist	general
Biology / laboratory_safety_test	clinical-laboratory-scientist	primatologist	general
Chemistry / L2_General	physical-chemist	neuroscientist	general

Appendix

Table 11: SciKnowEval domain and task mappings.

Regular expression	Profile
(?i)\b(mice\|mouse\|rat\|rats\|pig\|pigs\|rabbit\|zebrafish\|animal\|anesthe\|euthan\|i.p.\|i.m.\|intraperitoneal\|subcutaneous\|gavage)\b	comparative-medicine-researcher
(?i)\b(arabidopsis\|leaf\|leaves\|seedling\|seeds?\|plant\|root tissue\|agrobacterium)\b	plant-physiologist
(?i)\b(antibod(y\|ies)\|elisa\|flow cytometry\|facs\|immunostain\|immunofluorescence\|western blot\|cytokine\|t cells?\|b cells?\|splenocyte)\b	immunologist
(?i)\b(e. ?coli\|bacteria\|bacterial\|lb (broth\|agar)\|agar plate\|colon(y\|ies)\|yeast\|fung(us\|al)\|inoculat\|od600)\b	microbiologist
(?i)\b(protein purification\|purif(y\|ied) protein\|sds-page\|chromatograph\|his-tag\|dialys\|ni-nta\|gel filtration\|crystalliz\|enzyme assay\|kinetic)\b	biochemist
(?i)\b(hesc\|ipsc\|stem cell\|organoid\|embryoid)\b	stem-cell-biologist

Appendix

Table 12: BioProBench keyword rules. The first matching rule is used.

Rule family	Inventory	Evaluated	Positives kept	Adversarial	Default rejected	Strict rejected
alias	33	22	2/2	6	1	6
any of	6	4	1/1	2	0	2
all of	8	5	1/1	2	0	2
regular expression	8	5	1/1	4	2	4
identifier set	16	13	2/2	4	3	4
labeled sets	2	2	2/2	2	2	2

Appendix

Table 13: BioMysteryBench rule families and deterministic self-tests. Positive cases are correct answers phrased in several ways; adversarial cases should be rejected. Inventory and Evaluated columns sum to 87 and 60, respectively. The test counts describe synthetic family fixtures, not tested benchmark problems. “Strict” applies all three stricter variants.

	Run 1 (60 problems)			Mean of three runs
Grading variant	Base	Match	$\Delta$ [95% CI]	Base	Match	$\Delta$ [95% CI]
Corrected primary (v2)	60.0	48.3	$-$ 11.7 [ $-$ 23.3, 0.0]	56.7	46.7	$-$ 10.0 [ $-$ 16.7, $-$ 3.3]
Frozen implementation (v1)	58.3	46.7	$-$ 11.7 [ $-$ 23.3, 0.0]	55.6	46.1	$-$ 9.4 [ $-$ 16.1, $-$ 2.8]
No lookup penalty	60.0	50.0	$-$ 10.0 [ $-$ 21.7, +1.7]	57.2	47.2	$-$ 10.0 [ $-$ 17.2, $-$ 3.3]
Strict: negation	60.0	48.3	$-$ 11.7 [ $-$ 23.3, 0.0]	56.7	46.7	$-$ 10.0 [ $-$ 16.7, $-$ 3.3]
Strict: hedging or alternatives	60.0	48.3	$-$ 11.7 [ $-$ 23.3, 0.0]	56.7	46.7	$-$ 10.0 [ $-$ 16.7, $-$ 3.3]

Appendix

Table 14: Grading-rule sensitivity for the profile–baseline solve-rate difference (percentage points, problem-level bootstrap). The corrected implementation of the planned endpoint is primary; frozen v1 retains the original parser defects. “No lookup penalty” keeps truncation but ignores the heuristic lookup flag. Strict variants apply to the corrected implementation. All comparisons with frozen scoring and alternative rules are post-plan.

Condition	Run	Token stop	Time stop	Provider error	Wrong	Correct	Flagged	Solved
baseline	1	15	4	0	5	36	0	36
baseline	2	11	6	1	5	37	1	36
baseline	3	18	7	0	5	30	0	30
generic	1	20	2	0	3	35	2	33
profile	1	21	6	0	3	30	1	29
profile	2	24	9	0	2	25	0	25

Appendix

Table 15: Disjoint outcome components for every BioMysteryBench run (60 problems per row). “Provider error” is a completed run whose final message carried a provider error and no answer line. “Solved” equals correct answers minus those voided by the lookup rule.

Scenario	Base (%)	Match (%)	$\Delta$ [95% CI]
Corrected primary (three runs)	56.7	46.7	$-$ 10.0 [ $-$ 16.7, $-$ 3.3]
Identified watchdog cells scored zero	56.1	46.1	$-$ 10.0 [ $-$ 16.7, $-$ 3.9]
Runs 2 and 3 only	55.0	45.8	$-$ 9.2 [ $-$ 16.7, $-$ 2.5]

Appendix

Table 16: Post-plan restart checks with corrected scoring (60 paired problems). Intervals resample problem-level means over the included runs. These are not bounds on all missing attempts.

Condition	Items	Frozen correct	Corrected correct	Added credit
baseline	448	363	369	6
persona	448	358	364	6
generic	448	357	364	7
mismatch	448	364	370	6
profile	448	366	373	7

Appendix

Table 17: ClimaQA trailing-comma correction on the same complete items. All changes add credit for an exact term match after stripping a trailing comma, without semantic judging.

Benchmark	Sampled	Analyzed	Excluded
SuperGPQA	1,200	1,193	7
HLE-MC	390	385	5
MedXpertQA	400	400	0
ChemBench	489	485	4
SciKnowEval	600	588	12
ClimaQA	452	448	4

Appendix

Table 18: Text sample flow. Excluded items lack a successful provider response in at least one condition after retry passes; they are removed from every condition. Format failures remain in the analysis and score zero.

Benchmark	$n$	Base	Role	Generic	Other	Match	$\Delta$ [95% CI]
SuperGPQA	1,193	71.4	71.7	72.7	72.2	72.2	+0.8 [ $-$ 0.6, +2.1]
HLE-MC	385	42.1	43.9	44.2	42.9	41.3	$-$ 0.8 [ $-$ 3.1, +1.7]
MedXpertQA	400	66.8	65.2	67.5	65.8	64.5	$-$ 2.2 [ $-$ 4.5, $-$ 0.3]
ChemBench	485	93.0	92.8	92.0	93.6	92.2	$-$ 0.8 [ $-$ 2.2, +0.5]
SciKnowEval	588	86.1	85.0	84.7	85.0	85.4	$-$ 0.7 [ $-$ 2.1, +1.0]
ClimaQA	448	82.4	81.2	81.2	82.6	83.3	+0.9 [ $-$ 0.9, +2.7]

Appendix

Table 19: Accuracy (%) on complete text items and solve rate on BioMysteryBench. Base, Role, Generic, Other, and Match denote baseline , persona , generic , mismatch , and profile , respectively.

\Delta

is Match

-

Base in percentage points, with an unadjusted 95% bootstrap interval. The planned primary agentic endpoint is the mean over three runs; the run-1 row is the only one containing the generic condition.

Readout	baseline	generic	profile
Solved / attempted	36/60	33/60	29/60
Token-limit stops	15	20	21
Time-limit stops	4	2	6
Lookup-rule flags	1	3	2
Mean assistant turns	20.5	22.7	20.7
Mean cumulative tokens	258,264	348,787	346,065

Appendix

Table 20: BioMysteryBench resource use and stopping reasons in run 1 (60 runs per condition). Token- and time-limit stops are disjoint in these data; lookup flags are reported separately. The last row conditions on completion and compares different subsets of problems.

Condition	Problem-runs	Solved	Token stops	Time stops	All stops
baseline	180	102	44	17	61
profile	180	84	63	25	88

Appendix

Table 21: BioMysteryBench resources across three runs each of baseline and profile. Each row repeats the same 60 problems three times; the 180 problem-runs are not independent problems. Generic has no three-run summary.

Arm	Condition	Run	$n$	Score (%)	Token stops	Time stops	Reasoning tokens (median)
SuperGPQA subset	baseline	1	280	70.4	–	–	582
SuperGPQA subset	baseline	2	280	72.1	–	–	612
SuperGPQA subset	baseline	3	280	72.5	–	–	580
SuperGPQA subset	profile	1	280	72.1	–	–	1,340
SuperGPQA subset	profile	2	280	72.1	–	–	1,298
SuperGPQA subset	profile	3	280	72.9	–	–	1,397

Appendix

Table 22: Repeated runs of the baseline and profile conditions. SuperGPQA rows use the 280 subset items complete in all 15 cells (300 scheduled); BioMysteryBench rows use all 60 problems. Reasoning tokens are the provider’s reported usage at the low setting; the agentic harness retains only cumulative totals.

Figure 6: SuperGPQA subfield accuracy on all-five complete items, part 1 of 2. Rows are sorted by the exact observed profile–baseline difference, breaking ties alphabetically, not by a prespecified topic order. Hollow circles denote baseline, filled circles profile, and hollow diamonds ties. Right-hand columns give item counts and signed differences in percentage points. Both parts use the same accuracy scale, starting at 30%. Most rows have only 30 items; these are descriptive estimates without subfield-level intervals or multiplicity correction. Matched profiles are in Table 7 .

Figure 7: SuperGPQA subfield accuracy, part 2 of 2. Ordering, axis limits, symbols, item counts, and profile–baseline difference columns continue Figure 6 ; these are not independent subgroup tests.

Condition	Scheduled	Failed	Correct	Delivered success (%)
baseline	1,200	335	648	54.0
persona	1,200	172	764	63.7
generic	1,200	14	866	72.2
mismatch	1,200	8	863	71.9
profile	1,200	14	859	71.6

Appendix

Table 23: Post-plan SuperGPQA first-pass operational diagnostic. The first recorded outcome of each scheduled cache key defines the unretried pass. Every failure scores zero; each row includes all 1,200 scheduled items, without fill-pass successes. Correct delivered answers, rather than success-conditioned accuracy, are the numerator.

Benchmark	baseline	persona
SuperGPQA	23.8	5.8
HLE-MC	24.1	4.9
MedXpertQA	1.8	0.2
ChemBench	10.8	0.4
SciKnowEval	22.0	0.2
ClimaQA	16.2	0.2

Appendix

Table 24: Pass-level failure events per 100 jobs. Each event is a job exhausting up to five attempts in a retry pass; a job can contribute events in multiple passes. These are not percentages of distinct failed jobs. SuperGPQA’s unretried first pass is excluded from this table; its error rates in the main text use the first recorded outcome for each scheduled job, without mixing in successful fill-pass responses.

Condition	ERR F1	ORD Kendall $\tau$
baseline	0.721	0.866
persona	0.729	0.875
generic	0.726	0.866
mismatch	0.711	0.859
profile	0.706	0.854

Appendix

Table 25: Protocol secondary metrics on the same complete cases as the accuracy analysis. ERR F1 treats ERROR as positive; ORD reports mean Kendall

\tau

and assigns

-1

to an unparsable ordering.

Benchmark	Grouping	Groups	Primary unit	$\Delta$	Cluster CI	Stratified-item CI	Item CI
SuperGPQA	subfield	40	cluster	+0.8	[ $-$ 0.6, +2.1]	[ $-$ 0.7, +2.2]	[ $-$ 0.7, +2.2]
HLE-MC	subject	55	cluster	$-$ 0.8	[ $-$ 3.1, +1.7]	[ $-$ 4.2, +2.6]	[ $-$ 4.4, +2.9]
MedXpertQA	body system	12	cluster	$-$ 2.2	[ $-$ 4.5, $-$ 0.3]	[ $-$ 5.0, +0.5]	[ $-$ 5.2, +0.8]
ChemBench	topic	8	cluster	$-$ 0.8	[ $-$ 2.2, +0.5]	[ $-$ 2.3, +0.6]	[ $-$ 2.3, +0.6]
SciKnowEval	task	22	cluster	$-$ 0.7	[ $-$ 2.1, +1.0]	[ $-$ 2.2, +0.9]	[ $-$ 2.2, +0.9]
ClimaQA	split	2	item	+0.9	[+0.3, +1.9]	[ $-$ 0.9, +2.7]	[ $-$ 0.9, +2.7]

Appendix

Table 26: Resampling units and intervals for the profile–baseline contrast (percentage points). “Primary unit” is the unit used for the interval in Table 19 .

Weighting and resampling	Estimate (pp)	95% interval
equal (nine tasks); stratified-by-task item bootstrap; tasks fixed	$-$ 0.6	[ $-$ 1.5, +0.2]
equal (seven benchmarks; BioProBench tasks averaged first); stratified-by-task item bootstrap; tasks fixed	$-$ 0.6	[ $-$ 1.4, +0.3]
item-weighted (pooled over 4,488 items); stratified-by-task item bootstrap; tasks fixed	$-$ 0.4	[ $-$ 1.2, +0.4]
equal (nine tasks); task bootstrap over nine task-level estimates (nine units only)	$-$ 0.6	[ $-$ 1.2, $-$ 0.0]

Appendix

Table 27: Macro-average of the profile–baseline difference under alternative weightings (post-plan sensitivity). The stratified-by-task bootstrap resamples items within each fixed task; the task bootstrap resamples the nine task-level estimates and has only nine units.

Benchmark	Sampled	Missing Base/Match	Complete-case $\Delta$	Bounds	Failures-as-zero
SuperGPQA	1,200	7 / 0	+0.8	[+0.6, +1.2]	+1.2
HLE-MC	390	4 / 0	$-$ 0.8	[ $-$ 1.3, $-$ 0.3]	$-$ 0.3
MedXpertQA	400	0 / 0	$-$ 2.2	[ $-$ 2.2, $-$ 2.2]	$-$ 2.2
ChemBench	489	4 / 0	$-$ 0.8	[ $-$ 1.0, $-$ 0.2]	$-$ 0.2
SciKnowEval	600	12 / 0	$-$ 0.7	[ $-$ 1.5, +0.5]	+0.5
ClimaQA	452	4 / 0	+0.9	[+0.4, +1.3]	+1.3

Appendix

Table 28: Residual missingness after all retry passes. “Bounds” assign each missing binary score its most and least favourable value on the original sample; they are deterministic bounds, not confidence intervals. “Failures-as-zero” scores a missing response as incorrect under the actual retry policy.

Figure 8: Within-run contrasts on the 280 SuperGPQA items complete in all 15 cells (300 scheduled). Each marker compares the named condition with baseline in the same run. Symbols match Figure 3 ; fixed vertical offsets separate nearby points. No intervals or common noise threshold are implied.

Figure 9: Catalog-coverage strata, each on a separately labeled row. Columns give the stratum’s item count and profile–baseline difference with an unadjusted 95% bootstrap interval; positive values favor the profile. Specialist/general labels indicate profile availability, not validated scientific quality. Strata are confounded with topic and are not randomized matching strategies. Tasks with a single stratum repeat their overall result.

Figure 10: Planned secondary text contrasts on the same all-five complete items as Figure 1 (

n

in parentheses). Points and unadjusted 95% bootstrap intervals compare the matched profile with the persona sentence, generic rigor document, or different-domain profile. All three panels use identical axis limits and physical scales; positive differences favor the matched profile. Resampling units are those in Table 26 . These controls are practical alternatives, not isolated prompt ingredients.

Arm	Condition	Setting	$n$	Reasoning $>0$ (%)	Reasoning tokens	Output tokens
SuperGPQA subset	baseline	minimal	286	86.7	592	1,131
SuperGPQA subset	profile	minimal	286	96.2	1,242	1,770
SuperGPQA subset	baseline	low	286	85.0	591	1,130
SuperGPQA subset	profile	low	286	98.6	1,294	1,786
ChemBench numeric	baseline	minimal	87	96.6	256	440
ChemBench numeric	profile	minimal	87	97.7	412	626

Appendix

Table 29: Reasoning-token usage reported by the provider under the requested minimal and low settings, on the items complete in all four cells. The minimal request did not remove reasoning tokens.

Prompt Engineering

Prompt engineering designs and evaluates instructions, context, examples, and learned prompts to steer AI model behavior.

See all 124 papers

Oct 23, 2026

PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers

Scientific agents collaborate with researchers whose expertise and preferences differ, but benchmarks usually hold the collaborator fixed. We introduce PersonaScienceBench: 200 tasks from five benchmarks across four scientific domains. Simulated researchers provide context and feedback according to persona profiles, while task objectives and grading remain fixed. Agents encounter researcher preferences only through conversation. Across eighteen modelconfigurations, task performance, fact recovery, and self-repetition reveal different strengths. Detailed GPT 6 Astra analyses show distinct agent responses across researcher profiles; interaction traces illustrate targeted clar ification and workflow revision. These behaviors can differ even when final scores match. PersonaScienceBench therefore evaluates how agents collaborate with diverse researchers alongside whether they complete the task.

Siyang Wu, Weizhi Zhang, Chen Yang +29

University of Chicago · University of Illinois Chicago · Harvard University +15

Scientific Agent Benchmarks LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks AI Agents for Scientific Discovery Human-AI Collaboration

Ask PDF

Oct 23, 2026

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.

Tianyun Zhong, Wei Wang, Jialin Li +17

University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4

Scientific Agent Benchmarks LLM Agent Evaluation AI Agent Benchmarks AI for Science Process-Based Agent Evaluation

Ask PDF

Aug 21, 2026

K-Bench: measuring model performance on real scientific agent requests

Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference solutions, or simulators with a known generative structure. Real scientific requests arrive differently. They are underspecified, they carry attachments, and they lack ground truth. We report K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs. Three blinded language-model judges scored every run against an eight-dimension rubric. On a rubric whose 8-anchor is defined as work a domain scientist would accept with minor edits, no model clears the line under all three judges. gpt-5.6-sol has the highest pooled mean, 8.04, but its 95% interval [7.80, 8.23] spans the threshold, and two of the three judges rank claude-opus-5 first instead. We therefore report the ordering of systems as the reproducible quantity, the absolute level as an attribute of the instrument, and the top of the table as unresolved. Across all 39,934 scored judgments -- the eight dimension scores plus a holistic overall for each assessment, excluding not-applicable cells -- 47.6% fall below the 8-point threshold. Difficulty is not uniform across the rubric: scientific accuracy averages 6.22 against 7.33 for communication, on identical denominators and in the same direction within every one of the nine models. The single leading failure tag is overclaiming, on 31.4% of assessments. We argue that the informative quantity for scientific agents is not a leaderboard position but the joint distribution of what was delivered, what was claimed, and what artifacts were produced.

Aubrey M. Brueckner, Darshil Patel, Yuhuan He +1

1K-Dense, Inc.

AI for Science LLM Agent Evaluation AI Agent Benchmarks LLM Agent Reliability Scientific Agent Benchmarks

Ask PDF

Abstract

Prompt Engineering

Explore similar work

PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

K-Bench: measuring model performance on real scientific agent requests