Praxist: From Experimental Artifacts to Solution Lineages

Jin Li; Ahmed Murtadha; Zhiyu Wang; Qiwen Chen; William Chen; Yifei Wu; Guan Wang; Cheng Gong

cs.MAAug 26, 2026

Praxist: From Experimental Artifacts to Solution Lineages

Authors: Jin Li, Ahmed Murtadha, Zhiyu Wang, Qiwen Chen, William Chen, Yifei Wu, Guan Wang, Cheng Gong,

Organizations: Sapient Intelligence · University of Pennsylvania · Tsinghua University · Nanyang Technological University ·

Abstract

Autonomous R&D agents now write, run, and improve executable artifacts under automated evaluation-but largely as laboratory instruments: shown on curated benchmarks, with gains that are hard to trace to a cause and costs well above what sustained engineering practice absorbs. The limitation is structural. Most systems treat each attempt as nearly self-contained, so logs, memories, and search trees record what happened without establishing which design element produced an improvement, whether its evidence survived validation, or how it recombines with others. Long campaigns therefore keep re-learning the same lessons. We introduce Praxist, a lineage-centered generational system that converts reproducible artifacts and evaluator outcomes into a typed evidence graph of findings, lane-structured frontiers, and agendas. Separating local artifact construction from cohort-level evidence synthesis lets later attempts inherit validated mechanisms, unresolved claims, and useful constraints, and leaves results attached to an inspectable lineage. On the standardized 75-task MLE-bench suite, the finalized official-grader results give Praxist 60 medals (80.0%), 49 of them gold, against 55 medals (73.3%) and 34 gold for a Claude Code baseline on Claude Opus 4.8-at a recorded model spend of $3054 versus$ 38370, roughly a twelfth of the cost. Five case studies-quantitative trading, LiDAR-inertial-visual SLAM, tokamak magnetic control, rocket landing, and job-shop rescheduling-carry the same process into open-ended engineering problems, improving on each task-native baseline in headline accuracy, survival, or resource cost, with the discovery path on record. Stronger artifacts at an order of magnitude less spend, each backed by an auditable lineage, are, to our knowledge, first brought together here: the operating profile production research requires, not the one a benchmark demonstration establishes.

Figures & tables

Figure 1: One generation of the Praxist loop. Agents inherit the accumulated evidence and run deliberately different experiments, each producing a reproducible artifact scored by an external evaluator (top). Outcomes become typed claims, pooled into updated evidence and the next generation’s plan: continue, stop, validate, or explore (middle). A lineage records every step (bottom).

Symbol	Meaning
$g,\;G$	generation index and number of generations
$\mathcal{S}_{g}=(\mathcal{F}_{g},\mathcal{A}_{g},\mathcal{G}_{g},\mathcal{L}_{g})$	research state inherited into generation $g$
$C$	cohort size: number of parallel peers per generation
$p_{i},\;i\in\{1,\dots,C\}$	the $i$ -th peer of a generation
$d_{i}$	design contract assigned to peer $p_{i}$
$c_{i}\in\mathcal{C}$	design cell occupied by $d_{i}$ in the behavior space $\mathcal{C}$

Table 1: Notation used in the method; lane names and vocabularies are those of the configuration used in this work. Detailed schemas are given in Appendix A .

Agent	Base LLM	Low (22)	Medium (38)	High (15)	All (75)	G/S/B
		Any Medal (%)				All-task
Claude Code	Claude Opus 4.8	81.8	76.3	53.3	73.3	34/16/5
Praxist (ours)	deepseek-v4-pro	90.9	81.6	60.0	80.0	49 /10/1

Table 2: MLE-bench Any Medal rate (%) by complexity tier and base LLM, with all-task medal composition. Bold marks the best value in each column.

Category	#	Task	Metric	Claude Code	Praxist (ours)
Image Classification	2	alaska2-image-steganalysis	weighted AUROC $\uparrow$	0.9260	0.9140
	4	cassava-leaf-disease-classification	accuracy $\uparrow$	0.9010	0.9021
	17	kuzushiji-recognition	F1 $\uparrow$	0.9714	0.9741
	23	rsna-breast-cancer-detection	probabilistic F1 $\uparrow$	0.4719	0.2752
	25	statoil-iceberg-classifier-challenge	log loss $\downarrow$	0.1451	0.1271
Image (Other)	40	petfinder-pawpularity-score	RMSE $\downarrow$	16.77	16.86

Table 3: Per-task comparison of the Claude Code + Opus 4.8 baseline and Praxist on 19 representative Medium- and High-tier MLE-bench competitions. Shading encodes the medal ( gold , silver , bronze ; unshaded = none); bold marks the better score in the metric’s direction. The # column indexes the full 75-task listing of Appendix C .

Figure 2: Rocket task overview. Initial states are drawn from three fixed banks—a nominal area-uniform disk, a row-paired slower near-OOD bank, and a faster hard-OOD outer annulus—and are flown by a deterministic hybrid controller: rolling ZEM/ZEV guidance in phase P0, fuel-commit and phase guards, and a P1/P2 terminal-descent corridor. Geometric attitude control feeds a closed-form pitch/yaw box QP that splits each torque demand between gimbal and grid fins, with the RCS restricted to roll. The frozen C05 6DoF plant advances under RK4 at $0.1$ -s steps, and every trajectory is scored once, at the interpolated first landing-leg contact, by a single joint success predicate. The evidence strip records results on the shared task and objective—the $4.03\%$ starting artifact, Weco’s reported $17.12\%$ , and the selected controller at $12{,}288/12{,}288$ —and, separately, its full-bank audit ( $40{,}959$ , $40{,}959$ , and $40{,}960$ successes per $40{,}960$ -row bank).

System	Landing success	Relative gain	Evaluations	Recorded spend
Starting artifact	$4.03\%$	—	—	—
Weco	$17.12\%$	$4.25\times$	$793$	US$1,009.66
Praxist	$\mathbf{100\%}$	$24.8\times$	$697$	US$196.05

Table 4: Rocket case study: landing success of the shared starting artifact, the Weco autonomous-optimizer baseline 20 , 57 , and Praxist on the same task package and landing_success_rate objective. Relative gains are taken against the starting artifact, which is the given controller and so has no search budget of its own. Evaluation counts are each system’s own unit of work: candidate controllers scored for Weco, completed evaluation jobs for Praxist .

Figure 3: Quant task overview. Daily market, liquidity, and account state are mapped to 100 allocation scores; a deterministic execution adapter converts those scores into constrained long/cash weights, and orders execute at the next open with per-side costs. The evaluator retrains the policy on each trailing 36-month window, leaves a five-trading-day gap, tests the next quarter, and chains all 28 quarterly returns into calendar-time CAGR.

Year	Praxist policy	Baseline	Worst quarter	Positive quarters
2019	64.17%	31.20%	5.59%	4/4
2020	194.51%	53.34%	7.34%	4/4
2021	47.59%	31.76%	0.81%	4/4
2022	$-$ 19.66%	$-$ 22.88%	$-$ 29.14%	3/4
2023	62.24%	34.41%	5.18%	4/4
2024	39.96%	19.98%	3.58%	4/4

Table 5: Quant case study: rolling walk-forward results of the discovered policy versus the paired all-eligible equal-weight baseline (2019Q1–2025Q4, 28 quarterly windows). Calendar-year rows compound that year’s four independently cash-reset quarterly windows; worst quarter and positive quarters refer to the Praxist policy.

Figure 4: SLAM task overview. Synchronized IMU, LiDAR, and camera measurements update one shared ESIKF state in sequence, producing an online 6-DoF trajectory and map. CovSched , the blue gate, intervenes only at the visual pathway: it schedules sparse-direct visual updates by marginal value and admits map points only when they add new geometry, leaving the update order and the sensor residuals unchanged. Evaluation aligns the estimated and reference trajectories and reports full-sequence APE RMSE.

Sequence	FAST-LIVO2 RMSE	CovSched RMSE	Coverage	Evaluator-captured VIO time
eee_01	0.0582	0.0279	1.03	77.8%
eee_02	0.0602	0.0408	0.90	75.6%
eee_03	0.0725	0.0322	0.79	70.2%
nya_01	0.0537	0.0308	0.99	73.6%
nya_02	0.0856	0.0353	1.76	75.6%
nya_03	0.0848	0.0339	0.99	75.9%

Table 6: SLAM case study: full-sequence APE RMSE (m) of the FAST-LIVO2 baseline versus the discovered CovSched module on all fourteen NTU-VIRAL sequences. Coverage is the ratio of CovSched to baseline ground-truth-associated samples, and the last column the reduction in the evaluator-captured amortized visual-path (VIO) processing time; both mean-row entries there are macro averages of per-sequence values, not ratios of means. Because the two arms stamp their poses differently, the APE columns are reported side by side with no relative-improvement column; Section 3.5 bounds the difference between them.

Figure 5: Fusion task overview. At every 0.5-ms step the controller compares plasma targets with the scenario’s observation and adds 12 bounded voltage residuals to nominal coil feedforward; the two closed-loop controllers evaluated here read the target values from the harness rather than from the probe observation, as noted in the text. FreeGSNKE advances the nonlinear tokamak state and terminates on hard physics violations; the benchmark scores 15 episodes by aggregate survival and tracking precision, which we report on both the benchmark’s full horizon and the common horizon defined in the text.

			Common-horizon WNRMSE $p_{95}$ ( $\downarrow$ )						Full-hor.
Controller	Survival (/1500)	Compl.	main	axis	ip	xpt	cert	all	all
Zero-feedback baseline	1039	0.267	4.77	6.10	5.12	3.85	4.74	4.89	4.89
MAST-U PCS-style	1222	0.733	2.63	3.54	2.63	1.38	2.76	2.99	4.42
Praxist (ours)	1264	0.667	2.55	3.48	2.43	1.21	2.81	2.86	4.65

Table 7: Fusion case study: MAST-U magnetic-control benchmark (5 scenarios

\times

3 seeds

\times

100-step horizon; 1,500 aggregate steps maximum). Survival counts steps before a hard physics termination; Compl. is the fraction of the 15 episodes completing the full horizon; WNRMSE

p_{95}

is the 95th percentile per-step weighted normalized tracking error (lower is better). The six common-horizon columns cap each episode at the zero-feedback baseline’s survival length for that scenario and seed; the last column is the benchmark’s original full-horizon aggregate, and Table 36 gives that metric per scenario. Each all entry is pooled over every scored step rather than averaged across the scenario columns. Scenario columns abbreviate main_shape , axis_ip , ip_shape , xpoint , and cert_shape . Best value per column in bold. Both closed-loop controllers read their target errors from the harness rather than from the probe observation, so the table is an internally matched comparison inside a privileged-state harness, as stated in the text.

Figure 6: Job-Shop task overview. Each external event—a batch of new orders, a machine fault, or a repair—triggers a replan from the visible residual problem: released, unstarted operations with duration estimates built from nominal times and completed work only. A frozen Transformer actor–critic proposes a complete operation–machine plan, visible LNS refines it, and an admission gate decides whether to deploy it or keep rule 3; the shop simulator then plays the plan forward to the next event, and an independent validator checks the schedule. Blue marks the three components Praxist modified: the duration estimate, the admission gate, and a restricted repair after faults end or orders arrive. The evaluation strip reports mean makespan improvement over rule 3 on the fixed 50-episode manifest.

Method	Mean	95% CI	P10	CVaR 10	Worst	W/T/L
Earlier PPO checkpoint	0.547	$[-0.127,1.179]$	$-2.680$	$-4.190$	$-8.429$	30/7/13
Frozen incumbent	1.381	$[0.632,2.162]$	$-1.125$	$-2.973$	$-5.775$	26/9/15
Praxist (ours)	3.375	$[2.415,4.364]$	0.000	$-$ 1.889	$-$ 4.575	43 /4/3
Oracle-event-rule reference	3.502	$[2.876,4.156]$	0.485	0.134	0.000	47/3/0

Table 8: Job-Shop case study: makespan improvement over rule 3 (%) on the fixed 50-episode MK07–MK15 manifest; positive values are shorter makespans. W/T/L counts episodes won, tied, and lost against rule 3. The oracle-event-rule row is a future-aware, non-deployable hindsight reference that picks the best of 48 dispatching rules at every event using the true future tape; it is neither ranked nor an upper bound on direct plans.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Method-level role
Allowed task context	Defines the public information, constraints, resources, and task description available to the run.
Artifact form	Defines the reproducible experimental unit $a$ that grounds evaluator outcomes and later research claims.
Evaluator semantics	Defines how artifacts produce task-grounded outcomes $e=\textsc{Eval}(a)$ and how validity, metric direction, failure states, or structured reports are interpreted.
Role-local instructions	Specialize peers, PI roles, and Chair synthesis to the task family while preserving the common Praxist cycle.
Evidence conventions	Normalize outcomes into findings $\phi$ , frontier status, agenda items, memory updates, and lineage records.

Appendix

Table 9: Task-family realization components. MLE-style tasks instantiate these components with competition context, submission artifacts, metric semantics, and grader outcomes; other task families instantiate them with their own artifact forms and evaluators.

Record	Contents and role
Artifact $a$	Minimal reproducible file set or object required to reproduce, inspect, or evaluate one attempt. In MLE-style tasks, this is typically a submission-centered bundle containing the submission, code, and supporting files needed for reproduction.
Evaluation record $e$	Evaluator outcome attached to an artifact, including validity state, score or report, metric direction, evidence stage, outcome interpretation, parent provenance, and generation/peer provenance.
Finding $\phi$	Artifact-grounded claim containing the attempted intervention, observed outcome, supporting evidence, evidence maturity $m$ , limitations, and recommended inheritance action $\alpha$ .

Appendix

Table 10: Evidence records used by Praxist . These are conceptual records; task families may instantiate them with different concrete fields.

Stage	Rank	Meaning	Mature?
smoke	0	Cheap sanity check, parser check, tiny run, or other minimal validity probe.	No
scout	1	Partial probe or early score used to decide whether a direction deserves more compute.	No
scored_complete	2	Complete task-grounded score or report for the artifact under the standard evaluator path.	Yes
full_T1	3	Full Tier-1 evaluation with stronger completion or reproducibility evidence than a single scored run.	Yes
T2	4	Replicated, ablated, or otherwise strengthened evidence beyond Tier 1.	Yes
T3 / forced_T3	5	Highest-maturity evidence, including replication or forced validation used for final claims.	Yes

Appendix

Table 11: Evidence-stage ladder. Maturity

m

is inherited from the artifact’s evaluation stage. Preliminary stages can inform memory and diagnostics but are not durable frontier evidence. Ranks 0–2 are the generic ordering the core enforces for every task; the replication rungs above them are task-owned labels, shown here with the names used by the trading campaign, and other campaigns supply their own (the rocket evaluator, for instance, reports canary , development , and complete ). The core treats any task-declared replication stage as mature evidence above scored_complete .

Field	Meaning
Intervention	What was changed or tested, expressed in the vocabulary of the design contract $d_{i}$ .
Outcome	What happened under the evaluator or diagnostic protocol, including raw task-grounded evidence when available.
Evidence	Artifact reference, evaluation record, limitations, validity state, and links to parent evidence.
$\tau$	Finding type: positive, negative, diagnostic, uncertain, or procedural.
$m$	Evidence maturity, inherited from the evidence-stage ladder in Table 11 .
$\alpha$	Recommended inheritance action: reuse, validate, avoid, diagnose, preserve, or archive.

Appendix

Table 12: Finding tuple fields for

\phi=(\text{intervention},\text{outcome},\text{evidence},\tau,m,\alpha)

\tau

and

\alpha

name the method-level roles that inheritance is defined over. The released implementation files a finding under a task-facing record enum ( result , hypothesis , insight , challenge , error ) and carries the inheritance recommendation in next-step-intent and parent-usage metadata rather than in a single

\alpha

column, so an implementer should read this table as the semantics to preserve, not as the literal field contract of the share_finding tool.

Action	Agenda implication
Reuse	Treat the evidence as a parent, baseline, mechanism, or constraint when mature enough.
Validate	Schedule reproduction, ablation, or stronger evaluation before confirming the claim.
Avoid	Prevent later peers from repeating an invalid assumption, intervention, or procedure.
Diagnose	Allocate a focused diagnostic contract to explain a failure mode or uncertainty.
Preserve	Keep the lesson as durable memory even if it is not a current frontier parent.
Archive	Retain provenance while removing the item from active inheritance.

Appendix

Table 13: Inheritance actions used by finding extraction and agenda formation. As with

\tau

, these are the method-level roles; the emitted agenda expresses them through a finer next-step-intent vocabulary that distinguishes, for example, repairing a failure mode from ablating or falsifying a mechanism.

Field	Function
Selected candidate and variant	Identifies the candidate design and names the artifact variant that peer $p_{i}$ should construct.
Design cell $c_{i}$	Assigns the contract to a cell $c_{i}=(\text{mechanism family},\text{intervention surface},\text{intent})$ in the behavior space $\mathcal{C}$ .
Intent	Purpose of the attempt within its design cell: exploit, repair, bridge, ablate, falsify, explore, diagnose, control, or audit—the same vocabulary balanced by the intent cap in Table 15 .
Mechanism hypothesis	States the causal or functional mechanism expected to produce the outcome.
Why selected	Explains why the direction is worth one peer of the cohort budget under the current frontier, agenda, and Gems.
Rejected alternatives	Records at least several nearby alternatives so the lineage can distinguish intentional coverage from accidental omission.

Appendix

Table 14: Deep Innovation Gate (DIG) design-contract fields for structured exploration.

Constraint	Default cap	Purpose
Same formal design cell	1 peer	Prevent exact duplicate contracts unless a reproduction is explicitly requested.
Same mechanism family	0.34 of cohort	Avoid spending the generation on one approach family.
Same intervention surface	0.50 of cohort	Preserve coverage across where artifacts are changed.
Same intent	0.60 of cohort	Balance exploit, repair, bridge, ablate, falsify, explore, diagnose, control, and audit intents.
Same semantic family	0.34 of cohort	Reduce near-duplicate concepts even when their formal cells differ.
Same parent lineage	0.50 of cohort	Prevent one parent from monopolizing the generation.

Appendix

Table 15: Quantified Diversity (QD) cohort allocation constraints used to keep the

C

peers diverse. Fractions are applied to the cohort size and then interpreted as integer peer caps.

Role	Synthesis function
Builder PI	Constructs the strongest evidence-backed mainline and proposes concrete next-generation tests.
Skeptic PI	Audits unsupported claims, fragile interpretations, invalid evidence, and risks that require checks.
Portfolio PI	Allocates effort across approach families, exploitation, exploration, validation, and diagnosis.
External-validity PI	Checks evidence boundaries, reproducibility, task constraints, and claims that may overfit to available feedback.
Chair	Merges PI memos into an agenda, resolves conflicts, narrows claims, and assigns peer contracts within the generation budget.

Appendix

Table 16: PI/Chair synthesis roles. Role names denote review functions; task families specialize the domain vocabulary used by each role.

Item	Contract
Mini panel	Builder PI and Skeptic PI. Used when the run needs lightweight mainline construction plus fragility audit.
Full panel	Builder PI, Skeptic PI, and Portfolio PI. Used as the default mode for balancing score, reliability, and coverage.
High-stakes panel	Full panel plus External-validity PI. Used when final claims, evidence boundaries, or reproducibility risks require extra scrutiny.
Round 0: evidence freeze	Builds the shared evidence core and role-private packs from $\Phi_{g}$ , $\mathcal{F}_{g+1}$ , recent agendas, Gems, and lineage.
Round 1: independent memos	Each PI writes an independent interpretation before seeing other PI conclusions.
Round 2: anonymized cross-review	PI memos are cross-reviewed against fixed questions about support, fragility, validation, missing controls, and portfolio coverage.

Appendix

Table 17: Panel modes and synthesis rounds. The full panel is the default; high-stakes mode adds the external-validity role and an explicit confidence round.

Status	Meaning	Typical agenda effect
Confirmed ( $\mathcal{F}^{\mathrm{cf}}$ )	Reliable inheritable evidence, mechanism, artifact, or constraint.	Continue: use as parent, baseline, comparison point, or constraint.
Candidate ( $\mathcal{F}^{\mathrm{cd}}$ )	Promising evidence with incomplete maturity or incomplete coverage.	Explore or validate: reproduce, ablate, or combine with caution.
Diagnostic ( $\mathcal{F}^{\mathrm{dg}}$ )	Failure mode, invalidated assumption, control, or process observation.	Stop, constrain, or diagnose: prevent repeated errors and schedule focused tests.
Validation ( $\mathcal{F}^{\mathrm{vl}}$ )	Evidence requiring reproduction, ablation, or checking before promotion.	Validate: assign validation peers or define a minimal test.

Appendix

Table 18: Frontier statuses and their agenda effects.

Agenda field or disposition	Meaning
Generation metadata	Identifies which generation the agenda controls and which evidence freeze produced it.
Cross-peer hypotheses	Shared claims with minimal tests, kill conditions, and promote conditions.
Peer contracts	Exactly $C$ executable contracts for the next generation, each with role, parent or independent target, success signal, and forbidden actions.
Panel summary and consensus actions	Chair-level synthesis of what the PI panel believes should continue, stop, validate, diagnose, or explore.
Claim-boundary updates	Narrowing statements that prevent later peers from overstating what the evidence supports.
Continue	Usually emitted for confirmed evidence that should be reused as a parent, baseline, or constraint.

Appendix

Table 19: Agenda schema and lane-to-disposition mapping.

Category	Role
Validated mechanism	Preserves a reusable strategy, transformation, or design pattern across generations.
Rejected assumption	Records an assumption weakened by evidence so that later peers avoid relying on it.
Recurring failure mode	Summarizes a diagnostic pattern observed across attempts or generations.
Procedural constraint	Preserves task-specific or task-family-specific constraints needed for valid artifacts.
Evidence boundary	Records the scope within which a conclusion is supported.

Appendix

Table 20: Common Gem categories. The categories are descriptive roles rather than a stored field: a Gem record carries its source finding, task lane, metric, evidence stage, and selection provenance, and its category is read from that content.

Recipient	Routed context
Peer $p_{i}$	Agenda-relevant frontier entries, relevant Gems, parent lineage, the assigned design contract $d_{i}$ , and task-family constraints.
PI role	Evidence pack for $\Phi_{g}$ , current frontier, relevant Gems, recent agenda history, and role-specific review questions.
Chair	PI memos, anonymized cross-review signals, agenda candidates, frontier state, and validation or dissent items requiring arbitration.
Future run or human reader	Final artifact $a^{\star}$ , confirmed frontier evidence, Gems, and lineage $\mathcal{L}_{G}$ explaining the solution path.

Appendix

Table 21: Selective-memory routing. Each role receives the subset of state it needs to act without treating every prior observation as equally inheritable.

Component	Recorded relation
Artifact	Reproducible experimental unit produced or modified by a peer.
Evaluation record	Task-grounded outcome attached to an artifact.
Finding	Research claim supported by an artifact and evaluation record.
Design contract	Pre-artifact intent that motivated a peer’s attempt.
PI/Chair decision	Synthesis decision derived from generation-level evidence.
Frontier update	Inheritance-status decision for a finding, artifact, or evidence item.

Appendix

Table 22: Lineage trace components.

Edge type	Meaning
derived_from	The child artifact, finding, agenda, or Gem was constructed from the parent evidence.
supports	The source provides evidence in favor of the target claim, promotion, agenda item, or Gem.
challenges	The source weakens, falsifies, or limits the target claim or direction.
updates	The source revises the target, such as a later frontier entry superseding an earlier candidate.
related_to	The source is relevant to the target but does not imply support, challenge, derivation, or update.

Appendix

Table 23: Lineage edge vocabulary. Edges are typed so that later peers and readers can distinguish inheritance, support, conflict, and update relations.

Policy	Quota success (%)	P10 (%)	Mature mean	GPU util. (%)	CPU util. (%)
Passive peers (no backfill)	27.03	0.0	1.72	10.00	11.71
Idle backfill, 25% mature follow-up	94.37	79.2	16.99	23.62	24.30
Idle backfill, 50% mature follow-up	96.54	90.0	17.53	23.55	24.26
Idle backfill, 75% mature follow-up	97.64	94.5	17.93	23.50	24.24
Boolean maturity signal	98.65	97.6	17.78	23.55	24.30
Nonredundant thin token	98.36	96.6	17.67	23.55	24.29

Appendix

Table 24: Simulated scheduling policies over 348 physically feasible scenarios. Quota success is the probability that a generation meets its mature-evidence quota

Q

before the horizon; P10 is the 10th percentile of per-scenario success.

Condition	Scenarios	Boolean (%)	Thin token (%)	Debt controller (%)
Low CPU pressure ( $h<0.35$ )	232	99.21	99.10	99.94
High CPU pressure ( $h\geq 1.20$ )	52	95.74	94.45	99.49
CPU saturated over half the generation	59	96.20	95.05	99.55
Observed GPU utilization $\geq 50\%$	64	96.87	96.42	99.76
Observed GPU-memory utilization $\geq 50\%$	67	95.33	94.63	99.76

Appendix

Table 25: Simulated quota success under operational resource pressure. Strata are defined by the observed state of the full-only reference within the same scenario.

Quantity	Default	Role
Generations $G$	8	Maximum number of generation boundaries in a run.
Cohort size $C$	5	Number of parallel peers per generation.
Promotions per generation	2 plus anchor extras	Baseline count of primary metric promotions before task-specific anchor additions.
Per-peer safety cap	5 hours	Hard upper bound for any single peer; generation closing is event driven.
Gem compression	Disabled	When enabled, resets every 6 generations, with at most 3 resets and 4 active Gems.
DIG candidate count	8	Number of candidate designs considered before cohort allocation.

Appendix

Table 26: Representative default configuration values.

Setting	Value
Research-agent model	deepseek-v4-pro[1m]
Cohort size	12 peers
Promotions per generation	Top 4
Maximum generations	20
GPU budget per experiment	6 GPU-h
Experiment scheduling	Single-GPU job, one job per GPU

Appendix

Table 27: Praxist campaign configuration behind the finalized MLE-bench ledger.

Prompt family	Primary input	Expected output
Peer research	Task context, inherited state, agenda, design contract, artifact constraints.	Reproducible artifact and artifact-grounded evidence.
Design allocation	Frontier, agenda, Gems, coverage state, prior design cells.	Peer design contracts covering diverse research directions.
Finding extraction	Artifact, evaluation record, peer notes, diagnostics.	Positive, negative, diagnostic, uncertain, or procedural findings.
Frontier update	Findings, evaluation records, prior frontier, validation needs.	Inheritance status and promotion or validation decisions.
PI memo	Evidence pack, frontier state, Gems, task constraints.	Role-specific interpretation, objections, proposed experiments, and peer contracts.
Chair agenda	PI memos, cross-reviews, evidence pack, frontier state.	Next-generation agenda with peer contracts and claim boundaries.

Appendix

Table 28: Prompt families used by Praxist .

Stage	Evaluated artifact or intervention	Score	Role in the trajectory
Generation 0	Class-balanced focal-loss BERT	0.98620	Initial hypothesis
Generation 0	Matched BCE ablation, BERT seed 42	0.98657	Confirmed parent
Generation 1, C07	Metadata-only null model	0.76106	Diagnostic finding
Generation 1	Equal average of the two BERT seeds	0.98679	Two-model finding
Generation 1	Add DistilRoBERTa to the two-seed ensemble	0.98723	Three-model parent finding
Generation 1	Add BiLSTM to the three-model ensemble	0.98711	Negative finding

Appendix

Table 29: Research trajectory for the Jigsaw Toxic Comment Classification submission selected in generation 1. Scores are recorded column-wise ROC AUC values.

Study	Repetition behind the reported result	Execution record
MLE-bench	75 tasks; one finalized result per task; one benchmark sweep, so no repeated seeds and no medal-rate variance estimate.	Shared pool of H100 80GB GPUs; one GPU per experiment, up to eight experiments in flight per task; cohort of 12 peers; 24-hour task wall cap, 36 hours on 5 of the 75 tasks.
Rocket	Complete evaluation: 4,096 trajectories from each of the nominal, near-OOD, and hard-OOD banks = 12,288 landing trajectories, plus 1,024 frozen roll-disturbance cases. Separate post-run audit: all 40,960 rows of each bank = 122,880 landing trajectories.	16 peers, promote 4, 2-hour generation window, 0.0075 configured GPU-h per experiment; up to eight one-GPU evaluations on 8 H100 80GB GPUs; 12 committed generation boundaries, terminated by SIGTERM during generation 12 after 14.35 hours.
Quant	28 quarterly walk-forward windows, each retrained from scratch on the trailing 36 months; the five-seed tier evaluates 145 window-seed cells, but the reported artifact stopped at the one-seed, 29-cell tier.	12 peers, promote 4, 18-hour generation window, 18 configured GPU-h per experiment; 37 generation boundaries over a 176-hour artifact wall span.
SLAM	Fourteen NTU-VIRAL sequences, one accepted run per method and sequence, selected from a campaign containing replays and re-runs; no training seeds and no variance estimate. Candidates compile the pipeline and replay bags.	8 peers, promote 2, 5-hour generation window; sampled pods ran the CPU FAST-LIVO2 path on a 168-logical-core host with no NVIDIA device exposed.
Fusion	5 scenarios $\times$ 3 seeds $\times$ a 100-step horizon = 15 episodes and at most 1,500 survived steps per controller; no optimizer iterations.	5 peers, promote 2, 6-hour generation window; the reported comparison is three sidecar runs of 28.05, 30.32, and 30.64 minutes.
Job-Shop	Complete evaluation: 50 fixed MK07–MK15 episodes, paired against rule 3 and re-run with the task’s comparators; deterministic, so no seed variance. 119 such evaluations over the campaign.	12 peers, promote 3, 3.5-hour generation window; one H100 80GB GPU per evaluation on an 8-GPU host; 13 generations (0–12) over a 46.8-hour artifact wall span, terminated by SIGTERM.

Appendix

Table 30: Execution summary of the six studies in Section 3 . Generation counts are committed generation boundaries, except for Fusion, whose legacy trajectory records completed generation result sets instead.

Item	Frozen value
Plant	Swordfish C05 six-degree-of-freedom rigid body, through the task’s frozen adapter
Integration	RK4, one substep, $0.1\,$ s step, at most $900$ steps ( $90\,$ s)
Dry mass	$22{,}200\,$ kg
Initial main propellant	$7{,}000\,$ kg
Initial total mass	$29{,}200\,$ kg
Initial altitude	$2{,}000\,$ m

Appendix

Table 31: The frozen Rocket protocol. Every entry is fixed for the whole campaign: a candidate may change the controller, and nothing else.

Metric	Frozen baseline	Praxist controller	Change
Landing successes	$495/12{,}288$	$12{,}288/12{,}288$	—
Landing success rate	$4.0283\%$	$100\%$	$+95.9717$ pp
Wilson $95\%$ lower bound	$3.6948\%$	$99.9687\%$	descriptive
Nominal-bank success	$5.6641\%$	$100\%$	same $4{,}096$ cases
Near-OOD-bank success	$6.4209\%$	$100\%$	same $4{,}096$ cases
Hard-OOD-bank success	$0\%$	$100\%$	same $4{,}096$ cases

Appendix

Table 32: Panel A: matched complete-protocol comparison of the frozen task baseline against the evaluated controller. All rows use the same

12{,}288

landing trajectories, except the two roll rows and the coupling row, which use the frozen

1{,}024

-case roll suite. Rows marked adverse are those on which the controller is worse than the baseline.

Bank	Successes	Success rate	Wilson $95\%$ LB	COM sink P95	Fuel P05	Min. successful fuel
Nominal	$40{,}959/40{,}960$	$99.9975586\%$	$99.9861709\%$	$0.32283$ m/s	$5.7309\%$	$4.6021\%$
Near-OOD	$40{,}959/40{,}960$	$99.9975586\%$	$99.9861709\%$	$0.32778$ m/s	$5.9162\%$	$4.8016\%$
Hard-OOD	$40{,}960/40{,}960$	$100\%$	$99.9906223\%$	$0.32927$ m/s	$3.3543\%$	$2.8908\%$
Pooled (descriptive only)	$122{,}878/122{,}880$	$99.9983724\%$	$99.9940652\%$	$0.32636$ m/s	$3.6904\%$	$2.8908\%$

Appendix

Table 33: Panel B: post-run all-row audit of the frozen selected controller over every row of each source bank. Per-bank counts are the primary claim; the pooled row is a descriptive sum.

Metric	Allocator off	Allocator on	Change
Landing / hard / worst-bin success	$100\%$	$100\%$	ceiling retained
COM sink P95	$0.337891$ m/s	$0.326373$ m/s	$3.4088\%$ lower
Lateral speed P95	$0.065620$ m/s	$0.047199$ m/s	$28.0727\%$ lower
Gimbal saturation step rate	$0.12303\%$	$0$	removed
Grid total variation	$10.906433$ rad	$0.046014$ rad	$99.5781\%$ lower
Gimbal total variation	$0.866120$ rad	$0.884548$ rad	$2.1277\%$ higher (adv.)

Appendix

Table 34: One-key allocator ablation on the frozen complete protocol. The two configurations share the same

0.97

gimbal cap, differ only in alloc_qp_enable , and are both already at

100\%

landing success.

Stage	Main accumulated change	Complete landing success
Baseline	Original SF-CAC v2 first-contact controller	$4.0283\%$
Gen 1	Fuel-commit trigger, two-band descent reference, settled-release gate, and P0 lateral-horizon floor $28\to 20\,$ s; then guidance period $0.2\to 0.1\,$ s	$47.9411\%\to 49.2920\%$
Gen 2	P0 pitch/yaw $\omega_{n}:0.55\to 1.0$ , $\zeta:0.9\to 0.7$ , and reference-rate limit $6\to 10^{\circ}\,\mathrm{s^{-1}}$	$98.3561\%$
Gen 3–4	Reference-rate limit restored $10\to 6^{\circ}\,\mathrm{s^{-1}}$ ; low-altitude P0 bandwidth schedule added; grid budget capped at $0.24\,$ rad	about $98.99\%$
Gen 6–8	Initial-radius/slew branch, $6.5^{\circ}\,\mathrm{s^{-1}}$ reference rate, and sink guard	$99.9674\%\to 99.9919\%$
Gen 9	Remove the $r_{0}<450\,$ m branch’s fuel-commit vertical-speed conjunct while retaining axial-slew decoupling and the $-15\,\mathrm{m\,s^{-1}}$ release sink guard	$100\%$

Appendix

Table 35: Condensed lineage along the actual parent chain, with the complete-protocol landing success recorded at each stage.

	Zero-feedback	MAST-U PCS-style		Praxist (ours)
Scenario	full $=$ common	full	common	full	common
main_shape	4.77	4.18	2.63	8.50	2.55
axis_ip	6.10	5.76	3.54	6.26	3.48
ip_shape	5.12	3.37	2.63	3.22	2.43
xpoint	3.85	2.87	1.38	2.98	1.21
cert_shape	4.74	13.46	2.76	3.76	2.81

Appendix

Table 36: Fusion case study: full-horizon and common-horizon WNRMSE

p_{95}

(lower is better) per benchmark scenario. The full horizon is the benchmark’s original definition, scoring every step a controller survives; the common horizon caps each episode at the zero-feedback baseline’s survival length for that scenario and seed. Aggregate rows are pooled over all scored steps rather than averaged across scenarios.

Gen.	Change	Mean [95% CI]	CVaR 10	p99 (ms)
—	Frozen incumbent	1.381 [0.632, 2.162]	$-2.973$	1653
0	Control: LNS on rule 3’s plan, no neural plan	$-1.712$ [ $-2.794$ , $-0.711$ ]	$-10.416$	1663
1	Gate threshold lowered from 4% to zero	1.746 [0.777, 2.772]	$-4.062$	1707
3	Prior-shrinkage duration estimator	2.668 [1.899, 3.486]	$-1.024$	1717
3	Ablation: pure prior (no running average)	2.524 [1.719, 3.356]	$-1.277$	1713
4	Disagreement-gated estimator bridge	2.981 [2.123, 3.883]	$-1.503$	1714

Appendix

Table 37: Job-Shop lineage: complete 50-episode evaluations on the path to the selected artifact. Mean is the makespan improvement over rule 3 (%) with its paired-bootstrap 95% interval; p99 is the 99th-percentile decision latency. The last two rows are other Pareto points of the run, off the selected path.

Category	#	Task	Metric	Claude Code	Praxist (ours)
Image Classification	1	aerial-cactus-identification	AUROC $\uparrow$	1.000	1.000
	2	alaska2-image-steganalysis	weighted AUROC $\uparrow$	0.9260	0.9140
	3	aptos2019-blindness-detection	quadratic kappa $\uparrow$	0.9354	0.9320
	4	cassava-leaf-disease-classification	accuracy $\uparrow$	0.9010	0.9021
	5	cdiscount-image-classification-challenge	accuracy $\uparrow$	0.7661	0.6138
	6	dog-breed-identification	log loss $\downarrow$	0.1886	0.1162

Appendix

Table 38: Full MLE-bench per-task comparison (Part I of II): image classification, image-to-image, and text classification. Shading encodes the medal (gold, silver, bronze; unshaded = none); bold marks the better score in the metric’s direction, assigned on full-precision values.

Category	#	Task	Metric	Claude Code	Praxist (ours)
Image (Other)	39	histopathologic-cancer-detection	AUROC $\uparrow$	0.9987	0.9979
	40	petfinder-pawpularity-score	RMSE $\downarrow$	16.77	16.86
	41	rsna-miccai-brain-tumor-radiogenomic-classification	AUROC $\uparrow$	0.5341	0.6588
Audio Classification	42	freesound-audio-tagging-2019	LRAP $\uparrow$	0.7471	0.7445
	43	mlsp-2013-birds	AUROC $\uparrow$	0.9603	0.9555
	44	tensorflow-speech-recognition-challenge	accuracy $\uparrow$	0.9796	0.9736

Appendix

Table 39: Full MLE-bench per-task comparison (Part II of II): remaining task categories. Conventions follow Table 38 .

Autonomous Experimentation

Autonomous experimentation uses AI, active learning, and robotics to design, run, and interpret experiments with less human intervention.

See all 153 papers

Apr 27, 2026

The Last Human-Written Paper: Agent-Native Research Artifacts

Scientific publication compresses a branching, iterative research process into a linear narrative, discarding the majority of what was discovered along the way. This compilation imposes two structural costs: a Storytelling Tax, where failed experiments, rejected hypotheses, and the branching exploration process are discarded to fit a linear narrative; and an Engineering Tax, where the gap between reviewer-sufficient prose and agent-sufficient specification leaves critical implementation details unwritten. Tolerable for human readers, these costs become critical when AI agents must understand, reproduce, and extend published work. We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package structured around four layers: scientific logic, executable code with full specifications, an exploration graph that preserves the failures compilation discards, and evidence grounding every claim in raw outputs. Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste. On PaperBench and RE-Bench, ARA raises question-answering accuracy from 72.4% to 93.7% and reproduction success from 57.4% to 64.4%. On RE-Bench's five open-ended extension tasks, preserved failure traces in ARA accelerate progress, but can also constrain a capable agent from stepping outside the prior-run box depending on the agent's capabilities. Our code is open-sourced at https://github.com/Orchestra-Research/Agent-Native-Research-Artifact.

Jiachen Liu, Jiaxin Pei, Jintao Huang +34

University of Michigan · Stanford University · Ohio State University +22

AI-Assisted Scientific Research Scientific Communication Scientific Reproducibility Automated Peer Review Autonomous Research Agents

Ask PDF

Jul 19, 2026

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Automated Algorithm Discovery Automated Evaluation Autonomous Research Agents

Ask PDF

Oct 1, 2026

YouRA: A Persistent-State Architecture for Evidence-Traceable Autonomous Research Agents

End-to-end research agents can now produce complete scientific papers, yet manuscript claims often diverge from executed experiments. This gap is structural: research state, failure histories, and claim-evidence alignment are not maintained as persistent, verifiable state across long-horizon pipelines. We present YouRA (Your Research Agent), an architecture for stateful, evidence-traceable autonomous research. YouRA preserves research state, execution evidence, and failure history across the research trajectory by integrating three components: a Verification State Architecture (VSA) that tracks hypotheses, gates, and evidence pointers; an Independent Controller that turns state and reflection records into lifecycle, recovery, and debate/review control while separating control from execution; and Stateful Reflection that logs failures as structured lessons and routes recovery through bounded repair, redesign, or reset. On MLR-Bench's predefined ten-task end-to-end subset, YouRA improves over both MLR-Agent and AI Scientist V2 on scalar Overall across all three matched backbones. An automated diagnostic using MLR-Bench's hallucination taxonomy reports intersection/union counts for four fact-based failure types, and data-provenance diagnostic shows more real-data-based outputs. Ablating each of the four components (the VSA, the Independent Controller, MCP tool access, and reflection-guided recovery) supports their separable contributions. Removing either core-state component drops YouRA below the full system. Code: https://github.com/PrayPrey/Your-Research-Agent.

Yoonkyu Woo, Woojin Lee, Jin-Xia Huang

Electronics and Telecommunications Research Institute, Republic of Korea · Department of Artificial Intelligence, University of Science and Technology, Republic of Korea

AI Agent Reliability AI Agents for Scientific Discovery Memory-Augmented Agents Autonomous Research Agents

Ask PDF

Abstract

Autonomous Experimentation

Explore similar work

The Last Human-Written Paper: Agent-Native Research Artifacts

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

YouRA: A Persistent-State Architecture for Evidence-Traceable Autonomous Research Agents