cs.CLAug 18, 2026
SaveArtifact-centered Claim-aware Observability for Autonomous Scientific Agents
Organizations: Advanced Photon Source, Argonne National Laboratory, Lemont IL, USA
Abstract
Autonomous scientific agents now increasingly propose ideas, write code, run experiments, analyze results, and even draft papers. Observe and audit those agents are necessary but logging every model call is not enough, scientists also need to inspect the artifacts and claims that the systems produced and their relations. This is driven by the fact that failures in scientific agent systems are often distributed across several objects. A manuscript claim may cite the wrong evidence, a search process may select a degenerate candidate, a laboratory novelty claim may depend on an unstated rule, or a multi-agent plan may change without a visible trigger. Existing tracing, experiment tracking, and archival provenance tools are valuable, but their native objects do not make these scientific audit relations first-class. We argue that autonomous scientific systems should emit portable, claim-aware artifact lineage as a minimum audit layer. We propose a compact observability profile organized around individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands. In this profile, scientific claims are ordinary individuals with explicit evidence bindings and verification records. The profile is intended as a semantic layer that complements current telemetry and provenance standards. Execution details can remain in OpenTelemetry. Final packages can export to PROV-O or RO-Crate standards.
Figures & tables
Figure 1: The observability shift. Runtime spans record nested calls and timing. Claim-aware artifact observability records candidate artifacts, the operators that derive them, evaluator outputs, archive decisions, and evidence links for claims.
| Audit question | Required relation | Example failure exposed |
|---|---|---|
| Where did this result come from? | Parent individuals, derivation operator, emitter stream | Final candidate descends from a failed or out-of-policy branch |
| What supports this claim? | Claim individual, evidence references, evidence bindings | Manuscript cites a table whose value contradicts the sentence |
| Who or what accepted it? | Fitness record, evaluator identity, threshold or rubric | One unstable evaluator dominates a multi-evaluator pipeline |
| How was the search steered? | Steering command, applied operator, affected archive or plan | Human correction is overwritten by a later autonomous step |
| What was discarded? | Archive-update and rejection events | Negative results or failed replications disappear from the record |
Table 1: Design requirements derived from scientific audit questions. The profile separates derivation, evidence, evaluation, and steering because each supports a different class of review.
| Step | Record | Audit-relevant fields |
|---|---|---|
| 1 | Individual: experiment log | id=i_exp , kind=experiment_log , payload hash, metric table URI. |
| 2 | Individual: draft | id=i_draft , kind=manuscript_draft , payload hash for generated text. |
| 3 | Operator: extract claim | Parent i_draft , child i_claim , stream writer , extractor version. |
| 4 | Individual: claim | kind=claim , text payload, evidence_refs=[i_exp] , binding to metric row and value. |
| 5 | Operator: verify | Parent i_claim , context i_exp , evaluator claim_numeric_checker . |
| 6 | Fitness: verification | Target i_claim , value unsupported , reason evidence_mismatch , extracted and claimed values. |
Table 2: A compact trace for an unsupported numerical claim. The payloads can live outside the trace. The trace preserves the audit relations.
Figure 2: Six trace shapes that can share the same artifact-centered vocabulary: linear refinement, multi-phase pipeline, tree search, evolutionary population, self-modifying artifact, and closed-loop discovery.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Core records and their reference structure. Payloads can remain in notebooks, databases, repositories, instrument stores, or archives. The trace records the identities and relations needed for audit.
| Record | Minimum fields | Integrity rule |
|---|---|---|
| Individual | id , run_id , kind , created_at , payload reference or hash | The identifier denotes one inspectable artifact version: draft, code, dataset, sample, measurement, plan, policy, evaluator, or claim. |
| Operator | id , op_type , parent_ids , child_ids , emitter stream, timestamp | Children are derived from earlier parents or explicitly imported external objects. Context references do not imply derivation. |
| Fitness record | evaluated individual, evaluator identity and version, value or verdict, timestamp | The evaluator version is part of the result. A later evaluator may disagree without overwriting the earlier record. |
| Evidence binding | claim identifier, evidence identifier, locator, extracted value or rule, binding role | A claim can cite many evidence objects, and each citation can fail independently. |
| Archive event | archive identifier, member identifier, add/remove action, rule or policy, timestamp | Membership history is event-sourced. The current archive is a derived view. |
| Run and stream | run identifier, stream identifier, invoker identity, parent run when applicable | Parallel agents, tools, humans, and instruments can emit into separate streams without requiring separate schemas. |
Table A.1: A minimal event contract for portable artifact-centered traces.
| Query | Records read | Expected answer |
|---|---|---|
| List unsupported final claims | Final report archive, claim individuals, evidence bindings, verification fitness | Claims with no supporting evidence, failed verification, stale evidence, or explicit unsupported status. |
| Trace a number in the manuscript | Claim individual, locator, evidence binding, source table or run artifact, verifier record | The evidence object, extracted value, claimed value, and pass/fail status. |
| Recover the best candidate’s ancestry | Archive event, selected individual, lineage operators, parent individuals, fitness records | The branch or population history that produced the selected artifact. |
| Detect archive collapse | Archive events, lineage graph, payload hashes, diversity metrics when present | Whether high-scoring candidates share recent ancestors, duplicate payloads, or one evaluator loophole. |
| Find the cause of a plan revision | Plan individuals, revision operators, context references, message operators, steering commands | The result, reviewer artifact, tool failure, or human intervention that changed the plan. |
| Audit self-modification | Agent-policy individuals, self-modification operators, later operator context | Which agent version produced each result and whether evaluator or policy changes preceded acceptance. |
Table B.1: Schema-level audit queries enabled by the event contract.
| Case | Minimal event sequence | What becomes inspectable |
|---|---|---|
| Hallucinated number | draft_4 extract_claim claim_23 . claim_23 binds to run7_metrics . metric_checker_v2 emits fail with observed value rather than claimed value . | The sentence is no longer buried in prose. The trace identifies the claim, the evidence table, the extracted value, the checker version, and the repair or propagation path. |
| Evolutionary collapse | program_87 and program_91 produce children through mutation and crossover. Benchmark fitness rises. Archive events admit near-duplicate payload hashes under one selection rule. | Improvement can be separated from diversity. A reviewer can see whether the archive exploited a benchmark corner rather than discovering robust alternatives. |
| Plan revision | A runner artifact reports a failed experiment. A critic stream emits a review. A human steering command changes the priority. revise_plan derives plan_v2 from plan_v1 with both records as context. | The plan change has a visible cause. Cross-agent handoff and human steering are part of lineage rather than chat transcript residue. |
| Closed-loop novelty | sample_s17 , protocol_p4 , and xrd_file_s17 feed an extraction operator. A novelty rule emits a verdict. A manuscript claim binds to the rule, reference set, and instrument file. | A later correction can identify whether the disputed claim came from the measurement, the extraction, the database comparison, or the novelty rule. |
| Self-modification | agent_v2 edits its prompt and evaluator definition, producing agent_v3 . Later candidate-generation operators list agent_v3 as context. | The trace remains acyclic by version while showing that the process generating later artifacts changed. |
Table C.1: Small trace fragments for diagnosing common audit failures.
| Package component | Contents | Failure mode avoided |
|---|---|---|
| Event stream | JSONL, database export, or graph export containing the contract in Table A.1 | Reviewers receive files but cannot reconstruct relations among claims, evidence, and evaluations. |
| Payload manifest | Hashes or URIs for drafts, code, tables, datasets, notebooks, instrument files, figures, and external references | A trace points to objects whose identity changed after the run. |
| Validator report | Reference checks, acyclicity checks, required-field checks, final-claim coverage, redaction accounting | Missing or malformed relations are discovered only during manual review. |
| Inspection queries | Saved queries for unsupported claims, lineage, archive history, steering effects, and redaction boundaries | Each reviewer reconstructs a different audit procedure. |
| Telemetry export | Span identifiers, model metadata, tool calls, latency, token counts, errors, and links to produced individuals | Execution debugging is separated from scientific artifact audit. |
| Archival export | PROV-O, RO-Crate, OpenLineage, or domain archive bundle when applicable | Final research objects lose the running history that explains how they were selected. |
Table D.1: A compact package for review, benchmarking, or later archival export.
More in this topic
AI Agent Auditing
AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.
See all 261 papers