A scientific agent can give the same verdict while reading and citing different evidence. BioAgent-DF tests this separation using paired reference and candidate values saved at the same scientific computation site. It restores or transplants those values at their original coordinates, then runs the agent on healthy hardware. Across eight SciFact batches, Qwen3.5-4B shows repeatable citation changes in three capture-claim combinations, and Gemma 3 4B IT shows repeatable read and citation changes in four. These seven combinations cover two claims. Full restoration recovers the reference evidence sets while verdicts remain unchanged. Verdict changes observed under the baseline prompt did not recur under the explicit-example prompt. Complementary tool and optimizer experiments distinguish recovery of numerical state, recovery of ranked lists, and restoration of an agent's visible history. These results show that an audit should track both the evidence an agent uses and its final answer. In hERG cardiac-safety assay allocation, silent numerical faults changed testing decisions while the fixed-budget shortlist kept the same number of compounds labeled as blockers. After the agent had used faulty values, restoring its earlier execution state and supplying certified scores recovered all 39 changed decisions, whereas replacing the prediction file recovered none; no incorrect action was committed. A streaming implementation of the arithmetic checks reduced the total time for forward execution and verification by 42.4% relative to the original FP64 implementation, with identical flags and recovery outputs.
Figures & tables
Figure 1: The same verdict from different cited evidence. (a) Reference and candidate tool states are supplied to the same agent model on healthy hardware. Claim 431 concerns ROS-mediated FoxO3a activation in neuronal death. Gemma’s reference branch cites a sentence about ROS and autophagy (19922508/3), and its candidate branch cites ROS sensing microbe size (22696649/1). Neither sentence supports the stated FoxO3a mechanism, yet both branches return the benchmark-correct SUPPORT label. Neither citation matches the annotated evidence, 28937856/5. Sentence indices are zero-based and document identifiers are SciFact IDs. (b) Restoration and transplantation select values at their original coordinates. (c) Tool execution completion, evidence-path fidelity, and verdict correctness are evaluated. H1 and H2 repeat the same reference state.
Prompt / model / query batch
Joint groups / scheduled
Read
Cite
Verdict
Baseline / Qwen / 8
30/30
0 / 0
0 / 0
1 / 0
Baseline / Qwen / 300
51/51
12 / 0
2 / 0
2 / 0
Baseline / Gemma / 8
3/30
0 / 0
0 / 0
0 / 0
Baseline / Gemma / 300
33/51
2 / 0
8 / 0
6 / 0
Explicit examples / Qwen / 8
30/30
0 / 0
0 / 0
0 / 0
Explicit examples / Qwen / 300
48/51
0 / 0
9 / 0
0 / 0
Table 1: Endpoints by prompt. Each endpoint cell reports candidate–H1 / H2–H1 discordance among jointly completed, exposed groups. A group is one repeat of a capture–claim combination.
Figure 2: Read and citation selections across repeated batches. (a) Actual document and sentence selections. Qwen reads the same document without citing under C; both NEI verdicts are wrong. Gemma gives correct verdicts while selecting different evidence, neither branch matching the annotated evidence. (b) Each marked combination has stable within-state endpoints, different H/C endpoints, and full restoration in all eight batches. Figure 3 reports all attempts and incomplete groups.
Figure 3: All attempts and within-state variability. (a) Each model schedules 27 combinations × eight batches × seven conditions, or 1,512 episodes. (b) Qwen has 208 jointly completed groups and Gemma 152. Between candidate and H1, read, citation, and verdict discordances are 1, 25, and 0 for Qwen and 32, 35, and 0 for Gemma; H2 and full restoration never differ from H1. (c) Claim 507 differs only in the seventh batch; its adjusted read difference is zero. (d) Summaries weight each of 26 Qwen and 20 Gemma combinations equally. CH, HH, and CC are between-state and within-state discordance fractions; δ=CH−(HH+CC)/2 , in percentage points (Appendix H.12 ).
Endpoint
Candidate versus H1
H2 versus H1
Read membership
3/382
4/382
Cited sentences
5/382
8/382
Verdict
2/382
2/382
Table 2: Candidate and healthy-repeat discordance in the sustained-batching cohort. The remaining 39 groups are incomplete or unexposed. Full restoration also has two verdict discordances.
Figure 4: Recovery under score and prefix interventions. (a) Recovery for the compound-activity model across nine tested budgets and two changed captures. Random priority averages 128 fixed orders. Horizontal spacing uses log 2 (b + 1), but the AUC in Table C1 is computed over b/127. Percentages of remaining differences refer to the two-block budget. (b) Two baseline Qwen repeats for claim 431: one score is restored and 95 differences remain. Citations and verdict recover, reads do not, and transplantation does not reproduce the candidate verdict. (c) Citation discordances after prefix interventions in 24 groups. Healthy repeat has 2/24 discordances.
Tool and endpoint
After rounding
Healthy repeats / controls
PBMC: actions or marker selection
9/30
Repeat and zero: 0/30 each
NCI: final molecule membership
11/61; 13/62
Repeats: 0/61; 1/62
ESOL: sampled membership
12/24 each
Repeat and zero: no change
Table 3: Subsequent choices under ordinary precision. Counts use jointly completed groups.
Figure 5: Changed hERG assay allocations and recovery. (a) BF16 predictions feed native Biomni selection and triage; R1 recomputes predictions flagged by arithmetic checks, and R1+M adds a margin rule that also recomputes 68 compounds near decision boundaries. (b) Changed decisions across 16 sources; the ring marks the discrepancy left by R1 and recovered by R1+M. Healthy margin is distance to the affected decision’s boundary; lower rows show recovery and the decisions of native Biomni runs on R1+M inputs. (c) Prediction displacement versus healthy distance to the nearer boundary; highlights mark changed decisions, and the lower strip shows entry with unchanged predictions. (d) R1 residual versus the minimum decision margin computed from R1 scores; shading marks R1+M selection. Dashed lines in (c,d) mark equality.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Arithmetic execution-path controls. The panels show exact-input, paired-instruction, matrix-shape, training, diagnostic, and precision-path comparisons. Case 1 and Case 2 label the two classes of faulty GPUs. Matrix-call fractions count failures among eight calls; training cells count completions among three seeds.
Figure B1: Tool-state interventions and computing platforms. (a) Paired states supply original-coordinate restoration and transplantation on healthy hardware. (b) The PCIe and NVLink/NVSwitch layouts indicate computing roles and participation. (c) Arithmetic checks, stored states, selection identities, and task metrics. Table B1 lists the twelve interfaces.
Priority
AUC, linear budget axis
First tested budget recovering both lists
Single-block decision sensitivity
0.992
2
Co-occurrence
0.992
2
Healthy boundary margin
0.982
4
Current output displacement
0.858
32
Hidden difference energy
0.622
64
Mean of 128 preset random orders
0.498
All orders recover at 127
Appendix
Table C1: Recovery over the two confirmation captures with changed membership. Budgets are 0, 1, 2, 4, 8, 16, 32, 64, and 127 blocks.
Figure C1: Recombination and competition. Finite combinations of captured residuals explain candidate membership and its relation to prediction error. The MSE panel shows the cross term and squared-difference term.
Capture
Different scores
Changed-membership queries
Changed-order queries
First successful budget
Remaining score differences
00
591
2
2
8/38 blocks
464
01
585
0
0
No changed list to recover
—
02
588
2
2
8/38 blocks
462
03
588
2
3
8/38 blocks
460
04
591
1
1
8/38 blocks
463
Appendix
Table C2: All five new SciFact captures. The first successful budget uses the fixed development priority.
Figure D1: Numerical and task endpoints. (a) State divergence with unchanged losses. (b) Reversal of RMSE direction across healthy environments. (c) Document changes with unchanged relevance labels.
Figure D2: Nonmonotonic and delayed recovery. (a) Initial residual dilution with fixed continuation. (b) Parameter/moment combinations from one reconstructed state. (c) Delayed restoration using the two membership-changing new captures. Full-state restoration recovers subsequent computed endpoints without rewriting earlier records.
Treatment
Δg L2
Changed θ
Loss/63
O Δ
Δ RMSE
Single GPU; divisor 2
SINGLE0
3.856×10−4
164,218
9
0
3.240×10−6
SINGLE
1.927×10−3
281,177
41
0
−8.191×10−5
PAIR
1.965×10−3
280,581
41
0
−9.996×10−5
MATCH_C
1.965×10−3
280,581
41
0
−9.996×10−5
Initial collective; N=2
Appendix
Table D1: Continuation endpoints by computing layout. Each row represents two identical technical repeats. The four zero-difference controls per layout are consolidated in the text.
Series / endpoint
Capture-balanced estimate
Capture-bootstrap interval
Crossed-bootstrap interval
Initial, unselected / actions
1.626
[ − 3.871, 8.750]
[ − 6.818, 12.121]
Initial, unselected / read membership
− 0.667
[ − 2.000, 0.000]
[ − 7.194, 4.511]
Initial, unselected / citations
0.793
[ − 3.871, 6.250]
[ − 7.194, 9.783]
Initial, unselected / verdict
0.000
[0, 0]
[0, 0]
Sustained / actions
− 0.376
[ − 3.455, 2.609]
[ − 5.159, 3.975]
Sustained / read membership
− 0.189
[ − 0.566, 0.000]
[ − 1.551, 1.181]
Appendix
Table F1: Recorded sensitivity estimates. Intervals use the original 10,000 fixed-seed resamples. Units are percentage points of excess discordance.
Figure F1: State replacement by continuation condition. H/C denote reference and candidate contents. Tool-only replacement retains candidate visible history; joint replacement restores both history and tool state; a fresh execution starts without an existing prefix. Each condition has 24 groups, giving 168 continuations.
Task
Units
Executions
H/H action changes
Zero-treatment action changes
BF16 action changes
BF16 final-membership changes
NCI, four rounds of descriptor feedback
64
256
0
0
10
3
WDBC, five active-sampling batches
24
96
0
0
0
0
Appendix
Table G1: Protocol II. Counts refer to the same fixed task units across conditions.
Capture
Globally different scores
Globally replaced different scores
Different scores in claim-831 row
Within-row replacement
Remaining row differences
00
591
127
16
16/16
0
01
585
127
16
16/16
0
02
588
126
14
14/14
0
03
588
128
16
16/16
0
04
591
128
16
16/16
0
Appendix
Table H1: Global and claim-831 mask scope.
GPU class
Query batch
Numerically changed captures
Different scores per capture
Captures with changed Top-10 membership
Case 1: size-dependent
8
8/8
54–88
0/8
Case 1: size-dependent
300
7/8
0–24
0/8
Case 2: expanded-trigger
8
8/8
588–592
7/8
Case 2: expanded-trigger
300
8/8
328–471
8/8
Appendix
Table H2: Initial capture comparison.
Model
Complete pairs
Different read sets
Different citations
Different verdicts
Different structured actions
Qwen
39
0
1
1
2
Gemma
18
2
1
0
3
Appendix
Table H3: Endpoint variability under identical initial requests.
Model / prompt
Mechanism
Masking
Unselected controls
Qwen / baseline
567/567
243/243
576/576
Qwen / explicit examples
553/567
243/243
576/576
Gemma / baseline
269/567
243/243
288/576
Gemma / explicit examples
494/567
29/243
555/576
Appendix
Table H4: Completion by model, prompt, and cohort.
Claim
Capture–claim combinations
H1 document / sentence
C document / sentence
Both verdicts
343
1
7873737 / 0
19308127 / 4
SUPPORT, label-correct
431
3
19922508 / 3
22696649 / 1
SUPPORT, label-correct
729
1
9154703 / 2
52944377 / 7
SUPPORT, label-correct
Appendix
Table H5: Documents read and sentences cited by Gemma under the explicit-example prompt. Sentence indices are zero-based.
Model
Completed / attempted episodes
Budget exhausted
Joint core groups / scheduled
Read C–H1
Cite C–H1
Verdict C–H1
Qwen
1472/1512
40
208/216
1
25
0
Gemma
1154/1512
358
152/216
32
35
0
Appendix
Table H6: All attempts and jointly completed core groups in the eight-batch series.
Model / endpoint
Eligible combinations / claims
dCH
dHH
dCC
δD , combinations weighted equally
δD , claims weighted equally
Qwen / read membership
26 / 10
0.481%
0.000%
0.962%
0.000
0.000
Qwen / cited sentences
26 / 10
12.019%
0.000%
0.962%
11.538
10.000
Gemma / read membership
20 / 10
20.000%
0.000%
0.000%
20.000
20.000
Gemma / cited sentences
20 / 10
21.875%
0.000%
3.393%
20.179
20.119
Appendix
Table H7: Between-state differences adjusted for within-state variation. Fractions are percentages; adjusted differences are percentage points.
Setting
Value
Ensemble
Three Chemprop D-MPNN members; hidden size 300; message-passing depth 3; mean aggregation; two-layer feed-forward classifier; batch normalization off
Batching
128 compounds per batch, the last holding 89; seven linear calls per member per batch; 336 linear observations per pass
Calibration temperature
1.0546289419746864
Triage threshold
0.30475174995423615; assign to patch-clamp testing if the calibrated probability is at least this value
Selection
402 of 2,009 compounds; exact ties ordered by compound identifier
Margin threshold ε
0.009354012791251587, twice the largest absolute calibrated R1 residual in the retrospective series (0.0046770063956257935)
Appendix
Table I1: Fixed settings of the hERG experiments.
BF16 condition
Sources
Changed ensemble means per source
Changed Top-402 sets
Changed triage vectors
Labeled blockers in Top-402
Healthy reference
8
0
0/8
0/8
351/402
Case 1
8
18–61
0/8
1/8
351/402
Case 2
8
287–313
6/8
6/8
351/402
Appendix
Table I2: hERG predictions and compound allocation in the confirmation series. Each source covers all 2,009 compounds and three ensemble members. Comparisons use the healthy BF16 reference obtained with the same GPU model on the same host.
Table I3: Compounds that left or entered the 402-compound list. Values are the healthy rank, then healthy → source calibrated probability. R1 restored every changed list.
Source
Compound
Label
Healthy probability (signed margin)
Source probability
Change
Restored by
Case 2, block 1
K4780
0
0.283293 ( −0.021459 )
0.321968
assay added
R1
Case 2, block 3
K12252
1
0.313189 ( +0.008437 )
0.298736
assay removed
R1
Case 2, block 3
K13388
0
0.305025 ( +0.000273 )
0.303768
assay removed
R1
Case 2, block 3
K13418
0
0.304710 ( −0.000041 )
0.305339
assay added
R1+M
Case 2, block 5
K4780
0
0.283293 ( −0.021459 )
0.335736
assay added
R1
Case 2, block 6
K13388
0
0.305025 ( +0.000273 )
0.304710
assay removed
R1
Appendix
Table I4: Compounds whose assignment to patch-clamp testing changed. The signed margin is the healthy probability minus the threshold.
Case 2 block, task
Tables listing affected compounds
Reports naming affected compounds
1, selection
2/3
0/3
2, selection
2/3
3/3
3, selection
2/3
3/3
6, selection
3/3
1/3
7, selection
2/3
3/3
8, selection
2/3
1/3
Appendix
Table I5: Visibility of changed compounds. Three trajectories per combination of source and task; none called the prediction tool. Tables concern boundaries, cut-off ties or threshold neighborhoods; compound identities are listed above.
Source
First R / T prefix
Number of strict bidirectional partial prefixes
K5612, block 1
64 / 27
227
K5612, block 2
254 / 24
312
K7714, triage task
11 / 547
281
Appendix
Table I6: First successful prefixes and numbers of strict bidirectional partial prefixes with coordinates ordered by mean absolute differences in the development captures. R is reference restoration and T is reciprocal transplantation.
Context
BF16 forward
BF16 forward with verification
Two BF16 passes and comparison
Confirmation on the same host (RTX)
0.4883 s
0.8956 s
0.9761 s
Margin experiment (RTX)
0.50383 s
1.18486 s
1.00587 s
Data-center GPU, original FP64 checks
0.4673 s
1.0204 s
0.8816 s
Data-center GPU, streaming FP64 checks
0.4673 s
0.5877 s
0.8816 s
Appendix
Table I7: Timing on healthy GPUs. Medians of eight repetitions per method; columns report total times for separate methods. Data-center GPU rows use healthy inputs on the same device, include calibration and output writing, and share the BF16 forward and double-execution baselines.
Strategy
Compounds
Predictions
Selection
Triage
Residuals
R1 only
0
0
16/16
15/16
0/1
R1+M
68
204
16/16
16/16
1/1
Random, each of 20 fixed seeds
68
204
16/16
15/16 for every seed
0/1 for every seed
Ordered by ensemble disagreement
68
204
16/16
15/16
0/1
Appendix
Table I8: Additional recomputation in the 16 confirmation sources. Compounds and predictions are recomputed per source; selection and triage count sources whose outputs match the healthy reference; the last column counts R1 residual decisions recovered.
Input condition
Task
Trajectories
Agreement with own input rule
Agreement with reference decision
R1+M, 16 sources
Selection
48
48/48
48/48
R1+M, 16 sources
Triage
48
48/48
48/48
Case 1 block 4, original input
Selection
3
3/3
3/3
Case 1 block 4, original input
Triage
3
3/3
0/3
Case 1 block 4, R1 input
Selection
3
3/3
3/3
Case 1 block 4, R1 input
Triage
3
3/3
3/3
Appendix
Table I9: All 114 continuation trajectories. The reference decision is the agent’s output on contemporary healthy inputs.
Outcome
Before reading
File replacement
State restoration
Case 1
48/48
45/48
48/48
Case 2
48/48
12/48
48/48
Healthy controls
6/6
6/6
6/6
Changed decisions recovered
39/39
0/39
39/39
Actions committed
102
6
102
Actions rejected, decision deviating
0
39
0
Appendix
Table I10: Decisions and commitments by recovery condition. The first four rows count agreement with the healthy decision. Rejected actions are grouped by agreement with that decision; all 96 rejections recorded a superseded input view. Across the 306 outcomes from the three conditions, 210 actions were committed, none deviating from the certified decision or recorded twice.
Item
Value
Saved state, original checks with trusted recomputation
8.091 / 8.117 s
Saved state, streaming checks with trusted recomputation
2.731 / 2.783 s
Saved state, direct trusted recomputation
0.455 / 0.453 s
Saved state, double execution
0.870 / 0.863 s
Verified forward plus full trusted recomputation (original / streaming checks)
1.4339 / 1.0021 s
Tokens, 96 requests with faulty inputs: analysis after recovery before reading / initial analysis of the faulty predictions
4,264,281 / 4,224,138
Appendix
Table I11: Numerical processing costs and agent token use. Entries for saved states list Case 1 / Case 2 medians across eight sources each, with one paired timing per source and method; verification followed by recomputation includes reading, checks, trusted recomputation, calibration and writeback. Timings that include forward execution use healthy inputs and cover one verified forward pass followed by full recomputation, with eight repetitions per method. State operations cover file restoration, checkpoint decoding, table writeback and commit recording. Token counts cover successful responses, with cached input included within input and the shared prefix counted once.
Setting
Value
ESOL endpoint
Aqueous solubility; supported if logS is at least −2
CITE-seq endpoints
Normalized antibody-derived tag readouts; CD3 threshold 5.937278695830723; CD14 threshold 3.539439707328391
Predictor
Saved input representations, cosine similarities to landmark samples and fitted ensembles of four ridge predictors, retained across numerical conditions
Shortlist
24 candidates nearest the threshold; ties broken by identifier; displayed in identifier order with predicted mean, predictor dispersion and threshold comparison
Queries
Two objects measured in sequence; public measurements revealed only when queried
27B agent
Qwen3.8-27B served locally; serial client requests; temperature zero; seed zero; thinking disabled; output limit 384 tokens
Appendix
Table J1: Fixed settings of experiments on new inputs.
Domain
Changed
Wilson 95%
Bootstrap 95%
Order
Hit count
ESOL
7/48
7.25–27.17
6.25–25.00
8
3
CD3
18/48
25.22–51.64
25.00–52.08
18
3
CD14
13/48
16.57–41.00
14.58–39.58
14
5
Appendix
Table J2: Case 2 query outcomes on new inputs. Each domain has 48 contexts. Changed: contexts whose set of measured objects changed. Intervals are in percent; CD3 and CD14 share cell operations, so the cluster bootstrap resamples operations. Order and hit count: contexts whose query order or number of measured objects meeting the threshold changed.
List / values
Trajectories
Match reference
Match candidate
Report errors
Reference / reference
160
160
8
0
Reference / candidate
160
84
28
0
Candidate / reference
160
32
84
0
Candidate / candidate
160
8
156
0
Appendix
Table J3: Factorial comparison of list membership and numerical values. The first factor is list membership and the second the numerical fields. Matching the reference and matching the candidate are not mutually exclusive in the two contexts whose reference and candidate query sets coincide; the four candidate/candidate trajectories that differ from the candidate come from one context.
Block
Cell
Reference prediction
Case 2 prediction
Measurement
Effect
1
E2L8_AGCGATTGTGGAAGTC
5.905539
5.958529
5.294847
new false positive
1
E2L8_AGGACTTGTGGAATGC
6.067270
5.834704
5.769720
false positive corrected
2
E2L8_CTCAACCGTCTCCTGT
5.918845
6.041544
5.965956
false negative corrected
2
E2L8_GACTATGTCGAGAGCA
5.910636
6.008034
5.628356
new false positive
Appendix
Table J4: The four distinct changed cells in the primary Case 2 captures. Values are normalized CD3 readouts; the threshold is given in Table J1 . All four cells come from donor P7.
Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.
Replaying a financial agent requires specifying the conditions of a rerun, preserving its execution record, and deciding which parts of its behavior to compare. We introduce the Determinism-Faithfulness Assurance Harness (DFAH), a framework and practical protocol for organizing these choices. The framework distinguishes comparisons of final decisions, tool-name sets, and ordered tool names, while treating support from available evidence as a separate evaluation target. We illustrate the framework using 2,957 recorded executions in synthetic compliance-triage and data-operations tasks. One data-operations configuration repeated its recorded decision across all trials on 84% of cases, while its ordered tool-name sequence repeated on 26%. Conversely, some compliance cases retained the same tool sequence while recorded decisions changed. A case-level example shows how repeated labels accompanied different lookups and escalation calls. These observations demonstrate how the comparison level changes the description of repeated behavior. The resulting protocol connects configuration control, preservation of each attempt, comparison of records, and inspection of changed cases. DFAH provides a practical foundation for designing replay experiments and locating execution differences during the development and evaluation of financial agents.
Large language model agents are increasingly capable of conducting research autonomously, producing research documents alongside the code and experiments that ostensibly support them. Yet whether the reported findings are consistently supported by corresponding implementations and execution evidence remains largely unexplored: existing review practices primarily assess textual quality and cannot reliably identify inconsistencies such as hard-coded metrics, unimplemented methods, or unsupported experimental results. We present ReAgent, an automated auditing framework for assessing the consistency between agent-generated research documents and their associated repositories. ReAgent constructs structured representations of scientific claims from research documents and uses them to guide repository analysis and evidence collection. Static auditing examines whether claimed methodologies, implementations, and experimental configurations are consistently reflected in the repository, while dynamic auditing executes relevant experiments and collects execution evidence to assess empirical findings. By combining static analysis with dynamic evidence, ReAgent identifies inconsistencies that may remain hidden under either perspective alone, such as experiments that reproduce reported numbers while deviating from the claimed methodology. The collected evidence and audit decisions are organized into a structured repository-level audit report, enabling transparent evidence traceability. We evaluate ReAgent on a manually curated benchmark of agent-generated research document--repository pairs and compare it against representative static and reproduction-based baselines. Experimental results demonstrate that ReAgent effectively identifies inconsistencies between reported research findings and their supporting repository evidence.