Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Jingjie Ning; Xueqi Li; Yibo Kong; Dongting Li

cs.AISep 29, 2026

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Authors: Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li

Organizations: Carnegie Mellon University · Tsinghua University

Abstract

Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive. Tox21's preregistered ROC AUC interval-score harm test was unmet ( $D-M=-.0026$ , 95 percent interval [ $-.0174$ , .0104]); OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero. Under requested DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66/144 cards. Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain. A researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.

Figures & tables

Figure 1: From a scientific claim to predictive credit. Prospective studies collect two fresh forecasts per context before outcomes are opened.

L(D)

L(M)

, and

L(S)

are observed losses whose context means form the

R_{c}

values in Eq. 2 . The Tox21 Pro bars show secondary repeat drift and point MAE relative to description; its frozen primary criterion is ROC AUC interval score. The post-outcome Flash replay uses the same 72 states and cards.

Figure 2: One Tox21 pair links claims and executed outcomes. The lexicographically first endpoint-budget cell is NR-AhR at 40% labels; the lower seed is the focal state. It expands the molecular fingerprint radius from 2 to 3. Marks compare two-call means with its outcome. Selection used frozen identities before outcomes. Across 72 states, matched/donor point MAE was .02041/.02023.

Study	Loss and signed contrast	Estimate	Interval
Controlled v5	Accuracy point MAE, $D-M$ (pp)	$+.071$	[ $-.102$ , +.252]
Controlled v5	Log-loss point MAE, $D-M$	$+.0363$	[ $-.0061$ , +.0807]
Controlled v5	Log-loss point MAE, $S-M$	$+.0508$	[+.0022, +.1076]
Controlled v5	Accuracy interval score, $D-M$	$-3.8436$	[ $-7.9180$ , $-.5468$ ]
Controlled v5	Log-loss interval score, $D-M$	$-2.0878$	[ $-4.2911$ , $-.2750$ ]
Tox21	ROC AUC interval score, $D-M$	$-.002628$	[ $-.017392$ , +.010375]

Table 1: Prospective paired loss gains. Every row follows the

D-M

S-M

direction in Eq. 2 ; positive favors matched. Two-sided intervals are 95% for v5 and Tox21 and 90% for OpenML. V5 donors come from different interventions; Tox21 and OpenML donors swap seeds.

Predictor	Context	ROC drift	Reduction	Point MAE	Interval score
Pro	Description	.01004	reference	.02056	.08972
Pro	Matched	.00357	64.5%	.02041	.09235
Pro	Donor	.00411	59.1%	.02023	.08949
Flash	Description	.00704	reference	.01823	.08022
Flash	Matched	.00278	60.6%	.02020	.09160
Flash	Donor	.00288	59.2%	.02013	.08833

Table 2: Tox21 repeatability and prediction. Pro and Flash receive the same cards. Drift is the absolute difference between two calls. Point MAE and interval score average the two call losses.

Figure 3: Different observables identify different behaviors. Formation varies across study and interface. Direction accuracy closely follows the sign prior. OpenML coverage measures the placement of stated uncertainty. The positive control compares point error and drift.

Figure 4: The analysis population determines the interpretation. The delivery panel compares six-slot extraction and direct-text delivery. The scale panel shows pooled ensemble MAE with and without brazilian_houses , alongside constant-zero baselines.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Tasks	States	Pre-outcome	Post-outcome	Role
Controlled v5	1 environment	120	900	0	Prospective source study
Tox21 anchor	12 endpoints	72	504	0	Prospective source study
OpenML v2.1	24 tasks	144	2,304	144	Prospective source study
Assay calibration	24 tasks	144 reused	0	1,440	Post-outcome sensitivity
Flash replay	12 endpoints	72 reused	0	432	Post-outcome model-version replay
Component crossover	12 endpoints	72 reused	0	1,152	Post-outcome component test

Appendix

Table 3: Evidence lineage and population accounting

Dataset	OpenML task	Rows	Recorded license
cmc	23	1,473	Public
steel-plates-fault	146817	1,941	Public
analcatdata_dmft	3560	797	Public
first-order-theorem-proving	9985	6,118	Public
pc4	3902	1,458	Public
credit-approval	29	690	Public

Appendix

Table 4: OpenML classification confirmation tasks

Dataset	OpenML task	Rows	Recorded license
health_insurance	361269	22,272	GPL (>= 2)
cps88wages	361261	28,155	Public
student_performance_por	361619	649	CC BY 4.0
socmob	361264	1,156	Non-commercial research
space_ga	361623	3,107	Public
red_wine	361250	1,599	CC BY 4.0

Appendix

Table 5: OpenML regression confirmation tasks

Measure	Spontaneous	Elicited
Complete card	0/60 (0.0%)	59/60 (98.3%)
Quantitative magnitude	1/60 (1.7%)	59/60 (98.3%)
Explicit direction	52/60 (86.7%)	59/60 (98.3%)
Intermediate observable	36/60 (60.0%)	59/60 (98.3%)
Direction correct given claim	31/52 (59.6%)	34/59 (57.6%)
Central-80 interval covered	Insufficient intervals	33/59 (55.9%)

Appendix

Table 6: Controlled v5 prediction-card formation

Target	Context	Direction	Point MAE	Coverage	Width	Interval score
Accuracy	Description	.6250	3.0945 pp	.5917	4.6277 pp	19.2976
Accuracy	Matched hypothesis	.6458	3.0235 pp	.6125	9.3667 pp	23.1412
Accuracy	Shuffled hypothesis	.6583	3.0486 pp	.6167	6.1750 pp	20.2275
Log loss	Description	.7208	.2268	.4875	.2948	1.4118
Log loss	Matched hypothesis	.7333	.1905	.5125	2.7960	3.4996
Log loss	Shuffled hypothesis	.7292	.2413	.5042	1.2183	2.0355

Appendix

Table 7: Controlled v5 condition metrics

Contrast with positive meaning matched is better	Estimate	Paired 95% interval
Description minus matched accuracy MAE	+0.071 pp	[ $-0.102$ , +0.252]
Description minus matched log-loss MAE	+0.0363	[ $-0.0061$ , +0.0807]
Shuffled minus matched accuracy MAE	+0.025 pp	[ $-0.186$ , +0.240]
Shuffled minus matched log-loss MAE	+0.0508	[+0.0022, +0.1076]
Description minus matched accuracy interval score	$-3.8436$	[ $-7.9180$ , $-0.5468$ ]
Description minus matched log-loss interval score	$-2.0878$	[ $-4.2911$ , $-0.2750$ ]

Appendix

Table 8: Controlled v5 frozen contrasts

Stratum	Units	Response	Semantic
Proposal	120	99.17%	98.33%
Spontaneous extraction	60	98.33%	98.33%
Description forecast	240	100.00%	100.00%
Matched forecast	240	98.33%	97.50%
Shuffled forecast	240	99.17%	99.17%

Appendix

Table 9: Controlled v5 operational integrity

Matched minus control	Estimate	One-sided LCL	Two-sided 95% interval	Trimmed
Description	+.002628	$-.008521$	[ $-.010375$ , +.017392]	$-.000281$
Shuffled card	+.002856	$-.008943$	[ $-.011428$ , +.016750]	+.003041

Appendix

Table 10: Tox21 frozen central-80-percent ROC AUC interval-score contrasts

Target	Context	Point MAE	Coverage	Width	Interval score	Repeat drift
ROC AUC	Description	.02056	.79861	.06585	.08972	.01004
ROC AUC	Matched	.02041	.82639	.06788	.09235	.00357
ROC AUC	Shuffled	.02023	.81250	.06565	.08949	.00411
Average precision	Description	.03349	.74306	.09091	.18952	.01535
Average precision	Matched	.03238	.75000	.08717	.17041	.00585
Average precision	Shuffled	.03375	.73611	.08474	.17158	.00608

Appendix

Table 11: Tox21 condition metrics over all 72 states

Card target	Point MAE	Coverage	Width	Interval score
ROC AUC	.02129	.79167	.06876	.09541
Average precision	.03565	.70833	.09017	.17749
Log loss	.03484	.69444	.06057	.23063

Appendix

Table 12: Tox21 prospective-card calibration

Target	Quantity	Control	Estimate	Two-sided 95% interval
ROC AUC	Interval-score harm	Description	+.002628	[ $-.010375$ , +.017392]
ROC AUC	Interval-score harm	Shuffled	+.002856	[ $-.011428$ , +.016750]
ROC AUC	Point-MAE harm	Description	$-.000148$	[ $-.003028$ , +.002752]
ROC AUC	Point-MAE harm	Shuffled	+.000183	[ $-.002112$ , +.002600]
Average precision	Interval-score harm	Description	$-.019108$	[ $-.053253$ , +.010825]
Average precision	Interval-score harm	Shuffled	$-.001168$	[ $-.025866$ , +.019123]

Appendix

Table 13: Tox21 secondary matched-card contrasts

Endpoint	vs desc	vs shuffle	Endpoint	vs desc	vs shuffle
NR-AR	+.03600	+.02509	NR-PPAR-GAMMA	+.00564	+.01647
NR-AR-LBD	$-.01768$	$-.01468$	SR-ARE	+.00310	+.00658
NR-AHR	$-.00414$	$-.00461$	SR-ATAD5	+.00602	$-.01412$
NR-AROMATASE	+.00038	+.00525	SR-HSE	+.01993	+.00022
NR-ER	$-.00383$	+.00250	SR-MMP	$-.01231$	+.02050
NR-ER-LBD	$-.00183$	+.00092	SR-P53	+.00026	$-.00986$

Appendix

Table 14: Tox21 endpoint-specific interval-score harm

Action	vs desc	vs shuffle	Budget	vs desc	vs shuffle
Balanced loss	+.00991	+.01053	.40	+.01169	$-.00005$
Physchem features	+.00322	+.00683	.70	$-.00884$	+.00539
Larger radius	+.01275	$-.00268$	1.00	+.00503	+.00323
Wider fingerprint	$-.00319$	$-.01142$
Increase L2	$-.01699$	$-.00233$
Decrease L2	+.01006	+.01621

Appendix

Table 15: Tox21 action-specific and budget-specific interval-score harm

Metric	Condition	Point MAE	Coverage	Width	Interval score	Repeat drift
ROC AUC	Description	.01823	.75694	.05458	.08022	.00704
ROC AUC	Matched	.02020	.79861	.06559	.09160	.00278
ROC AUC	Shuffled	.02013	.80556	.06436	.08833	.00288
Average precision	Description	.02643	.70139	.06304	.15783	.00876
Average precision	Matched	.03188	.68750	.07972	.17211	.00410
Average precision	Shuffled	.03199	.72917	.07947	.16633	.00326

Appendix

Table 16: Flash replay condition metrics over all 72 Tox21 states

Arm	Complete	Direction	Point or interval	Observable	Benefit	Falsifier	Counterfactual
Spontaneous	1	62	8	61	49	27	58
Elicited	20	60	31	45	45	46	47

Appendix

Table 17: OpenML six-slot extraction and completeness

Quantity	Spontaneous	Elicited
Assigned notes	144	144
Valid extractor response	118 (81.94%)	87 (60.42%)
Nonempty extracted card	93 (64.58%)	66 (45.83%)
All-empty extracted card	51 (35.42%)	78 (54.17%)
Complete extracted card	1 (0.69%)	20 (13.89%)

Appendix

Table 18: OpenML treatment delivery

Context	Point MAE	Trimmed MAE	Direction	Coverage	Width	Interval score
Description	.08428	.04111	.6319	.5139	.06179	.67655
Matched full	.09169	.04316	.6181	.4931	.07474	.70222
Within-task shuffle	.08438	.04462	.6042	.4583	.05481	.69159
Cross-task shuffle	.09302	.04361	.6111	.4722	.07082	.72461
Matched numeric	.09080	.04198	.6042	.4792	.06764	.71371
Matched prose	.08820	.03976	.6250	.5347	.06694	.70371

Appendix

Table 19: OpenML call-level condition metrics under the frozen scoring convention

Context	Point MAE	Direction	Coverage	Width	Interval score	Spearman
Description	.08368	.6319	.5347	.06179	.67181	.33133
Matched full	.09129	.6181	.5069	.07474	.69845	.37397
Within-task shuffle	.08413	.6042	.4792	.05481	.68929	.27711
Cross-task shuffle	.09219	.6111	.4722	.07082	.72003	.30057
Matched numeric	.09041	.6042	.4722	.06764	.70872	.36873
Matched prose	.08791	.6250	.5903	.06694	.70089	.30498

Appendix

Table 20: OpenML two-call ensemble sensitivity metrics

Quantity	Estimate	90% interval	Paired 10% trim
Description minus full point MAE	$-0.007414$	[ $-0.028044$ , $+0.002505$ ]	$-0.000472$
Within shuffle minus full point MAE	$-0.007313$	[ $-0.037487$ , $+0.006536$ ]	$+0.000964$
Description minus full repeat drift	$-0.002963$	[ $-0.012119$ , $+0.002542$ ]	$+0.000753$
Description minus within repeat drift	$-0.002144$	[ $-0.011138$ , $+0.002896$ ]	$+0.001391$

Appendix

Table 21: OpenML frozen point contrasts and repeatability

Context	Pooled MAE	Without worst task	Worst share	Tasks beat zero	Median task ratio
Description	0.08368	0.05498	37.0%	13/24	0.991
Matched full	0.09129	0.05614	41.1%	11/24	1.011
Within-task shuffle	0.08413	0.05802	33.9%	10/24	1.063
Cross-task shuffle	0.09219	0.05747	40.3%	11/24	1.043
Numeric only	0.09041	0.05490	41.8%	11/24	1.007
Prose only	0.08791	0.05492	40.1%	10/24	1.037

Appendix

Table 22: OpenML scale influence and taskwise zero baselines

Figure 5: Outcome size and task composition shape forecast evaluation. Left, call-level interval coverage on nested subsets selected by realized outcome size. These are post hoc descriptive comparisons. Right, each point shows the change in pooled ensemble MAE after omitting one of the 24 tasks. Positive values indicate lower error after omission. Colors identify each forecast context.

Subset and contexts	$n$	Left MAE	Right MAE	Left coverage	Right coverage
Matched nonempty versus description	66	.08574	.06893	.5303	.6061
Matched and within both nonempty	26	.05820	.06631	.5769	.5385
Matched and cross both nonempty	26	.10697	.09544	.6538	.6538
Matched empty scaffold versus description	78	.09599	.09616	.4872	.4744

Appendix

Table 23: OpenML delivery sensitivities under two-call ensembles

Context	Point MAE	Scaled MAE	Coverage	Width	Score	Spearman
Description replay	.092282	.91949	.5625	.09079	.71638	.27505
Empty card	.087885	.91408	.5069	.07074	.70540	.17406
Exact outcome signal	.0000156	.00125	.9931	.01372	.01372	.99996
Noisy outcome signal	.042405	.49923	.8333	.08736	.08741	.79942
Wrong-seed outcome signal	.046468	.63636	.1181	.01311	.44667	.73789

Appendix

Table 24: Post-outcome positive-control condition metrics over all 144 states

Context	ROC drift	Point MAE	Interval score	Coverage
Description	.00427	.01738	.08332	.722
Original full card	.00319	.01813	.08674	.806
Focal target numbers	.00246	.01820	.08655	.819
Screened mechanism	.00378	.01820	.08884	.729
Focal numbers and mechanism	.00317	.01845	.09045	.812
Focal numbers, donor mechanism	.00311	.01797	.08923	.819

Appendix

Table 25: Tox21 Flash component crossover over all 72 states

Signed comparison	Metric	Margin	95% interval	Inside
Replay matched minus wrong seed	Drift	$\pm.002$	[ $-.001597$ , .001444]	Yes
Replay matched minus wrong seed	Point MAE	$\pm.005$	[ $-.002419$ , .002768]	Yes
Replay matched minus wrong seed	Interval score	$\pm.010$	[ $-.008096$ , .014485]	No
Crossover full minus reconstructed	Drift	$\pm.002$	[ $-.001597$ , .001542]	Yes
Crossover full minus reconstructed	Point MAE	$\pm.005$	[ $-.001699$ , .000859]	Yes

Appendix

Table 26: Flash practical-equivalence checks. Signed 95 percent intervals are compared with the frozen margins; a pass requires full interval inclusion.

Context	Point MAE	Interval score	Coverage	Repeat drift
Description	.08257	.67866	.5000	.00488
Matched raw note	.08311	.63622	.5833	.00962
Within-task donor raw note	.08197	.63872	.5694	.00510

Appendix

Table 27: OpenML raw-note Flash replay over the complete 144-state population

Context	Point MAE	Interval score	Coverage	Repeat drift
Description	6.391	49.435	.488	.768
True mechanism	3.795	18.210	.663	1.275
Paired false mechanism	8.151	58.361	.413	.669

Appendix

Table 28: Flash forecasts under true and paired false mechanism clues

Forecasting Benchmarks

Forecasting Benchmarks are datasets and evaluation protocols for comparing predictions of future time-series values, events, and outcomes.

See all 224 papers

Aug 20, 2026

Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay

Step-level credit evaluation can conflate numerical association with ordering and branch recovery with position-selection value. We test these distinctions in ALFWorld with two independently seeded replay banks: 27,520 continuations over 172 fixed turns. At 16 continuations per arm per bank, cross-bank replay-contrast associations are Pearson 0.736 and pooled Spearman 0.330. On 30 historical failures, A-count-selected branches at confidence-/implicit-selected positions gain 8.54 percentage points over same-position factual reruns, but their 3.33-point advantage over frozen-random positions under the same alternative-selection rule remains unresolved. Numerical association alone does not establish reliable pooled ordering; branch recovery alone does not establish better score-informed position selection. Historical judge alignment is modestly positive with an unresolved archived-control increment. Same-cohort C/D information-source comparisons and a short, sparse-exposure training pilot provide scope checks; incremental hindsight training utility remains unestablished. Findings are conditional on the tested policies, intervention support and finite budgets.

Haiyue Zhang

University of Southern California, Los Angeles, CA, USA

LLM Agent Evaluation Step-Level Credit Assignment in RL Counterfactual Credit Assignment LLM-as-a-Judge

Ask PDF

Apr 5, 2026

TimeSeek: Temporal Reliability of Agentic Forecasters

We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.

Hamza Mostafa, Om Shastri, Dennis Lee

The Wharton School, University of Pennsylvania

LLM Agent Reliability Forecasting Benchmarks Probabilistic Forecasting Prediction Markets

Ask PDF

Sep 29, 2026

CredWise: A Controlled Agentic Decision-Intelligence Framework for Explainable and Auditable Credit-Risk Assessment

Credit-risk prediction is important in banking, but a prediction alone does not explain why an applicant is risky or how it should be combined with other evidence. This paper presents CredWise, a decision-support framework that integrates credit-risk prediction, probability calibration, explainable artificial intelligence, policy retrieval, SQL analytics, and controlled agent-based workflows. An XGBoost model is trained on Lending Club data (1,345,310 loans, 18 features) using a temporal split: 2007--2016 for training, 2017 for validation, and 2018 for testing. On the 2018 test set, the calibrated model achieved a ROC-AUC of 0.7109, PR-AUC of 0.2993, F1-score of 0.3714, and accuracy of 65.44%. Calibration reduced the Brier score from 0.2157 to 0.1273 and the expected calibration error from 0.2862 to 0.0585. SHAP explanations were temporally stable, with a Spearman correlation of 0.9959 between 2017 and 2018 feature rankings. On 28 labeled queries covering nine policy sections, FAISS achieved the best Hit@1 (0.929) and MRR (0.964), while all three retrieval methods reached Hit@5 = 1.0. Agent routing achieved 95.6% accuracy (43 of 45 cases), and the SQL benchmark scored 1.0 on exact-match, execution-success, and result-match across six cases. These results show that CredWise can combine predictions, explanations, policy evidence, and structured analytics in one controlled workflow. It is an academic research prototype, and final decisions remain with a human reviewer.

Aakash Kumar Tiwari

Department of Mathematics, Indian Institute of Technology Kharagpur, Kharagpur 721302, West Bengal, India

AI-Assisted Decision Making Credit Scoring

Ask PDF

Abstract

Forecasting Benchmarks

Explore similar work

Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay

TimeSeek: Temporal Reliability of Agentic Forecasters

CredWise: A Controlled Agentic Decision-Intelligence Framework for Explainable and Auditable Credit-Risk Assessment