Equal-information evaluation of credit assignment in modular financial reinforcement learning

Gennadiy Chistiakov; Yury Yanovich

cs.AIOct 10, 2026

Equal-information evaluation of credit assignment in modular financial reinforcement learning

Authors: Gennadiy Chistiakov, Yury Yanovich

Organizations: Independent Researcher, Bangkok, 10100, Thailand · Skolkovo Institute of Science and Technology, Moscow, 121205, Russia

Abstract

Credit assignment in modular financial reinforcement learning is difficult to evaluate because an attributed signal may improve a component loss without adding decision-relevant information or improving economic outcomes. We propose an evaluation framework combining equal-information reconstruction controls, fixed-utility comparisons and prespecified downstream guardrails. The framework is instantiated in an attribution-guided modular trading architecture and tested in three controlled studies covering update allocation, supervision-target representation and portfolio risk control. We find that residual-based allocation is exactly reproduced by an ordinary loss-budget control under the disclosed interface; direct shortfall supervision improves component prediction but fails decoded-outcome preservation; and component-wise critic supervision worsens risk-condition certainty equivalent and drawdown relative to scalar supervision under fixed utility. These results identify three distinct evaluation failures: redundant attribution signals, representation-induced prediction trade-offs, and misalignment between component supervision and downstream risk outcomes. A preserved historical engineering case further shows why module-level learning claims require run-level provenance. The contribution is a reusable methodology for evaluating credit assignment in expert systems operating under partial observability, delayed feedback and economic risk constraints.

Figures & tables

Figure 1: Documented AMATS case architecture. The LLM proposes intent; risk control determines feasible exposure; execution creates orders and fills. An attribution record supports module-specific feedback. Arrows define the design’s information and learning paths.

Study	Intervention and evaluated outcome	Replication and boundary
Allocation (v2)	Residual versus uniform task-gradient allocation; standardized prediction loss	6 paired path/initialization seeds; 264 main/sensitivity runs; no account
Representation (v3)	Native outcome versus shortfall supervision, crossed with routing; common decoded errors	20 paired seeds; 1,280 runs; exogenous submitted episodes
Portfolio control (v4)	Scalar versus component critic supervision; CE, drawdown and net return	30 paired seeds; 2,940 cells; two synthetic assets in one account
Historical probe (Bar)	Fixed forecast-to-position policy with alternative update allocation	3 initialization seeds on 2 selected years and 2 separately funded assets; modeled execution

Table 1: Executed evidence and its independent experimental units. Paired configurations share a seed/path; runs, clocks and bootstrap draws remain distinct from independent sample sizes. These studies instantiate selected interfaces of the AMATS architecture in Section 3 .

Figure 2: V3 factorial contrasts on common

L

, with paired 99.5833% Bonferroni-adjusted intervals. Negative target/routing differences favor the first treatment; interaction is two-sided. The component gain does not imply that the separate outcome guardrail passes.

Condition	Difference	Adjusted interval	Noninferiority
Stationary	$+0.0395$	$[0.0083,0.0710]$	Not established
Direction shift	$-0.0110$	$[-0.0851,0.0630]$	Not established
Execution shift	$+0.0713$	$[0.0468,0.0973]$	Not established
Both shifts	$+0.0744$	$[0.0069,0.1421]$	Not established

Table 2: V3 target contrast on decoded

y

standardized MSE, with 99.7917% Bonferroni-adjusted intervals. Passing requires the upper bound below

0.05

Contrast (RL-RISK minus comparator)	Difference	Adjusted interval	Decision
All-condition CE minus FIXED	$-0.2817$	$[-0.3186,-0.2285]$	Not supported
Risk-condition CE minus RL-SCALAR	$-0.0619$	$[-0.1132,-0.0241]$	Not supported
Stationary CE minus RL-SCALAR	$-0.0215$	$[-0.0545,0.0013]$	NI not established
Risk-condition MDD minus RL-SCALAR	$+0.0151$	$[0.0053,0.0286]$	Not supported
Risk-condition net return minus RL-SCALAR	$-0.0145$	$[-0.0280,-0.0048]$	NI not established
All-condition CE minus JOINT-DRE	$+0.0597$	$[0.0187,0.1033]$	Supported

Table 3: V4 confirmatory paired contrasts with 99.1667% Bonferroni-adjusted intervals. CE is bps/clock; MDD and net return are fractions. Positive CE/return and negative MDD favor RL-RISK.

Figure 3: V4 prespecified paired contrasts and adjusted intervals. SCALAR and JOINT abbreviate RL-SCALAR and JOINT-DRE; all, risk and stat. denote aggregates. Exact intervals are reported in Table S4.2 and Supplementary Material S4. The plot marks zero, not NI margins.

Claim	Evidence	Conclusion
Scheduler specificity	Exact attributed/loss-budget equality in 24 simulated and 12 historical main pairs	No attribution-specific advantage for this interface
V3-H1	$L$ /IS improve; decoded- $y$ NI fails in all 4 conditions	Complete criterion not supported
V3-H2/H3	Adjusted routing and interaction criteria fail in all 4 conditions	Neither gain confirmed
V3-H4	Both specified residual-dominance intervals positive	Supported dominance, not causal diagnosis
V4-H1/H2/H3	Adverse aggregate CE/MDD; return guardrails fail	Complete risk criteria not supported
V4-H4	Positive all-condition CE contrast against JOINT-DRE	Supported only for the specified bundle

Table 4: Claim hierarchy for the executed studies. Decisions follow the complete prespecified criteria, including downstream guardrails. Original full-system H1–H4 remain separate evaluation questions in Supplementary Material S6.

Reported measure	Reported value
Sharpe	2.4356
Win rate	55.62%
Maximum drawdown, magnitude	16.81%
CAGR	136.92%
Sortino	1.0442
Profit factor	3.3163

Table 5: E01 historical report: transcribed point estimates, not independently reproduced performance estimates. Currency, annualization and complete accounting/configuration provenance remain unverified.

Figure 4: E01 equity redrawn directly from the preserved CSV, including the December 2020 observations. Units are the account units reported in the source. The line preserves every timestamp/value pair. Configuration and accounting provenance are discussed in Section 8 .

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1.1 : Deterministic quadratic diagnostic, not market data. (a) The equal-weight optimum sacrifices both task-specific optima. (b) Execution-loss weights switch through

1,4,1,0.25

in blocks of 25 steps, with

\eta=0.05

and

\theta_{0}=0

. The dashed optima are

0,-0.6,0,0.6

. These are loss-weight switches, not detected market-regime transitions.

Scenario	Opportunity	Shortfall	Funding	Net	MC SE
Reference	12.123	7.500	0.500	4.123	0.653
Execution shock	12.123	15.500	0.500	-3.877	0.653
Forecast shock	4.123	7.500	0.500	-3.877	0.653
Partial fills	12.123	9.811	0.250	2.061	0.326

Appendix

Table S1.1 : Synthetic diagnostic results, 2,048 paired opportunities per scenario. Values are bps of intended notional, not portfolio returns. MC SE is the standard error of the simulated mean under the imposed independent generator, not uncertainty about market performance.

Quantity	Learn	Bar
Decision clock; direction horizon	Abstract clock; $h=4$	One hour; $h=4$ hours
Normalization; feature/target scale floors	512 clocks; $10^{-8}$ / 0.1 bp	512 valid clocks; same floors
Separate / shared network	$2(11\!-\!8\!-\!1)$ / $11\!-\!15\!-\!2$	Same; tanh hidden, linear output
Parameter counts	$2\times 105=210$ / 212	Same
Adam $(\eta,\beta_{1},\beta_{2},\epsilon)$	$(0.003,0.9,0.999,10^{-8})$	Same
Batch / replay / eligibility	16 / 256 per task / 16 labels	Same

Appendix

Table S2.1 : Executed main-configuration parameters. These values specify the public small learners and simulator, not missing full-stack deployment choices.

Condition	Difference	95% interval	98.75% interval
Stationary	$+0.0133$	$[+0.0065,+0.0204]$	$[+0.0048,+0.0225]$
Direction shift	$+0.0066$	$[-0.0056,+0.0187]$	$[-0.0091,+0.0216]$
Execution shift	$-0.0809$	$[-0.1654,+0.0048]$	$[-0.1901,+0.0280]$
Combined shift	$-0.0231$	$[-0.1128,+0.0795]$	$[-0.1341,+0.1099]$

Appendix

Table S2.2 : Attributed minus uniform combined loss. Negative favors attributed allocation. Paired seed/time-block intervals use 20,000 resamples. The 98.75% column adjusts the four primary condition-specific contrasts; 95% intervals are descriptive.

Figure S2.1 : Actually trained small learners on simulated tasks. Five-method losses have descriptive 95% intervals; the paired allocation contrast has nested 95% (thin black) and 98.75% (thick gray) intervals. The attributed and ordinary loss-budget results coincide exactly. S/D/E/B identify stationary, direction-shift, execution-shift and combined-shift conditions. This tests the disclosed update scheduler, not full AMATS or market alpha.

Figure S2.2 : Complete one-factor sensitivity grid for combined shifts: attributed minus uniform standardized loss with exploratory 95% intervals. Each panel reuses the same main-study baseline point. At minimum share 0.5 the policies coincide. The grid does not select a deployable setting or establish component contributions in full AMATS.

Variant	Stationary	Direction shift	Execution shift	Both shifts
RAW-U	1.2731	1.8002	1.6989	2.2879
RAW-R	1.2654	1.7850	1.7014	2.2837
ATTR-U	0.9481	1.2198	1.4291	1.8750
ATTR-R	0.9495	1.2188	1.4346	1.8883
RAW-J	1.2627	1.7487	1.6894	2.2432
ATTR-J	0.9533	1.2125	1.4331	1.8821

Appendix

Table S3.1 : V3 equal-seed mean common loss

L

(20 seeds per cell). J/P are descriptive shared-network controls. RAW-IS rows are independently trained equality controls, not additional independent evidence.

Figure S3.1 : V3 factorial trajectories: equal-seed means of trailing 32-clock valid-observation common loss. These are descriptive curves, not confidence bands or additional observations. Vertical lines mark frozen alternating intervention boundaries in shifting conditions. All four conditions and modular treatments are retained.

Figure S3.2 : Complete v3 both-shift target, routing and interaction sensitivity contrasts on

L

, with descriptive paired 95% intervals. The baseline reuses 80 main cells; each other setting has 20 seeds and four variants. No sensitivity replaces the frozen primary result.

Condition	FIXED	VOL	RL-SCALAR	RL-RISK	PCGrad	JOINT-DRE
Stationary	0.1939	0.1650	-0.2700	-0.2916	-0.2926	-0.2908
Direction shift	-0.5219	-0.5162	-0.5089	-0.6515	-0.6284	-0.8476
Execution shift	0.1056	0.0877	-0.2204	-0.2309	-0.2323	-0.2138
Risk shift	0.0911	0.0683	-0.3311	-0.3639	-0.3630	-0.4488
Direction+risk	-0.6315	-0.1717	-0.6241	-0.7547	-0.7388	-0.8161
Execution+risk	0.0248	0.0513	-0.2507	-0.2707	-0.2654	-0.3085

Appendix

Table S4.1 : V4 seed-mean certainty equivalent in bps per clock. Each cell equally weights 30 complete test paths. Intervals and hypothesis decisions use paired seed differences, not comparisons of these marginal means.

Contrast (RL-RISK minus comparator)	Difference	Adjusted interval	Decision
All-condition CE minus FIXED	-0.2817	$[-0.3186,-0.2285]$	Not supported
Risk-condition CE minus RL-SCALAR	-0.0619	$[-0.1132,-0.0241]$	Not supported
Stationary CE minus RL-SCALAR	-0.0215	$[-0.0545,0.0013]$	NI not established
Risk-condition MDD minus RL-SCALAR	0.0151	$[0.0053,0.0286]$	Not supported
Risk-condition net return minus RL-SCALAR	-0.0145	$[-0.0280,-0.0048]$	NI not established
All-condition CE minus JOINT-DRE	0.0597	$[0.0187,0.1033]$	Supported

Appendix

Table S4.2 : V4 confirmatory paired contrasts with 99.1667% Bonferroni intervals. CE is bps/clock; MDD and net return are fractions. Positive CE/return and negative MDD favor RL-RISK.

Year	Method	Net (%)	DD (%)	SR	Median SR	CE	Day+ (%)
2024	Uniform	-7.17	8.80	-4.65	-4.50	-0.531	38.5
2024	Attributed	-3.92	7.18	-2.70	-3.28	-0.302	40.2
2024	Loss	-3.92	7.18	-2.70	-3.28	-0.302	40.2
2024	Joint	-5.05	7.85	-3.22	-3.37	-0.382	39.3
2024	PCGrad	-4.17	6.98	-2.63	-2.69	-0.321	37.2
2024	Passive	1.90	5.42	0.87	0.87	0.089	49.2

Appendix

Table S5.1 : Historical public bar configuration, May–June in each year. Values are averages of separately funded account metrics: first across three seeds per asset, then equally across BTC/ETH; passive has two accounts, not six artificial seed replications. DD is hourly-sampled maximum drawdown; SR is descriptive annualized hourly Sharpe; median SR is the account median; CE is hourly certainty equivalent in bps; Day+ is the percentage of positive calendar days. Uncertainty for the central paired return contrast appears in Table S5.2 .

Window	Difference	95% interval
2024	$+5.54$	$[+1.43,+11.58]$
2025	$+2.27$	$[-5.27,+10.69]$
Equal-year pooled	$+3.90$	$[-0.90,+9.96]$

Appendix

Table S5.2 : Historical attributed minus uniform mean daily return. Positive favors attributed allocation. Intervals resample paired initialization seeds and common seven-day blocks; they are descriptive and conditional on the selected windows. Units are bps/day, not execution bps saved.

Figure S5.1 : Mean normalized equity of separately funded historical research accounts, retaining the full calendar window. Curves connect measured daily closes and the initial capital point; they are not a rebalanced portfolio or a confidence band. Attributed and loss-budget curves coincide exactly. Negative learned-policy outcomes are retained alongside the passive comparator.

Figure S5.2 : Complete historical fee/slippage sensitivity: paired attributed-minus-uniform mean daily return with descriptive 95% intervals. The base point is reused from the main matrix. Funding is unchanged; policy trajectories are retrained under each declared cost multiplier. Connecting lines do not estimate untested multipliers.

H	Required test	Observed evidence	Conclusion	Limitation
H1	Matched full-system median net Sharpe and CE	Trained controlled losses and exploratory historical account comparisons; attributed and loss-budget controls coincide	Full H1 untested; no attribution-specific gain established	Small learners and two historical windows, not full AMATS
H2	Market-transition penalty and recovery	Controlled intervention diagnostics retain 229 censored records across methods	Market H2 untested	Simulator change times are not observed market transitions
H3	Intended IC/shortfall gain and noninferiority elsewhere	All three shifted normalized-loss criteria fail; stationary meets the margin but combined loss worsens	Analogue not supported; full H3 untested	Own-policy order targets differ; cost differences are not paired savings
H4	Matched removal of each active architectural component	Complete six-factor learning grid and cost multipliers; minimum share 0.5 reproduces uniform	Partial tests completed; full H4 untested	Named stack, six heads, LLM, SAC and TCN not evaluated

Appendix

Table S6.1 : Original hypothesis status and the narrower evidence actually obtained. “Untested” refers to the original full-system claim, not to the absence of the reported public experiments.

Original model/stack label	AUC@ $h$	MASE	Sharpe	MDD
N-HiTS	0.61	0.89	1.9	0.18
PatchTST	0.64	0.84	2.6	0.15
ETSformer	0.63	0.85	2.4	0.16
Stacked	0.66	0.81	3.1	0.13

Appendix

Table S7.1: Original draft Table 1, retained with its explicitly illustrative status. The horizon

h

, sample, costs and comparison protocol were not specified sufficiently to reproduce this table. These cells are not new trained comparisons and are not E01 metrics.

Original item	Status in this paper
Walk-forward and regime-stratified evaluation	Explicit controlled study schedules and limited Bar windows are available. The full historical stack’s sample/split protocol is unverified.
Out-of-sample AUC/MASE and calibrated forecasts	Public evaluation dimensions and illustrative source table retained; no recoverable matched named-ensemble result.
Synthetic attribution correctness	Present deterministic event/gradient checks and trained equality controls are documented in Appendix S1 and the study appendices; they do not date or verify the source’s earlier checklist.
Execution bps versus a taker baseline	Original qualitative claim retained; historical comparator and cost protocol unknown. Own-policy Bar differences are not paired execution savings.
Token/latency enforcement and alerts	Original budget, kill-switch and 80% Prometheus alert statements retained in Section 4.2 ; no deployment measurement or enforcement log recovered.

Appendix

Table S7.2: Original validation and operating checklist, restated by evidence status. Design requirements are not counted as completed empirical checks.

Figure S7.1 : E02, 8-hour BTC-target horizon: preserved mutual-information and correlation matrices. Numerical cells and ordering come from CSVs; labels alone are rounded. Duplicate funding-spread rows are retained. Sample and estimator lineage are unverified, so the plot is not an out-of-sample predictive-performance test.

Figure S7.2 : E02, 24-hour BTC-target horizon, using the same respective color limits as Figure S7.1 . All 13 rows and eight lags are retained, including duplicate funding-spread rows. These are archived diagnostics, not independent alpha confirmations.

Figure S7.3 : E03: unchanged archived trades visualization. Source run linkage is unresolved; the displayed dates and markers alone do not connect it to E01 or identify the complete system configuration.

Figure S7.4 : E04: unchanged archived short-window equity/trades example. No numerical performance estimate is reconstructed from the image and no E01 metrics are transferred to it. Its configuration and full-stack linkage remain unverified.

Model	Initial configuration and rationale
N-HiTS	Three identity stacks; one block per stack; two 256-unit ReLU layers per block; pooling kernels $(4,2,1)$ ; interpolation downsampling $(4,2,1)$ ; linear interpolation. Multirate processing supplies a coarser-scale comparator.
PatchTST	Patch length 16, stride 8, no end padding; three encoder layers; width 128; eight attention heads; feedforward width 256; dropout 0.1. Patching supplies local temporal structure with 41 input patches.
ETSformer	Two encoder and two decoder layers; width 128; eight attention heads; feedforward width 256; dropout 0.1; three selected Fourier components. Its level/growth/seasonal structure supplies a distinct modeling assumption.

Appendix

Table S9.1: Proposed starting architectures. These settings were not used to generate any reported AMATS result.

Field	Type and meaning
schema_version	String fixed to amats-intent-v0.1 .
decision_id , as_of_utc	Identifier and UTC timestamp copied from the input.
heads	Array containing exactly the six named heads, each once.
head_id , status	Fixed head name; status is active or abstain.
direction , confidence	Direction $\in\{-1,0,1\}$ ; finite raw confidence in $[0,1]$ .
horizon_hours	$1$ , $4$ , $24$ , or null when abstaining.

Appendix

Table S10.1: Proposed response contract. The complete JSON Schema rejects additional fields.

Credit Assignment in RL

Credit assignment in reinforcement learning studies how to trace rewards and failures back to the actions, states, or agents that caused them.

See all 219 papers

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
Aug 28, 2026Pengcheng Li, Zhengyang Zhang, Dongxu Zhang +2Fine-Grained Credit Assignment RL for LLM Agents
Tsinghua University · Xi’an Jiaotong University · Jiaxing Nanhu University
From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
Apr 10, 2026Chenchen ZhangCredit Assignment in RL RL for Language Models
Independent Researcher
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
Jun 30, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +6Agentic RL Step-Level Credit Assignment in RL
1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University +1

Aug 28, 2026J/K move · Enter open · S save

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

Pengcheng Li, Zhengyang Zhang, Dongxu Zhang, Sui Huang+1

Tsinghua University · Xi’an Jiaotong University · Jiaxing Nanhu University

Fine-Grained Credit Assignment RL for LLM Agents Step-Level Credit Assignment Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Fine-grained credit assignment is a central challenge in reinforcement learning for long horizon LLM agents. Standard objectives often train from programmatically verifiable terminal rewards by broadcasting each sparse outcome to every action in a trajectory. Existing methods typically seek finer credit from the rollout side, constructing auxiliary trajectory signals or additional comparisons to estimate action importance. Although useful, these approaches still treat the verifier that judged success as a scalar reward, discarding its internal task structure. Our key insight is that many verifiable tasks already encode the relevant checks inside their terminal verifier. We propose VICT (VerifierInstrumented Credit Tracing), a training-time interface that exposes executable or evidence backed atoms and traces them back to actions through dependency-valid proof edges. VICT redistributes group-relative advantage only along those edges, shifting credit assignment from rollout-side inference to verifierside tracing. It preserves the original terminal reward, abstains when evidence is incomplete or ambiguous, and changes only the training-time advantage tensor, requiring no learned critic, process labels, branch rollouts, or inference-time verifier access. On ALFWorld and WebShop, VICT improves substantially over outcome-only training and achieves strong performance alongside recent fine-grained credit methods; ablations rule out dense atom rewards, final-commit credit, temporal proximity, and sparsity as sufficient explanations.

Abstract

Credit Assignment in RL

Explore similar work

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning