Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

Execution realism remains weakly standardized in research on large-language-model-based trading systems, limiting comparison and reproduction across backtests, simulations, and portfolio benchmarks. This article presents a PRISMA-ScR systematic scoping review with a nested execution-reproducibility evidence audit. Eight reproducible query families returned 796 query-level records; DOI/title deduplication left 687 unique records, 101 reports were sought for retrieval, and 59 full texts were recovered through direct and fallback open-access routes. Fifty-three reports are provisionally eligible for expanded evidence charting, subject to reconciliation of two blind author-validation packets. The charting framework separately evaluates point-in-time controls, temporal splits, held-out evaluation, trading costs and turnover, execution semantics, universe construction, architecture reporting, and artifact availability. Direct-trading, portfolio or alpha-construction, and benchmark studies are retained as distinct descriptive subgroups rather than pooled as a common performance estimand. Search responses, retrieval attempts, file hashes, evidence excerpts, screening decisions, and coding worksheets are archived with the review materials. Final field-level counts are intentionally withheld until author validation is complete.

LLM agents in financial markets may cite rules yet still submit orders that violate executable constraints or misread surveillance evidence. We introduce ReguSim, a controlled financial-compliance environment, and ReguBench, a target-marked monitoring benchmark, to separate four artifacts: stated reasoning, attempted action, execution enforcement, and monitor evidence. In trader runs with DeepSeek V4 Pro and Gemini 3.5 Flash, visible rules reduce but do not eliminate rejected actions, and incentive or persona framing shifts behavior. A bridge study shows that trader rationales can mislead an independent monitor unless enforcement evidence is shown. In monitoring, simple structured baselines either match or exceed prompt-only LLMs. The results frame financial compliance evaluation as an audit of rule-grounded actions and evidence use, rather than a single compliance score.

We study behavioral alignment and representation dynamics of large language model (LLM) agents in financial decision environments. TradeArena, an auditable trading-agent testbed with risk reports, execution simulation, memory, and replayable trajectories, lets us analyze how rationales, positions, and interventions evolve under market stress. Code and data artifacts are available through the TradeArena repository. We find pre-failure signatures: planning embeddings drift from normal centroids, fused plan-risk representations separate normal from pre-drawdown states, and local manifolds exhibit effective-rank contraction. Across 80 rolling failure anchors and eight LLM trajectories, this pattern persists across hash, LSA, Transformer, and white-box hidden-state probes. Stress tests with CoT-free target weights, lexical controls, OHLCV noise, and false audits show that rationale-level contraction can vanish without rationales, while intent-space and fused signatures remain informative. Structured risk feedback can act as an external alignment signal without fine-tuning, but not as a universal performance enhancer: true audit feedback improves calibration for some models, returns for others, and exposes cases where placebo or hidden feedback has higher short-horizon return but weaker alignment diagnostics. A 51-stock intraday experiment reveals a correlation blind spot: LLM rationales justify exposure to coupled assets that the risk layer clips. Finally, a financial-audit task suite shifts comparison from ``which model trades best'' to whether models can audit trajectories, respect execution boundaries, reproduce artifacts, and avoid claim overreach. These results support a research claim, not a profitability claim: auditable risk feedback and representation trajectories reveal when LLM financial reasoning is aligning, drifting, or failing.