Can LLMs Understand What They Predict? Evaluating Latent Causal Structure Inference in Financial Markets
Organizations: National Centre for Text Mining, The University of Manchester · The Fin AI · ELLIS Manchester
Abstract
Large language models (LLMs) are increasingly evaluated through their ability to predict future outcomes, implicitly assuming that predictive success reflects an understanding of the underlying system. However, accurate predictions can often be achieved without recovering the latent mechanisms that generate observable outcomes. To examine this assumption, we introduce LUCID, a benchmark that evaluates whether models can infer latent market structures from event descriptions and historical financial observations. Models generate event-induced causal graphs representing inferred market structure and are evaluated through a reconstruction-based framework. Our evaluation of 6 open-weight and proprietary LLMs shows that adding their generated graphs generally reduces reconstruction quality compared with using historical observations alone, although a few metrics improve. These findings suggest that inferring informative, event-specific market structures from observable evidence remains challenging for the evaluated LLMs.
Figures & tables
| Paradigm | Evaluation Target | Multimodal Evidence | Event- Conditioned | Explanation |
| LLMTIME ( Gruver et al., 2023 ) | Forecasting | ✗ | ✗ | ✗ |
| CiK ( Williams et al., 2024 ) | Forecasting | ✓ | ✗ | ✗ |
| ( Merrill et al., 2024 ) | Forecasting/Question-Answering | ✓ | ✗ | ✗ |
| FinBen ( Xie et al., 2024 ) | Forecasting/Decision-Making | ✓ | ✗ | ✗ |
| CLADDER ( Jin et al., 2023 ) | Local Causal Relationship | ✗ | ✗ | ✗ |
| LLMGC ( Liu et al., 2026 ) | Graph Recovery | ✗ | ✗ |
| Reconstruction Fidelity | Market Distribution Consistency | |||
|---|---|---|---|---|
| Model | Condition | (%) | (%) | (%) |
| Llama-3.3-70B | Real | |||
| Shuffle | ||||
| Mismatch | ||||
| Gemini-3.6-Flash | Real | |||
| Shuffle | ||||
| Reconstruction Fidelity | Market Distribution Consistency | ||
|---|---|---|---|
| Graph input | (%) | (%) | (%) |
| VAR | |||
| Node-shuffled | |||
| Event-mismatched | |||
| Model | Direct | Propagation |
|---|---|---|
| GPT-5 | 12.50 | 13.36 |
| Gemini-3.6-Flash | 0.05 | 5.04 |
| Llama-3.1-8B | 58.13 | 64.64 |
| Llama-3.3-70B | 42.71 | 34.42 |
| Qwen3-8B | 17.76 | 28.05 |
| Qwen3-32B | 7.07 | 7.32 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Year | Number of events | Percentage |
|---|---|---|
| 2018 | 38 | 4.1% |
| 2019 | 144 | 15.4% |
| 2020 | 132 | 14.1% |
| 2021 | 139 | 14.8% |
| 2022 | 107 | 11.4% |
| 2023 | 146 | 15.6% |
| Shared model output: Llama-3.1-8B; EVENT GLD; positive polarity; seven-day duration. | ||
| Reused explanation: “Investors seek safe-haven assets, increasing gold demand” (87 distinct events after text normalization). | ||
| Event and date | Expert source | Expert analysis and omitted context |
| A. Partial troop withdrawal 15 February 2022. Russia announces a partial withdrawal from the Ukrainian border; western officials report no immediate evidence of drawdown. | Erlam (2022) OANDA / MarketPulse 15 February 2022 | Attributes cooling gold demand to the withdrawal announcement’s de-escalation signal. The reused explanation addresses neither this easing signal nor uncertainty about the withdrawal’s credibility. |
| B. Market sell-off and liquidity intervention 12 March 2020. The Fed offers $1.5 trillion in liquidity as the Dow and FTSE suffer their worst day since 1987. | Reuters (2020) , quoting ANZ analyst Daniel Hynes 13 March 2020 World Gold Council (2020) 19 March 2020 | Discusses broad liquidation and gold sales to raise cash and meet margin calls during the sell-off. The reused explanation omits this competing source of selling pressure. |
| C. Depositor reassurance 21 March 2023. Janet Yellen pledges to protect depositors at smaller US banks amid withdrawals and banking-sector stress. | Morgan Stanley (2023) Wealth Management Global Investment Office 21 March 2023 | Links Yellen’s remarks and other support measures to recovering confidence. The reused explanation does not address how the reassurance intervention could alter crisis-related safe-haven demand. |
| Event and date | Model and edge | Generated explanation | Polarity |
|---|---|---|---|
| Jeremy Hunt refuses to rule out sanctions against China. 4 July 2019 | Llama-3.1-8B EVENT UUP | “ US dollar strengthens as investors seek safe-haven assets.” | negative ( ) |
| US Federal Reserve raises short-term interest rates again. 26 September 2018 | Llama-3.1-8B EVENT VIX | “Interest rate hike increases expected volatility ” | negative ( ) |
| US, Mexico and Canada reach a new trade deal. 1 October 2018 | Llama-3.1-8B EVENT VIX | “ Market volatility increases due to trade deal uncertainty” | negative ( ) |
| Russia is accused of a cyber-attack on the chemical weapons watchdog. 4 October 2018 | Llama-3.1-8B EVENT VIX | “Cyber-attack on chemical weapons watchdog increases uncertainty and volatility ” | negative ( ) |
| Fed chairman hints at the first interest rate cut in over a decade. 10 July 2019 | Qwen3-8B EVENT VIX | “Market uncertainty about rate cut timing increases volatility ” | negative ( ) |
| Brexit trade talks may continue beyond the MEPs’ deadline. 19 December 2020 | Llama-3.3-70B EVENT UUP | “The event may lead to a stronger U.S. dollar due to increased uncertainty and potential for safe-haven demand.” | negative ( ) |
| Reconstruction Fidelity | Market Distribution Consistency | ||||||
| Category | Model | Condition | |||||
| Proprietary | GPT-5 | TS-only | 0.166577 | 0.155001 | 0.002574 | 0.007270 | 0.008075 |
| TS+Graph | 0.173159 | 0.158475 | 0.002515 | 0.007045 | 0.007881 | ||
| (%) | |||||||
| Gemini-3.6-Flash | TS-only | 0.178335 | 0.165734 | 0.002541 | 0.007252 | 0.008034 | |
| TS+Graph | 0.179873 | 0.162173 | 0.002625 | 0.007724 | 0.008526 | ||
| Model | Access | Inference setting |
|---|---|---|
| GPT-5 | OpenAI Responses API | Reasoning effort: low |
| Gemini-3.6-Flash | Gemini API | Thinking level: low |
| Qwen3-8B-Instruct | Local vLLM | Temperature |
| Qwen3-32B-Instruct | Local vLLM | Temperature |
| Llama-3.1-8B-Instruct | Local vLLM | Temperature |
| Llama-3.3-70B-Instruct | Local vLLM | Temperature |
| Setting | Value |
|---|---|
| Evaluator | Qwen3.8-27B |
| Evaluation mode | Zero-shot |
| Thinking mode | Disabled |
| Decoding | Greedy decoding |
| Temperature | 0 |
| Maximum context length | 24,576 tokens |
| Top-3 mass (%) | Self-BLEU-4 | |||
| Model | D | P | D | P |
| Open-weight Models | ||||
| Llama-3.1-8B | 97.97 | 44.93 | 0.488 | 0.687 |
| Llama-3.3-70B | 76.24 | 24.60 | 0.512 | 0.667 |
| Qwen3-8B | 86.81 | 36.48 | 0.452 | 0.679 |
| Qwen3-32B | 45.98 | 18.85 | 0.489 | 0.574 |
| Pooled Exact Reuse | Macro Exact Reuse ( ) | |||
| Model | D | P | D | P |
| Open-weight Models | ||||
| Llama-3.1-8B | 58.13 | 64.64 | 30.70 | 31.68 |
| Llama-3.3-70B | 42.71 | 34.42 | 17.56 | 13.37 |
| Qwen3-8B | 17.76 | 28.05 | 6.84 | 9.10 |
| Qwen3-32B | 7.07 | 7.32 | 2.80 | 1.77 |