The Price of Thought: Does Test-Time Reasoning Pay in LLM Trading?

Jiayi Chen; Guiling Wang

cs.AISep 25, 2026

The Price of Thought: Does Test-Time Reasoning Pay in LLM Trading?

Authors: Jiayi Chen, Guiling Wang

Organizations: Department of Computer Science New Jersey Institute of Technology Newark, New Jersey, USA

Abstract

While inference-time reasoning in large language models (LLMs) promises better decision making, its higher computational cost may not yield better economic outcomes. Yet reasoning controls are rarely evaluated as economic interventions, where changes in model outputs must translate into better portfolios after trading costs. We conduct a controlled study of representative LLMs from the DeepSeek, GPT, and Gemini families. We vary reasoning effort while holding information available at each formation date, prompts, output formats, and portfolio construction fixed. Our evaluation covers a full year of U.S. equities under three input conditions: numerical, identifiable news, and masked news. It includes more than 800,000 asset predictions and repeated model generations. Across all three model families, additional reasoning does not produce a reliable improvement in net portfolio returns. For DeepSeek, where we examine the full progression from no reasoning to maximum reasoning, performance is nonmonotonic. Repeated generations also produce unstable treatment effects and portfolio selections, even when overall scores remain similar. These findings show that additional reasoning can change financial decisions without reliably improving their economic value, motivating validation for each task before deployment.

Figures & tables

Figure 1. Controlled evaluation pipeline. Within each backbone, reasoning level varies while information available by each formation date, prompts, output schema (required fields, types, and ranges), candidate assets, and portfolio rules remain fixed. The information conditions are numerical inputs, identifiable news with identifying details retained, and masked news with identifying details removed. Repeated calls on frozen formation dates assess the stability of scores, portfolios, and treatment effects. An experimental pipeline sends numerical, identifiable news, or masked news inputs through fixed tasks, two reasoning levels, structured asset scoring, portfolio construction with long and short positions, returns after costs, and a paired estimator of treatment effects.

Backbone	Baseline reasoning	Additional levels	Information conditions	Formation dates	Audit generations
DeepSeek-Flash	none	low, high, maximum	numerical; identifiable news; masked news	241	3 per condition
GPT-5.6 Luna	none	low	numerical; identifiable news; masked news	241	3 per condition
Gemini 3.1 Flash-Lite	minimal	low	numerical; identifiable news; masked news	241	3 per condition

Table 1. Experimental design. Each primary study uses 241 formation dates, meaning trading days when new holdings are selected. Each audit generation is produced by a repeated call on the frozen audit formation dates.

Panel of Figure 2 — Figure 2. Primary low versus baseline return effects across 241 return dates. Circles denote numerical inputs, squares denote identifiable news, and diamonds denote masked news. Horizontal bars are two-sided 95% HAC CIs, which account for changing variance and serial dependence. The solid line is zero and the dashed line is the $+0.25$ bps/day economic threshold. No interval establishes a material benefit. A horizontal forest plot shows nine effects of low reasoning relative to baseline across numerical, identifiable news, and masked news inputs. All confidence intervals cross zero and the positive economic threshold.

Backbone	Information	Estimate	95% CI	$p$	Adjusted $p$	Decision
DeepSeek	numerical	$+6.159$	$[-7.243,19.560]$	.368	1.000	inconclusive
DeepSeek	identifiable news	$+0.378$	$[-9.033,9.790]$	.937	1.000	inconclusive
DeepSeek	masked news	$+1.449$	$[-6.768,9.666]$	.730	1.000	inconclusive
GPT	numerical	$+3.540$	$[-2.531,9.611]$	.253	.759	inconclusive
GPT	identifiable news	$-0.626$	$[-5.905,4.653]$	.816	1.000	inconclusive
GPT	masked news	$+0.040$	$[-4.297,4.378]$	.986	1.000	inconclusive

Table 2. Primary low versus baseline results at a one way trading cost of 10 bps. Adjusted

p

-values use the Holm method across the three information conditions separately within each backbone.

Figure 3. Reasoning levels and repeated call stability. (a) DeepSeek effects averaged across generations for portfolios formed on 48 audit formation dates, relative to none; cells are descriptive because intervals for cumulative levels were not computed. (b) Three unconnected generation markers per backbone; diamonds and bars show the average within each return date and the 95% HAC CI adjusted for serial dependence. Repeated calls are not independent market samples. Panel a is a signed matrix of DeepSeek effects at none, low, high, and maximum reasoning for numerical, identifiable news, and masked news inputs. Panel b plots three separate generation markers for masked news and an aggregate confidence interval for each backbone. DeepSeek is consistently negative, while GPT and Gemini each change sign.

Backbone	Tasks	Prompt tokens	Completion tokens	Reasoning tokens	Cost (USD)	Invalid tasks
DeepSeek-Flash	16,176	801,697,928	288,221,145	268,969,372	297.52	0.00%
GPT-5.6 Luna	8,088	360,029,538	11,992,166	371,323	41.98	0.99%
Gemini 3.1 Flash-Lite	8,088	507,532,598	17,670,460	7,928,611	75.56	2.35%
Total	32,352	NA	NA	NA	415.05	NA

Table 3. Compute, cost, and reliability for all evaluated tasks within each backbone. Token counts are reported by providers and should not be compared as identical units of cognition. Invalid primary tasks is the share of primary tasks still unresolved after the frozen validation and repair policy. NA denotes not aggregated; token columns are not aggregated across providers, and total cost is summed before row values are rounded.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Baseline	Added reasoning	Provider control	Output limit
DeepSeek	none	low, high, maximum	thinking ; reasoning_effort	65,536
GPT	none	low	reasoning.effort	45,000
Gemini	minimal	low	thinkingLevel	8,192

Appendix

Table 4. Provider controls defining the reasoning intervention. Prompt, inputs, schema, candidate order, and output limit remain fixed within each backbone.

Condition	Low–none	Maximum–low	Maximum–none
Numerical	$-0.529$ $[-16.954,15.896]$	$-4.346$ $[-8.979,0.286]$	$-4.875$ $[-20.319,10.569]$
Identifiable news	$-8.525$ $[-18.208,1.158]$	$+4.472$ $[-1.595,10.538]$	$-4.053$ $[-12.949,4.843]$
Masked news	$-9.016$ $[-16.248,-1.784]$	$+0.770$ $[-4.812,6.352]$	$-8.246$ $[-17.641,1.149]$

Appendix

Table 5. DeepSeek audit across three generations, bps/day. Return date remains the market sampling dimension.

Contrast	MBB-5 95% CI	Cohort 95% CI
DeepSeek numerical	$[-7.264,20.328]$	$[-2.958,15.670]$
DeepSeek identifiable news	$[-9.099,9.287]$	$[-7.140,7.849]$
DeepSeek masked news	$[-6.635,9.764]$	$[-5.783,8.715]$
GPT numerical	$[-1.832,10.338]$	$[-2.104,9.436]$
GPT identifiable news	$[-6.020,4.586]$	$[-6.269,5.015]$
GPT masked news	$[-4.093,4.667]$	$[-4.767,4.919]$

Appendix

Table 6. Paired bootstrap intervals at a one way cost of 10 bps. MBB-5 is the moving-block bootstrap with five sessions per block; cohort resamples complete formation contributions. Each uses 50,000 deterministic resamples.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 924 papers

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
Feb 13, 2026Wei-Lin Chen, Liqian Peng, Tian Tan +5LLM Inference Efficiency Inference-Time Scaling
University of Virginia · Google
Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
Mar 31, 2025Rui Wang, Hongru Wang, Boyang Xue +9LLM Inference Efficiency Cost-Aware Inference
University of Edinburgh, Edinburgh, UK · University of Macau, Macau SAR, China · University of Illinois Urbana-Champaign, USA
When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
Apr 12, 2026Shu Zhou, Rui Ling, Junan Chen +3Overthinking in Language Models Test-Time Scaling
Nanjing University · Baidu · Nanjing University of Finance & Economics

Feb 13, 2026J/K move · Enter open · S save

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

Wei-Lin Chen, Liqian Peng, Tian Tan, Chao Zhao+4

University of Virginia · Google

LLM Inference Efficiency Inference-Time Scaling Efficient Language Model Reasoning LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.

Abstract

LLM Reasoning

Explore similar work

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models

When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling