Think about how human handles complex reading tasks: marking key points, inferring their relationships, and structuring information to guide understanding and responses. Likewise, can a large language model benefit from text structure to enhance text-processing performance? To explore it, in this work, we first introduce Structure of Thought (SoT), a prompting technique that explicitly guides models to construct intermediate text structures, consistently boosting performance across eight tasks and three model families. Building upon this insight, we present T2S-Bench, the first benchmark designed to evaluate and improve text-to-structure capabilities of models. T2S-Bench includes 1.8K samples across 6 scientific domains and 32 structural types, rigorously constructed to ensure accuracy, fairness, and quality. Evaluation on 45 mainstream models reveals substantial improvement potential: the average accuracy on the multi-hop reasoning task is only 52.1%, and even the most advanced model achieves 58.1% node accuracy in end-to-end extraction. Furthermore, on Qwen2.5-7B-Instruct, SoT alone yields an average +5.7% improvement across eight diverse text-processing tasks, and fine-tuning on T2S-Bench further increases this gain to +8.6%. These results highlight the value of explicit text structuring and the complementary contributions of SoT and T2S-Bench. Dataset and eval code have been released at https://t2s-bench.github.io/T2S-Bench-Page/.
Figures & tables
Figure 1: Performance of SoT and Importance of Text Structuring. We evaluated three models on eight distinct text-processing tasks using three prompting strategies: direct answering, Chain-of-Thought (CoT), and Structure of Thought (SoT). The horizontal axis shows the model’s performance with direct answering, while the vertical axis indicates the performance change relative to direct answering. Our evaluations follow standards from lm-eval and Longbench tasks. SoT consistently boosts performance across different tasks and models.
Figure 2: Construction Process of T2S-Bench, including Sample Collection, Muti-hop Reasoning and End-to-End Dataset Construction.
Domain
Specific Area
CS Algorithm / AI
(CS1) Model Architecture (CS2) Training / Inference Pipeline (CS3) RAG / Agent System Component Diagram
CS System
(CS4) End to end Pipeline (CS5) System Component Architecture (CS6) Serving / Inference System
Table 1: Major Science Domain Taxonomy (Compact). Abbreviations: CS Computer Science, LS Life Sciences, SS Social Sciences, ES Environmental Sciences, EM Economics & Management Sciences, PS Physical Sciences.
1. Fault Localization
Single
(FL2) First directly affected downstream node, (FL4) Key bottleneck / Dominator node, (FL6) Branch isolation: who will not be affected, (FL7) Fault amplification point in a feedback loop
Multiple
(FL1) Upstream root cause set, (FL3) Minimum cut set, (FL8) Multi fault explainability
Table 2: Task Taxonomy of T2SBench. Organized by Single choice , Multiple choice , and Mixed (Single+Multiple choice) .
Figure 3: (Left) Sample Distributions of different Dataset. (Right) Overview of T2S-Bench Sample Distributions.
Multi-choice QA Overall
Boundary Testing
Counterfactual Reasoning
Fault Localization
Functional Mapping
Structure Score
Model
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Node
Link
Gemini-2.5-Pro
81.40
91.56
80.00
91.78
90.65
93.46
74.51
90.62
89.86
91.01
58.09
84.32
Gemini-2.5-Flash
72.20
83.67
77.50
87.33
78.50
84.33
64.22
82.76
76.81
78.94
46.90
75.10
Gemini-2.0-flash
61.20
79.63
72.50
87.03
75.70
86.76
41.67
70.99
76.81
81.26
42.71
66.42
Gemini-2.0-flash-lite
53.40
72.41
65.00
81.08
79.44
83.74
29.90
63.35
62.32
66.57
39.22
69.18
GPT-5.2
71.80
84.32
77.50
90.47
84.11
89.69
62.75
82.24
69.57
71.45
50.57
77.76
Table 4: Overall performance on T2S-Bench. The leftmost two columns show overall performance (EM and F1 scores) on T2S-Bench-MR, multiple-choice dataset requiring multi-hop reasoning. The central columns represent accuracy for each question type within T2S-Bench-MR. The rightmost two columns show performance on the T2S-Bench-E2E dataset, separately evaluating node extraction (measured by average semantic similarity between predicted and reference nodes) and link extraction (measured by F1 score of correctly identified link pairs). The best-performing model within each model family on T2S-Bench-MR is highlighted with pink shading. Additionally, the top three performances in each metric are highlighted in red (first place), green (second place), and blue (third place).
T2S-Bench
LongBench
Scrolls
Model
MC (EM)
MC (F1)
HotpotQA
2WikiMQA
Qasper
GovReport
QMSum
ContractNLI
Quality
Summscreen
Qwen2.5-7B-Instruct
28.8
59.4
60.0
46.9
43.5
32.0
23.9
55.8
37.8
17.3
+ CoT
36.6
62.1
62.6
59.0
41.9
25.7
19.7
52.1
38.4
15.8
+ SoT
40.6
68.4
65.8
63.2
48.0
37.3
27.9
58.4
41.4
21.3
+ T2S-Train
46.1
73.5
68.2
65.3
51.2
41.2
30.9
60.3
42.8
24.5
LLaMA3.1-8B-Instruct
24.2
54.2
59.1
53.7
44.9
34.3
25.2
31.5
39.8
26.8
Table 5: Performance improvements after fine-tuning on the T2S-Train-1.2k. We fine-tuned Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct models for 100 epochs using GRPO on T2S-Train-1.2k (detailed training settings provided in Appendix F.1 ). Evaluations were conducted on various text-processing tasks within Longbench and SCROLLS using lm-eval.
Figure 4: F1 scores across different topics on T2S-Bench-MR. We selected one representative model from each model family; The first fig shows their average F1 scores across various domains. The remains fig illustrate individual model performances per domain, with the vertical axis indicating deviations from the average performance. The dark dashed rectangle represents the average performance (set to zero). Scores outside this rectangle indicate above-average performance, while scores inside indicate below-average performance..
Figure 5: Link F1 scores on MR-Bench-E2E across texts corresponding to reference graphs with varying node counts.
Figure 6: Correlation of model performance between T2S-Bench-MR and Longbench Pro Dataset.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Structure diagram examples across main domains
Overall
CS
Eco
Environ
Life
Phy
Social
Gemini-2.5-Pro
81.40
91.56
83.48
93.12
78.95
91.66
75.76
91.67
80.65
88.57
88.52
92.33
80.90
91.96
Gemini-2.5-Flash
72.20
83.67
73.91
81.18
65.79
82.04
71.21
85.53
65.59
79.09
81.97
88.81
76.40
88.14
Gemini-2.0-flash
61.20
79.63
61.74
77.07
61.84
80.95
60.61
80.81
59.14
79.32
67.21
76.12
58.43
83.66
Gemini-2.0-flash-lite
53.40
72.41
53.04
72.01
50.00
69.04
56.06
74.44
53.76
73.50
60.66
68.47
49.44
75.88
GPT-5.2
71.80
84.32
73.91
82.55
65.79
86.44
66.67
79.85
73.12
87.10
77.05
80.77
73.03
87.65
GPT-5.1
67.80
83.46
73.91
83.83
65.79
83.65
68.18
82.22
62.37
82.32
73.77
80.42
62.92
86.99
Appendix
Table 6: Performance on Multi choice QA. Shaded columns report EM, unshaded columns report F1.
Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85% memory / context savings with an 8.67% performance drop across mathematical tasks.
We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.
Sönke Tenckhoff, Mario Koddenbrock, Erik Rodner
KI-Werkstatt / FB2, University of Applied Sciences Berlin
Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from whether to structure' to `how to structure' for optimal reasoning preservation.
Vineet Kumar, Kanishka, Bhuvanesh Mandora
PayPal Artificial Intelligence PayPal, Bengaluru India