Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs

Zhuowen Liang; Xiaotian Lin; Zhengxuan Zhang; Yuyu Luo; Haixun Wang; Nan Tang

cs.CLMar 31, 2026

Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs

Authors: Zhuowen Liang, Xiaotian Lin, Zhengxuan Zhang, Yuyu Luo, Haixun Wang, Nan Tang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA

Abstract

Large language models (LLMs) are widely applied to data analytics over documents, yet direct reasoning over long, noisy documents remains brittle and error-prone. Hence, we study document question answering (QA) that consolidates dispersed evidence into a structured output (e.g., a table, graph, or chunks) to support reliable, verifiable QA. We propose a two-pillar framework, LiteCoST, to achieve both high accuracy and low latency with small language models (SLMs). Pillar 1: Chain-of-Structured-Thought (CoST). We introduce a CoST template, a schema-aware instruction that guides a strong LLM to produce both a step-wise CoST trace and the corresponding structured output. The process induces a minimal structure, normalizes entities/units, aligns records, serializes the output, and verifies/refines it, yielding auditable supervision. Pillar 2: SLM fine-tuning. The compact models are trained on LLM-generated CoST data in two stages: Supervised Fine-Tuning for structural alignment, followed by Group Relative Policy Optimization (GRPO) incorporating triple rewards for answer/format quality and process consistency. By distilling structure-first behavior into SLMs, this approach achieves LLM-comparable quality on multi-domain long-document QA using 3B/7B SLMs, while delivering 2-4x lower latency than GPT-4o and DeepSeek-R1 (671B). The code is available at https://github.com/HKUSTDial/LiteCoST.

Figures & tables

Figure 1: Structured data makes QA more accurate and reliable. (a) From raw document $D$ , we extract a table $T_{1}$ for query $Q_{1}$ and a graph $G_{1}$ for query $Q_{2}$ . (b) LLMs often fail when reasoning directly over unstructured text ( $Q_{1},D$ ; $Q_{2},D$ ), but succeed with structured inputs ( $Q_{1},T_{1}$ ; $Q_{2},G_{1}$ ).

Figure 2: (a) Direct prompting LLMs often causes hallucinations and format errors. (b) (Question, Document, CoST Template) $\Rightarrow$ LLM $\Rightarrow$ (CoST Trace, SSO), yielding verifiable and auditable QA.

Figure 3: Overview of LiteCoST , containing two stages: (1) CoST: Structure-First Reasoning and Trace Generation through structure analysis, trace generation, quality verification, and iterative refinement; and (2) SLM Fine-Tuning: SFT → GRPO process, including SFT for structure/format/steps, followed by GRPO with dual signals for answer/format quality and process consistency.

Figure 4: The GRPO training pipeline based on dual-level reward.

Close-Sourced Models & Large Language Models
Model	Model	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Model	Size	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
LLaMA-3.1-8B-Instruct	8B	55.03	0.20	51.60	0.15	51.50	0.04	44.75	0.02	51.32	0.10
GPT-4o-mini	8B	84.42	0.70	80.40	0.67	77.38	0.40	65.35	0.18	78.08	0.51
Qwen2.5-14B-Instruct	14B	83.74	0.57	82.12	0.56	69.96	0.24	66.41	0.10	75.60	0.38
GPT-4o ( Abacha et al., 2025 )	200B	84.10	0.73	80.53	0.60	81.50	0.50	64.30	0.25	79.32	0.54

Table 1: Comparison of different models generating structured outputs for long-document QA on the Finance Subset of Loong. Green highlights the remarkable improvements over the base model.

Backbone	Method	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Backbone	Method	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
LLaMA-3.2-3B-Ins	ODIE ( Jiao et al., 2023 )	68.89	0.39	61.30	0.30	61.11	0.15	49.75	0.07	61.21	0.23
	IEpile ( Gui et al., 2024b )	62.90	0.37	65.10	0.30	63.12	0.14	50.95	0.02	61.90	0.22
	Struc-bench ( Tang et al., 2024b )	55.13	0.15	51.05	0.15	48.16	0.06	44.10	0.08	49.90	0.11
	StructRAG ( Li et al., 2025c )	39.50	0.01	39.70	0.02	31.08	0.00	35.95	0.00	36.04	0.01
	LiteCoST (Ours)	81.27	0.53	78.08	0.49	78.34	0.36	64.75	0.16	76.95	0.40

Table 2: Performance of the Finance subset of Loong compared with other state-of-the-art methods

Figure 7: Comparison of extraction time and accuracy across different scale models.

Model	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Model	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
LLaMA-Ours	81.27	0.53	78.08	0.49	78.34	0.36	$64.75$	0.16	76.95	0.40
w/o Process Reward	$79.10$	$0.48$	$77.95$	$0.47$	$76.03$	$0.30$	$63.98$	$0.13$	$75.52$	$0.37$
w/o Outcome Reward	$76.07$	$0.45$	$73.10$	$0.41$	$71.72$	$0.23$	68.22	$0.15$	$72.55$	$0.32$
Qwen-Ours	$83.97$	0.62	81.55	0.59	81.00	0.43	67.98	0.18	79.93	0.48
w/o Process Reward	87.93	$0.63$	$77.95$	$0.55$	$77.32$	$0.42$	$60.60$	$0.15$	$77.39$	$0.46$

Table 3: Effect of different reward designs in ablation study on the Finance subset of Loong.

Close-Sourced Models & Large Language Models
IE Model	Model	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
IE Model	Size	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
GPT-4o-mini	8B	46.55	0.10	28.05	0.00	48.68	0.13	42.56	0.11	41.94	0.09
Qwen2.5-14B-Instruct	14B	48.45	0.08	21.90	0.01	57.31	0.16	26.74	0.02	37.51	0.06
GPT-4o	200B	50.05	0.06	27.00	0.01	61.16	0.14	33.11	0.03	42.06	0.06
LLaMA-3.2-3B-Instruct (Base)	3B	41.00	0.08	25.10	0.01	31.74	0.02	27.29	0.01	30.67	0.03

Table 4: Performance Comparison on the Legal subset of Loong. Green highlights the remarkable improvements over the base model, while Red indicates relative drops.

Table 5: Performance comparison on LongBench benchmark

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The prompt for selecting the most optimal structure.

Figure 9: The prompt for dynamically constructing schema.

Figure 10: The prompt for extracting table step-by-step.

Figure 11: The prompt for verifying the data quality.

Figure 12: The prompt for refining low-quality data.

Figure 13: The prompt for verifying the similarity of generated output and ground truth.

Figure 14: The baseline prompt template for different structured extraction.

Backbone	Method	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Backbone	Method	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
Qwen2.5-14B-Ins	Zero-Shot	83.74	0.57	82.12	0.56	69.96	0.24	66.41	0.10	75.60	0.38
	CoT ( Wei et al., 2022 )	85.93	0.63	81.38	0.57	73.28	0.30	67.70	0.26	77.51	0.44
	Dspy ( Khattab et al., 2023 )	80.90	0.55	80.90	0.60	76.86	0.34	63.60	0.21	76.99	0.44
	LiteCoST (CoST)	88.03	0.72	80.45	0.56	76.72	0.37	68.35	0.25	79.01	0.48
GPT-4o	Zero-Shot	84.10	0.73	80.53	0.60	81.50	0.50	64.30	0.25	79.32	0.54

Appendix

Table 6: The performance of various LLMs under different prompting settings, showing Average Scores (AS, 0-100) and Perfecr Rate (PR, 0-1) on the Finance subset of Loong benchmark.

Figure 15: The prompt for evaluation using LLM Score.

Figure 16: Performance (LLM score) versus document-set size for (a) CoST with baseline methods (zero-shot and CoT) and (b) LiteCoST-tuned models with different base models.

Company	Accounts Receivable
AXIM Biotechnologies, Inc.	$23,642
CIRTRAN CORP.	None
Arax Holdings Corp.	$453,837
High Wire Networks, Inc.	$4,483,1

Appendix

Table 7: Extracted table for the given query.

Model	Context	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Model	Length	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
Qwen2-72B-Ins	128k	$54.17$	$0.36$	$42.38$	$0.20$	$36.71$	$0.04$	$47.76$	$0.18$	$43.29$	$0.15$
SD $\to$ Qwen2-72B-Ins	128k	$57.30$	$0.31$	$54.80$	$0.38$	$61.46$	$0.23$	$46.36$	$0.16$	55.70	0.25
GPT-4o-mini	128k	$59.46$	$0.49$	$51.90$	$0.27$	$34.55$	$0.04$	$64.28$	$0.39$	$49.25$	$0.24$
SD $\to$ GPT-4o-mini	128k	$63.23$	$0.44$	$53.04$	$0.38$	$59.63$	$0.21$	$55.98$	$0.26$	58.02	0.29
GPT-4o	128k	$73.95$	$0.62$	$50.50$	$0.28$	$44.29$	$0.09$	$57.05$	$0.28$	$53.47$	$0.26$

Appendix

Table 8: Structured data quality evaluation of LiteCoST via Chain-of-Structured-Thought (CoST) on the Loong benchmark, compared against popular LLMs. SD denotes structured data.

Figure 17: Case study on representative information extraction tasks for large language models, comparing the information extraction capabilities of three Llama-3.2-3B-Instruct variants.

Dataset	ID	Source	Avg len	Metric	Language	#data
Single-Document QA
NarrativeQA	1-1	Literature, Film	18,409	F1	English	200
Qasper	1-2	Science	3,619	F1	English	200
Multi-Document QA
HotpotQA	2-1	Wikipedia	9,151	F1	English	200
2WikiMultihopQA	2-2	Wikipedia	4,887	F1	English	200

Appendix

Table 9: An overview of the dataset statistics for a subset of LongBench. ‘Source’ denotes the origin of the context. ‘Avg len’ (average length) is computed using the number of words for the English (code) datasets and the number of characters for the Chinese datasets. ‘Accuracy (CLS)’ refers to classification accuracy, while ‘Accuracy (EM)’ refers to exact match accuracy.

Model	Model	Spotlight Locating		Comparison		Clustering		Chain of Reasoning		Overall
Model	Size	AS	PR	AS	PR	AS	PR	AS	PR	AS	PR
$\mathtt{All\ Set}$ (10K-250K)
LLaMA-3.2-3B-Instruct	3B	49.90	0.16	52.10	0.14	47.89	0.07	46.85	0.06	49.37	0.11
Qwen2-7B-Instruct	7B	63.10	0.36	67.85	0.37	60.83	0.18	52.25	0.09	62.10	0.26
LLaMA-3.1-8B-Instruct	8B	55.03	0.20	51.60	0.15	51.50	0.04	44.75	0.02	51.32	0.10
GPT-4o-mini	8B	84.42	0.70	80.40	0.67	77.38	0.40	65.35	0.18	78.08	0.51

Appendix

Table 10: Full results of Table 1 . AS represents Avg Scores (0~~100) and PR denotes Perfect Rate (0~~1). Bold indicates the best result within each setting, and underlined indicates the second best.

Question Answering

Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.

See all 260 papers

Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answering
Jun 9, 2026Xiangjun Zai, Xingyu Tan, Chen Chen +2Retrieval-Augmented Generation Graph Retrieval-Augmented Generation
University of New South Wales, Australia · CSIRO, Australia · University of Wollongong, Australia
DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding
Apr 14, 2026Hao Yan, Yuliang Liu, Xingchen Liu +5Document Understanding Long-Document VQA
Huazhong University of Science and Technology · Huawei Inc.
EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering
Aug 21, 2026Xuanyu Meng, Jiashuo Sun, Jash Rajesh Parekh +1Document QA Retrieval-Augmented Generation
University of Illinois Urbana-Champaign Urbana, Illinois, USA

Jun 9, 2026J/K move · Enter open · S save

Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answering

Xiangjun Zai, Xingyu Tan, Chen Chen, Xiaoyang Wang+1

University of New South Wales, Australia · CSIRO, Australia · University of Wollongong, Australia

Retrieval-Augmented Generation Graph Retrieval-Augmented Generation Agentic RAG Document QA Hypergraph-Based RAG Memory-Augmented Agents

Open paper AskPDF arXiv

Long-document question answering (QA) requires large language models (LLMs) to reason over evidence scattered across lengthy documents, where answers often depend on event order, section-level context, and cross-part evidence connections. Although retrieval-augmented generation (RAG) reduces the input context by retrieving relevant evidence, existing structured RAG methods still face three limitations: costly query-agnostic knowledge organization, insufficient use of original document structure, and no reuse of historical reasoning experience. To address these limitations, we propose DocTrace, a multi-agent RAG framework for long-document QA that supports query-triggered knowledge organization, document-structure-aware and experience-guided reasoning. DocTrace preserves document hierarchy with a lightweight document structural tree index, constructs agent-shared hypergraph-structured working memory on demand during reasoning, and stores successful reasoning plans in graph-structured experience memory for future reuse, enabling adaptive exploration across related long-document questions. Experiments on four long-document QA datasets show that DocTrace achieves the best performance on three datasets, surpassing the strongest baseline, ComoRAG, by up to 8.85% in F1 and 4.40% in EM, while reducing the overall computational cost by 53.32%

Abstract

Question Answering

Explore similar work

Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answering

DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding

EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering