DuSE-Bench: Benchmarking Joint Reasoning over Evolving Dialogue and Documents

Guanjie Cheng; Jianfeng Gao; Yufei Tong; Xinkui Zhao; Shuiguang Deng

cs.CLOct 31, 2026

DuSE-Bench: Benchmarking Joint Reasoning over Evolving Dialogue and Documents

Authors: Guanjie Cheng, Jianfeng Gao, Yufei Tong, Xinkui Zhao, Shuiguang Deng

Organizations: Ningbo Global Innovation Center, Zhejiang University · School of Software Technology, Zhejiang University · College of Computer Science and Technology, Zhejiang University

Abstract

Persistent Web information assistants must combine evolving document facts with user requirements established and revised across sessions. Yet systematic evaluation of how changes in both sources affect response requirements and evidence attribution remains limited. To address this gap, we introduce Dual-Source Evolution Benchmark (DuSE-Bench), a benchmark comprising 1,000 tasks across five scenarios, including 700 whose required premises originate in both dialogue and documents. An evolving Support Graph guides task construction and preserves links from updated facts and reused conclusions to original turns and passages. Task annotations distinguish grounding source from state dependence and enable separate evaluation of response content, response type, and source attribution. Experiments across 11 baseline configurations reveal persistent challenges in joint reasoning over evolving dialogue and documents, with content accuracy on jointly grounded tasks reaching at most 56.7%. Moreover, improvements in content accuracy can coincide with poorer response choice, while content-correct responses frequently lack complete attribution to original evidence. Matched evidence substitutions further show that the effect of replacing one source's evidence depends on support from the other. These findings motivate coordinating dialogue memory and document retrieval to apply current user requirements and document facts together while preserving their original support. The data are available at https://github.com/JianfengGao7/DuSE-Bench.

Figures & tables

Benchmark	Data characteristics			Capabilities
Benchmark	Turns	Tokens (K)	Domains	Cross-session dialogue	Evolving documents	Joint reasoning	Response-type evaluation	Source attribution
LoCoMo	21.6	–	Daily life	✓	✗	✗	✓	✗
LongMemEval	10.4	–	Personal	✓	✗	✗	✓	✗
VersionQA	–	$\sim$ 13.4	Software	✗	✓	✗	✗	✗
TIDE	–	$\sim$ 3.1	Law	✗	✓	✗	✗	✗
mtRAG	15.3	$\sim$ 0.3	Multiple	✗	✗	✓	✓	✗

Table 1. Comparison of DuSE-Bench with related benchmarks. Turns are mean message counts per original session. Token counts (K) are per document or released source unit, reported as approximate means (

\sim

) or ranges. ✓ and ✗ indicate whether a benchmark meets the capability criteria; – denotes not applicable. Definitions and sources are provided in Appendix A.3 .

Figure 1. Joint reasoning over evolving dialogue and documents in a software-release task. Restoring either source’s earlier state changes the handoff action while the other remains fixed. A software-release request combines dialogue conditions with versioned document facts. The selected Node 20.19.0 entry classifies the error as Deprecated, and the current work phase is release handoff. The routing rule requires a tracked follow-up for retirement review. Restoring the earlier work phase changes the follow-up form with the document classification fixed. Restoring the earlier document classification changes the review destination with dialogue conditions fixed.

Figure 2. Construction workflow of DuSE-Bench. The evolving Support Graph guides session and task design. It preserves links to original evidence through updates and reuse to support task annotation and validation. Five panels connect original document collections, the Document Graph, a workflow plan, dialogue and task construction, and task annotation and validation. Document preparation preserves source text and links related versions, reports, or assessment stages in their documented order. The Document Graph grounds facts in original passages and records changes across linked documents. The plan assigns work goals, document introductions, and task coverage across sessions. Reviewed session designs guide dialogue generation, using earlier dialogue and the accumulated Support Graph. The Support Graph traces task dependencies to original turns and passages, retaining support as conditions change. The final panel summarizes task annotation and human review and shows the benchmark's documents, dialogue, annotated tasks, and Support Graph, with counts of dialogue turns and tokens across documents and dialogue.

Dimension	Categories
Grounding source	Dialogue, Document, Joint
State dependence	Static, Dialogue, Document, Dual
Integration structure	Direct, Chained
Response type	Answer, Clarification request, Conflict report, Abstention

Table 2. Task dimensions and response types in DuSE-Bench.

Scenario	Domain	Documents	Passages	Sessions	Turns	Tasks	Tokens (K)
1	News	38	211	17	810	200	153
2	Science	30	442	18	700	200	327
3	Software	22	781	20	690	200	467
4	Corporate	22	1,998	20	816	200	1,255
5	Environment	14	3,943	28	808	200	3,081
Total	Five domains	126	7,375	103	3,824	1,000	5,283

Table 3. DuSE-Bench statistics by scenario. Turns count individual messages. Token totals (K) cover document and dialogue text.

ID	Setting	Grounding source			State dependence (Joint)				Overall
ID	Setting	Dlg.	Doc.	Joint	Static	Dlg.	Doc.	Dual	Acc.	M-F1	F1
Long context
A1	LC-4K	59.3	47.3	54.6	52.7	64.0	50.0	55.0	54.20	39.16	26.04
A2	LC-32K	56.0	56.7	55.7	55.0	56.0	50.0	59.5	55.90	39.50	23.61
Retrieval
A3	Dlg-RAG	66.7	42.7	50.3	50.3	50.0	48.0	51.5	51.60	36.45	31.69

Table 4. Baseline performance on DuSE-Bench (%). Grouped columns report content accuracy by grounding source and state dependence (700 jointly grounded tasks). Overall columns report content accuracy (Acc.), response-type Macro-F1 (M-F1), and source attribution F1 across all 1,000 tasks. Dlg./Doc. denote dialogue/document. Bold and underlining indicate the best and second-best scores per column.

Figure 3. Cumulative mean document coverage and retrieval gaps on 700 jointly grounded tasks (140 per scenario). White and black curves show Top-10 coverage under Joint-RAG (A5) and document-only retrieval, respectively. Teal, gold, and gray denote joint coverage, loss to source competition, and the document-ranking gap. Coverage is the fraction of default-annotated document passages represented by retrieved chunks and is averaged over tasks encountered so far in request order. Five panels show cumulative mean document coverage on 700 jointly grounded tasks, with 140 per scenario. The horizontal axis counts jointly grounded tasks in request order. Coverage is the fraction of default-annotated document passages represented by retrieved chunks. The white boundary shows joint Top-10 coverage; the black boundary shows document-only Top-10 coverage. Teal is the covered fraction under joint retrieval, gold is the additional coverage available under document-only retrieval, and light gray is the fraction still uncovered by document-only retrieval. All values are averaged over tasks encountered so far, and the three areas sum to 100 percent.

Setting	Task transitions	$\Delta$ Coverage (pp)		$\Delta$ Acc. (pp)
Setting	Task transitions	Document	Dialogue	$\Delta$ Acc. (pp)
B1	$29/2/0$	$+4.23$	$+7.73$	$+2.29$
B2	$111/0/-3$	$+13.07$	$-8.79$	$-0.57$
B3	$22/20/-5$	$+4.74$	$+3.98$	$+1.71$

Table 5. Retrieval interventions (B1–B3) relative to Joint-RAG (A5) on 700 jointly grounded tasks.

\Delta

columns report changes in evidence coverage and content accuracy (Acc.) in percentage points (pp). Task transitions list zero-to-positive document-coverage recoveries for source-competition/document-ranking cases followed by positive-to-zero losses shown as negative counts. Coverage and case definitions are provided in Appendix B.3 .

Figure 4. Paired performance changes from Joint-RAG (A5) after gold-evidence substitution on all 1,000 tasks, in percentage points (pp). C1 replaces dialogue, C2 documents, and C3 both with annotated original evidence. Bars show pointwise 95% bootstrap intervals from session resampling within scenarios. Three panels show changes in content accuracy, response-type Macro-F1, and source attribution F1 relative to Joint-RAG (A5). In each panel, C1 substitutes annotated original dialogue turns, C2 original document passages, and C3 both. Points show paired changes in percentage points, with pointwise 95 percent bootstrap intervals obtained by resampling triggering sessions within each scenario.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Grounding source	State dependence				Total
Grounding source	Static	Dialogue	Document	Dual	Total
Dialogue	50	100	–	–	150
Document	50	–	100	–	150
Joint	300	100	100	200	700
Total	400	200	200	200	1,000

Appendix

Table 6. DuSE-Bench task counts by grounding source and state dependence, pooled across five scenarios.

Score	Quality criterion
5	Complete, accurate fulfillment.
4	Correct core results and essential qualifications, with minor omissions or peripheral inaccuracies.
3	Substantial correct content with an essential component missing or wrong.
2	Some concrete correct content, with most essential content missing or wrong, or an incorrect central conclusion undermining the result.
1	Marginally useful correct information or recognition of the topic.
0	No relevant correct content.

Appendix

Table 7. Scoring rubric for response content quality (0–5).

Setting	Model	Grounding source			State dependence (Joint)				Overall
Setting	Model	Dlg.	Doc.	Joint	Static	Dlg.	Doc.	Dual	Acc.	M-F1	F1
Joint-RAG	GPT-4.1-mini	64.0	58.7	52.0	53.7	56.0	43.0	52.0	54.80	36.37	33.69
	GPT-5.1	73.3	69.3	65.9	69.0	63.0	61.0	65.0	67.50	45.54	34.39
	Claude Sonnet 4.5	68.7	58.0	55.1	57.7	55.0	52.0	53.0	57.60	53.09	33.81
	Qwen2.5-7B-Instruct	45.3	42.7	33.9	36.3	32.0	26.0	35.0	36.90	38.00	19.10
RH + Joint-RAG	GPT-4.1-mini	68.0	58.0	56.7	53.7	59.0	56.0	60.5	58.60	33.01	31.93

Appendix

Table 8. LLM performance on DuSE-Bench (%). Grouped columns report content accuracy by grounding source and state dependence (700 jointly grounded tasks). Overall columns report content accuracy (Acc.), response-type Macro-F1 (M-F1), and source attribution F1 across all 1,000 tasks. Dlg./Doc. denote dialogue/document. Bold and underlining indicate the best and second-best scores per column.

ID	Content		M-F1	F1
ID	Acc.	Qual.	M-F1	F1
A5	54.80	70.70	36.37	33.69
Retrieval interventions
B1	57.10	72.72	34.39	35.06
B2	53.40	69.70	33.12	33.62
B3	56.70	72.12	38.29	36.83

Appendix

Table 9. Response performance under retrieval interventions ( B1–B3 ) and gold-evidence substitutions ( C1–C3 ) (%). A5 is the Joint-RAG baseline. Columns report content accuracy (Acc.), content quality (Qual.), response-type Macro-F1 (M-F1), and source attribution F1 across all 1,000 tasks. Bold and underlining indicate the best and second-best scores per column within each group.

Evidence-Grounded QA

See all 56 papers

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning
Jun 3, 2026Qiyang Xie, Jialun Wu, Xinjie He +4Conversational Memory Retrieval-Augmented Generation
Northeastern University · Johns Hopkins University · Columbia University +1
iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics
Mar 4, 2026Preetam Prabhu Srikar Dammu, Arnav Palkhiwala, Tanya Roosta +1AI Agent Benchmarks LLM Agent Evaluation
University of Washington Seattle, Washington, USA · UC Berkeley Berkeley, California, USA
MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records
Oct 20, 2026Seohyun Lee, Jongha Kim, Dohwan Ko +4Enterprise AI Agents Question Answering
KAIST, Daejeon, Republic of Korea · Korea University, Seoul, Republic of Korea

Jun 3, 2026J/K move · Enter open · S save

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

Qiyang Xie, Jialun Wu, Xinjie He, Su Liu+3

Northeastern University · Johns Hopkins University · Columbia University+1

Conversational Memory Retrieval-Augmented Generation Document QA RAG Evaluation Long-Term Conversational Memory

Open paper AskPDF arXiv

AI systems increasingly need to combine two demanding capabilities: navigating multi-session conversation history and performing deep reading comprehension within long documents. Yet no existing benchmark evaluates both simultaneously. We introduce MemoryDocDataSet, a synthetic benchmark of 50 micro-worlds and 1,000 QA pairs in which each instance comprises 3-5 personas, a temporal event graph spanning months of activity, 3-5 real long documents (20,000-50,000 tokens each sourced from the Caselaw Access Project), multi-session conversations grounded on those documents, and 20 question-answer pairs across five reasoning categories. The defining feature is the Hybrid source tag: questions requiring a system to first navigate conversation history to identify which document is relevant, then extract the answer from within that document. Hybrid questions account for 75.1% of the dataset. Dataset quality is characterised through a prompt-sensitivity self-consistency analysis using LLM-as-judge, yielding a median Cohen's

κ= 0.634

across all 50 micro-worlds. We evaluate six baseline configurations spanning truncated context, long-context LLMs, retrieval-augmented generation (RAG), and memory systems. The best baseline (RAG-Both) achieves 0.358 overall F1 and 0.342 on Hybrid. Document-only retrieval (RAG-Doc) collapses to 0.267 on Hybrid despite achieving 0.453 on Doc-only questions, demonstrating a clear joint-retrieval gap that motivates architectures unifying conversational memory with long-document navigation. We release the dataset, generation pipeline, and all baseline implementations.

Abstract

Evidence-Grounded QA

Explore similar work

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics

MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records