Online debates on platforms like Reddit or Kialo are structured as reply trees: each message can receive several replies, which in turn can each receive several replies, producing a branching structure rather than a linear conversation. Turning one path (or a mix of paths) through such a tree into a natural, turn-by-turn conversation among multiple speakers (a process we call linearization) requires deciding which messages to keep, in what order, and who each message is addressed to. To this aim, we present LLMberjack, a platform for creating multi-party conversations starting from existing debates, originally structured as reply trees. The system offers an interactive interface that visualizes discussion trees and enables users to construct coherent linearized dialogue sequences while preserving participant identity and discourse relations. It integrates optional large language model (LLM) assistance to support automatic editing of the messages and speakers' descriptions. We provide preliminary evidence about the platform's utility by showing how tree visualization facilitates the creation of coherent, meaningful conversation threads and how LLM support enhances output quality while reducing human effort. The tool is open-source and designed to promote transparent and reproducible workflows to create multi-party conversations, addressing a lack of resources of this type.
Figures & tables
Figure 1: Overview of the LLMberjack platform. The interface integrates reply-tree visualization, message selection tools for building linearized multi-party conversations (1), and LLM-support for editing messages and speaker profiles (2).
Figure 2: Screenshot of tree visualization for node 1.2.1 (left) and of the chat creation tab (right). Each node-box reports the speaker’s name on the top-right corner, and a preview of the message in the center (expandable).
Figure 3: Screenshot of the LLM-assisted message refinement page.
Nat.
Var.
Eng.
v turn
w Tree
65.62
34.37
49.99
1.82
w/o Tree
28.13
21.88
28.13
1.46
tie
6.25
43.75
21.88
/
κw
0.44
0.40
0.25
/
Table 1: Percentage of MPC comparisons where one setting (with or without tree visualization) was preferred over the others in terms of naturalness (Nat.), variability (Var.), and participants’ engagement (Eng.). The last column reports the average turn-selection speed in turns/minute ( vturn ). The final row shows inter-annotator agreement (weighted Cohen’s κw ).
Metric
w Tree
w/o Tree
Avg Number of Turns
12.17±1.43
12.42±1.47
Avg Addressees per Turn
1.53±0.72
1.57±0.73
Avg Interleaving score
0.22±0.21
0.37±0.26
Table 2: Structural metrics of the MPCs created with and without tree visualization (mean ± standard deviation). Number of turns per MPC and addressees per turn describe conversation size and engagement; the interleaving score describes how often turn order switches between first-level branches of the original reply tree, with higher values indicating more frequent switching.
Gen.
Len.
Style
Temp.
v tokens
w LLM
64.06
57.81
64.06
56.25
0.86
w/o LLM
17.19
4.69
25.00
31.25
0.47
tie
18.75
37.50
10.94
12.50
/
κw
0.36
0.58
0.43
0.44
/
Table 3: Percentage of times one setting (with or without LLM support) was preferred over the other in terms of general quality (Gen.), length (Len.), style (Style), and temperament (Temp.), together with the average refinement speed in tokens/second ( vtokens ). The final row reports inter-annotator agreement (weighted Cohen’s κw ).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Screenshot of the labelling interface. The left panel shows the conversation, where users can select a message and assign it the labels from the supported label types. The right panel reports the labels assigned at both the message level (for the selected message) and the conversation level.
Large language models (LLMs) are increasingly deployed for extended, multi-topic conversations, yet the flat, append-only structure of current conversation interfaces introduces a fundamental limitation: all context accumulates in a single unbounded window, causing topically distinct threads to bleed into one another and progressively degrade response quality. We term this failure mode logical context poisoning. In this paper, we introduce the Conversation Tree Architecture (CTA), a hierarchical framework that organizes LLM conversations as trees of discrete, context-isolated nodes. Each node maintains its own local context window; structured mechanisms govern how context flows between parent and child nodes, downstream on branch creation and upstream on branch deletion. We additionally introduce volatile nodes, transient branches whose local context must be selectively merged upward or permanently discarded before purging. We formalize the architecture's primitives, characterize the open design problems in context flow, relate our framework to prior work in LLM memory management, and describe a working prototype implementation. The CTA provides a principled foundation for structured conversational context management and extends naturally to multi-agent settings.
Online credit card discussions provide a natural setting for studying how consumers communicate about financial products. Simulating these discussions requires more than just generating individual comments, the generated threads should also match how real users express themselves and interact with others. We introduce CARD, a framework for generating realistic credit card discussion threads. Given a credit card post and its matched real thread, CARD uses non-verbatim guidance on reply structure, comment function, stance, tone, and conversational variation. A planner organizes these controls, a writer generates the discussion, and a calibration loop updates comments' populations that contribute to differences between the generated and real thread distributions. We evaluate CARD on real Reddit credit card discussions using lexical, semantic, behavioral, and structural metrics. CARD matches the distributions of real credit card discussions better than simulation baselines across multiple LLMs and also demonstrates smaller effect sizes and distribution distances across metrics. These results show that structured planning and targeted revision can generate the realism of simulated credit card discussions.
Yaoning Yu, Kai-Min Chang, Ye Yu +3
University of Illinois Urbana-Champaign United States · U.S. Bank United States · Stanford University United States +1
Multi-Agent Debate (MAD) improves the reasoning performance of Large Language Models (LLMs) through multi-round interaction. However, LLMs in MAD are highly susceptible to blind conformity. Existing individual evaluation methods, typically based on confidence or perplexity, fail to reflect the correctness of reasoning and may even exacerbate blind conformity. To address this, we shift the perspective from individual evaluation to group interaction. We define mutual referencing among LLMs as Debate Relationships and recognize that regulating these relationships is the key to mitigating blind conformity. In this paper, we propose a novel framework for Dynamically rEgulating debAte Relationships (DEAR) from the group perspective. At first, DEAR quantifies consensus and divergence as group evidence to capture the debate state. Then, DEAR operates through three stages: 1) What: perceiving group consultation tendency and uncertainty; 2) Who: introducing a Selection RL-Agent to dynamically select reference peers; and 3) How: adopting a Behavior RL-Agent to adaptively adjust generation behaviors. Notably, we formulate the execution of the two RL-Agents as a sequential decision-making process, jointly optimizing via multi-agent reinforcement learning. Extensive experiments demonstrate that DEAR achieves superior performance while significantly reducing token consumption.
Hao Wu, Shoucheng Song, Chang Yao +3
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China