Fixed prompt-based language models face challenges in open-domain conversations, where versatile topics require different expert advice or demonstrations to maximize their universal performance. Motivated by the dynamic prompt studies in language generation, we extract different chatting strategies from varied forms of text corpus, which steer the language model to different styles based on human principles. We design and deploy a conversational companion called SCorPion, which uses GraphRAG to extract and summarize the knowledge graph globally, yield community-summarized strategies, and produce query-focused optimal strategies. SCorPion has strategic thinking, yields personified and stylized responses, and can be controlled by explicit factors. Code and further results can be found in https://bachozean.github.io/scorpion-page/.
Figures & tables
Figure 1: The paradigm of SCorPion. SCorPion retrieves and ranks the chatting strategy from different textual corpus by GraphRAG, and produces an anthropomorphic response. The ranker can be controlled by outer factors to produce different response styles.
Figure 2: The overall framework of SCorPion. The pipeline starts from the strategy extraction from three types of corpus, then conducts the indexing and summarization stages of GraphRAG, and finally inference the response.
Method
ESConv
DailyDialog
EmpatheticDialogues
MIC
GYAFC
Shakespeare
B-2
R-L
B-2
R-L
B-2
R-L
B-2
R-L
B-2
R-L
B-2
R-L
LLaMA3-70B-Instruct
3.06
10.16
3.50
11.20
2.60
8.90
3.90
11.06
13.00
32.44
12.45
31.44
+ Self-Refine
2.97
10.12
3.40
10.83
2.81
9.74
-
-
-
-
-
-
+ CoT
1.85
9.67
2.09
7.62
2.60
9.06
-
-
-
-
-
-
+ BOT
3.44
10.95
2.78
9.67
2.10
8.18
-
-
-
-
-
-
+ SCorPion (ours)
4.32
13.00
3.79
12.01
3.74
13.06
6.22
15.13
15.24
36.14
15.06
37.43
Table 1: Results of BLEU-2 (B-2) and ROUGE-L (R-L) on ESC and style adaptation tasks.
Chatting Domain
Direct
CSIM
SCorPion (ours)
AvgLen
Rounds
AvgLen
Rounds
AvgLen
Rounds
Overall
32.39
2.87
43.75
3.79
44.61
4.50
- topic transition
25.24
1.39
50.06
2.31
50.94
4.05
- proactively asking questions
35.24
2.87
42.71
3.35
48.75
4.04
- empathy
36.17
3.49
58.47
2.66
53.59
3.64
- concept guidance
46.41
3.53
47.73
4.26
49.13
4.36
Table 2: Results on the Cskills Benchmark. Experiments are run on GPT 3.5.
Method
Interesting
Fluency
Emotion
Acceptance
Effectiveness
Sensitivity
Satisfaction
orignal dataset
3.10
3.51
3.61
3.40
3.10
3.50
3.30
Llama3-70B-Instruct
3.00
3.05
3.30
2.70
3.00
3.20
3.00
+ Self-Refine
3.00
3.40
3.35
2.90
3.10
3.00
3.05
+ CoT
3.10
3.25
3.30
2.80
3.00
3.10
3.03
+ BoT
3.60
3.50
3.80
3.40
3.40
3.80
3.55
+ SCorPion (ours)
3.90
3.55
3.75
3.40
3.50
3.90
3.57
Table 3: Averaged Human evaluations on ESConv, DailyDialog and EmpatheticDialogues.
Figure 3: Test results on S-Eval.
Case of MIC
Query
What’s the best part about your life?
w/ Care
I find joy in helping others and making a positive impact. What brings you joy and fulfillment in your life?
Query
What’s a good way to tell your parents you failed multiple classes this semester?
w/ Loyalty
Be honest and direct with your parents, explaining what happened and assuring them you’re committed to doing better.
Case of Shakespeare
Query
What kind of man are you?
Table 4: Cases of SCorPion on style adaptation tasks.
Figure 4: Emotional shifts by SCorPion on emotional intensity (left) and emotional contagion (right).
Figure 5: Controlled curves of SCorPion on style categories of ESC (left) and moral (right).
Ablated Methods
B-2
R-L
w/o conditon in CSS
3.64
12.34
w/o demonstration in CSS
3.05
11.81
w/o domain-focused summarization
3.71
11.46
w/ CA in GraphRAG
2.07
9.03
reversed ordered of CSS
3.97
13.00
SCorPion
4.32
13.00
Table 5: Ablation study on ESConv. CSS (community summarized strategy) is in contrast with the original CS.
Role
Factor
Response
user
-
thanks a lot random stranger, lol, no offense, it is just so random where we find support right lol i am thankful for all the advice. ( case of ESConv )
SCorPion
0.1
Oh, none taken, no offense at all, I totally get it.
0.5
Oh, none taken! I’m just glad I could help
1.0
Oh, no offense taken. I’m just glad I could help, even if it was just a little bit… you’re welcome, and I’m sending you lots of calm vibes
2.0
No offense taken, I’m just glad I could help. It’s funny how life works, and we find help in unexpected places, right?
10.0
Oh, no offense taken! I’m glad I could help, and I completely agree - support can come from the most unexpected places, and that’s what makes it so beautiful, right?
Table 6: Cases of SCorPion on style control tasks.
Figure 6: Sensitivity curves of B-2 (bar) and RT (line) on ESConv, with different k , ϵ , sizes of M and Mg .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Case 1
user
I’ve started learning how to cook.
assistant
Wow, congrats on learnin’ how to cook, dude! Now you can finally stop livin’ off ramen noodles and make some real food, What dish have you made so far?
user
I tried making pizza last week. It was a bit of a fail the first time, but it still tasted okay!
assistant
Pizza’s a tough one to nail on the first try, but hey, at least it didn’t taste like cardboard, right? You’re probably gonna try again with some tweaks, maybe change up the crust or toppings, and I’m sure it’ll be a freaking masterpiece next time!
Case 2
user
I’m going to have a job interview next week.
Appendix
Table 7: Open cases generated by SCorPion.
Case 1
user
I’ve started learning how to cook.
assistant
Holy crap, finally, you’re doing something with your life, lol! So, what kind of dishes are you gonna make, huh, you’re probably looking for some recipes or techniques to try, right? Don’t mess it up, just experiment and try new things, and for God’s sake, don’t come crying to me if your food tastes like trash!
user
I tried making pizza last week. It was a bit of a fail the first time, but it still tasted okay!
assistant
Damn, a pizza fail, that’s like a rite of passage or something! So, what went wrong, was it the crust, sauce, or toppings, or did you just screw up everything, haha? Anyway, at least it tasted okay, so you can try again and not completely suck at it, right?
user
I put too much sauce on it, so it ended up a little salty.
assistant
For crying out loud, too much sauce, that’s like a rookie mistake, dude! Alright, alright, so it’s salty, no big deal, just balance it out with something else next time, like, l don’t know, a freakin’ salad or something! Don’t sweat it, you’ll get the hang of it, or you’ll just end up with a lifetime supply of salty pizzas, haha!
Appendix
Table 8: Open cases generated by SCorPion.
Method
Turing Test
EqBench
EmoBench
SECEU
BotChat
ESC-Eval
Positive
Negative
Sentiment ↓
EQ
GPT-4
6.05
-
82.19
38.3 ± 2.3
16.3 ± 1.2
2.29
109.04
Llama3-70B-Instruct
13.60
1.96
82.04
43.3 ± 3.5
15.7 ± 1.2
2.16
111.52
+ SCorPion (ours)
13.47
2.88
82.67
48.0 ± 2.0
19.0 ± 1.0
2.10
112.53
Human
13.89
3
-
-
-
0
-
Appendix
Table 9: Emotional Intelligence Performances of Methods. Sentiment of SECEU measures the deviation from human result; therefore, smaller Sentiment means better result. The ‘Human’ row indicates results of human responses (in blue); For BotChat, the result is calculated from anonymous annotations; for ESC-Eval, 3 corresponds to full score.
Characters
Response
seeker
I am not happy with my friends. They were trying to let me down.
supporter
if i get you clearly, you mean your friend disappointed you. how?
Direct
It sounds like you’re feeling hurt and disappointed by your friends’ actions.
SCorPion
I can totally sense your frustration and disappointment, have you freaking talked to those so-called friends of yours about how their stupid actions made you feel, or do you need some suggestions on how to blast them and address this messed-up situation?!
seeker
There were an argument between two of my friend while trying to resolve the issue they all started targeting me.
supporter
i have a hard time with my friends too sometimes
Appendix
Table 10: Typical cases generated by SCorPion in ESC. Contents from the original dataset are italic , and results of SCorPion are bolded ..
Method
Fluence
Diversity
Empathic
Suggestion
Human
Tech
Overall
Llama3-70B-Instruct
3.00
3.00
3.00
2.99
1.96
3.00
2.00
+ SCorPion (ours)
3.00
2.99
3.00
2.93
2.88
3.00
1.99
Appendix
Table 11: Detailed results of SCorPion on ESC-Eval.
We introduce DiscoTrace, a method to identify the rhetorical strategies answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. The rich, community-sensitive answering behavior structurally revealed by DiscoTrace can guide the development of pragmatic LLM answerers that are more attuned to contextual information needs.
Neha Srikanth, Jordan Boyd-Graber, Rachel Rudinger
We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark
Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with the same information available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information (a single-turn view for the teacher and a multi-turn view for the student), transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC attains 92-100% of single-turn performance, yielding more efficient and helpful multi-turn conversations with single-turn capabilities largely intact.