Standard LLM benchmarks evaluate the assistant turn: the model generates a response to an input, a verifier scores correctness, and the analysis ends. This paradigm leaves unmeasured whether the LLM encodes any awareness of what follows the assistant response. We propose user-turn generation as a probe of this gap: given a conversation context of user query and assistant response, we let a model generate under the user role. If the model's weights encode interaction awareness, the generated user turn will be a grounded follow-up that reacts to the preceding context. Through experiments across 11 open-weight LLMs (Qwen3.5, gpt-oss, GLM) and 5 datasets (math reasoning, instruction following, conversation), we show that interaction awareness is decoupled from task accuracy. In particular, within the Qwen3.5 family, GSM8K accuracy scales from 41% (0.8B) to 96.8% (397B-A17B), yet genuine follow-up rates under deterministic generation remain near zero. In contrast, higher temperature sampling reveals interaction awareness is latent with follow up rates reaching 22%. Controlled perturbations validate that the proposed probe measures a real property of the model, and collaboration-oriented post-training on Qwen3.5-2B demonstrates an increase in follow-up rates. Our results show that user-turn generation captures a dimension of LLM behavior, interaction awareness, that is unexplored and invisible with current assistant-only benchmarks.
Figures & tables
Figure 1: User-turn generation as a probe of interaction awareness. Left: A conversation context consisting of system, user, and assistant turns delimited by special tokens (e.g., <|im_start|> , <|im_end|> ). Standard evaluation scores the assistant response for task accuracy. Our probe appends a <|im_start|>user header and lets model Mθ generate under the user role. A genuine follow-up indicates the model’s weights encode conversational awareness. Right: Genuine follow-up rate (%) vs. sampling temperature for six models across three datasets. At greedy decoding ( T=0 ), most models produce near-zero follow-ups. Increasing temperature surfaces latent interaction awareness in Qwen3.5 and GLM (e.g., Qwen3.5-27B rises from 0% to 22% on GSM8K), while gpt-oss remains near zero. Qwen3.5-9B starts with higher baseline follow-up on GPQA Diamond ( 13.1% ) and rises further with temperature.
Figure 2: Controlled perturbation examples. Left: Truncation removes the end of the assistant response, prompting the model to produce a reaction to complete the response. Right: Appending a generic question elicits a grounded critique rather than a prompt restatement. Assistant and user turns in the examples are generated by GLM-4.7. These controls demonstrate that interaction awareness can surface in specific contexts.
Figure 3: Cross-family dissociation: task accuracy vs. followup rate. Top row: task accuracy (%). Bottom row: followup rate (%). Five representative models on GSM8K, GPQA Diamond, and IFBench. Task accuracy does not predict follow-up quality: gpt-oss models produce the highest follow-up on GPQA despite lower accuracy than Qwen3.5 or GLM-4.7 . Full results including IFEval and GPQA Main in Table 5 .
Figure 4: Qwen3.5 family: interaction awareness across temperature. Genuine follow-up rate (%) vs. sampling temperature for all eight Qwen3.5 models on three datasets. At T=0 (greedy), follow-up is near zero for most models despite high task accuracy (Table 3 ). Higher temperatures surface latent interaction awareness, but this does not scale monotonically with model size: Qwen3.5-9B and Qwen3.5-27B often match or exceed larger models. MoE variants ( 35B-A3B , 122B-A10B ) consistently lag their dense counterparts.
Figure 5: Same question, same correct answer, different interaction awareness. Both models correctly answer a GPQA chemistry question (Answer: D). Left: Qwen3.5-9B generates a user turn that critically engages with the assistant’s reasoning about the Corey-Chaykovsky reagent and reaction conditions. Right: Qwen3.5-27B , a 3× larger model, restates the original prompt verbatim, an inability to generate realistic user turn after an assistant response.
GSM8K
GPQA Dia.
Model
Full
Trunc.
Full
Trunc.
Qwen3.5-27B
0.0
0.0
1.0
1.0
Qwen3.5-397B
0.8
4.9
2.0
2.0
gpt-oss-20b
1.2
5.8
1.0
18.7
gpt-oss-120b
0.0
24.2
20.7
65.7
GLM-4.7
1.0
55.0
2.0
39.4
Table 1: Controlled perturbations confirm the metric. Genuine-followup rate (%) under two perturbations. Left: Truncating the assistant response increases follow-up for non-Qwen models, confirming sensitivity to assistant-turn content. Right: Appending a generic question (e.g., “What do you think?”) to the assistant response. Base : unperturbed follow-up rate; +Question : follow-up rate after appending the question; % Changed : fraction of user turns whose text differs from the unperturbed generation, measuring whether the model attends to the perturbation at all. gpt-oss models respond most strongly to both perturbations; Qwen models are largely insensitive. Additional results in Appendix A .
GSM8K
Genuine Follow-up (%)
Held-out
Model
Acc.
IFBench
GPQA-Diamond
HealthBench
Coval
Base
62.9
1.0
2.0
36.7
19.4
SFT
40.3
48.0
46.0
54.4
45.2
RL
67.4
10.0
9.1
46.5
29.0
Table 2: Post-training on Qwen3.5-2B . GSM8K accuracy for reference. Followup rates on IFBench, GPQA Diamond and held-out conversational datasets (HealthBench, Coval).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
GSM8K Acc.
GSM8K
IFEval
IFBench
GPQA Main
GPQA Dia.
Qwen3.5-0.8B
41.6
1.0
0.0
3.0
0.0
0.5
Qwen3.5-2B
62.2
0.0
0.6
0.7
0.0
0.0
Qwen3.5-4B
86.5
0.0
1.1
1.0
0.0
0.5
Qwen3.5-9B
89.5
0.5
5.4
2.0
14.7
13.1
Qwen3.5-27B
95.8
0.0
1.1
1.0
1.8
1.5
Qwen3.5-35B-A3B
95.8
0.0
0.7
0.3
0.0
0.0
Appendix
Table 3: Qwen3.5 scaling: GSM8K answer accuracy (%) and genuine-followup rate (%) across all five benchmark datasets.
GSM8K
GPQA Diamond
IFBench
Model
T=0
0.3
0.7
1.0
T=0
0.3
0.7
1.0
T=0
0.3
0.7
1.0
Qwen3.5-0.8B
1.0
2.7
6.6
13.7
0.5
24.2
32.3
16.2
3.0
3.3
11.3
27.7
Qwen3.5-2B
0.0
0.0
1.7
4.7
0.0
1.5
0.0
7.6
0.7
2.0
13.3
21.7
Qwen3.5-4B
0.0
0.1
5.1
9.0
0.5
0.0
4.5
8.6
1.0
1.0
18.0
25.0
Qwen3.5-9B
0.5
0.5
3.2
7.7
13.1
15.2
16.2
19.7
2.0
3.3
10.0
36.0
Qwen3.5-27B
0.0
0.1
4.5
22.0
1.5
7.1
17.2
35.9
1.0
1.3
11.7
30.7
Appendix
Table 4: Full Qwen3.5 temperature sweep: genuine-followup rate (%) across all eight models and four temperatures on three datasets.
Model
GSM8K Acc.
GSM8K
IFEval
IFBench
GPQA Main
GPQA Dia.
Qwen3.5-9B
89.5
0.5
5.4
2.0
14.7
13.1
Qwen3.5-27B
95.8
0.0
1.1
1.0
1.8
1.5
Qwen3.5-397B-A17B
96.8
0.8
10.5
9.7
1.3
0.5
gpt-oss-20b
93.9
1.4
3.3
0.0
14.1
20.7
gpt-oss-120b
95.8
0.0
3.7
3.3
17.4
16.7
GLM-4.7
95.3
1.2
3.0
1.3
4.2
1.5
Appendix
Table 5: Cross-family genuine-followup rate (%) on the self-generated benchmark. Includes Qwen3.5-9B as a within-family outlier. GSM8K accuracy shown for reference.
Primary Label
Qwen3.5 (%)
gpt-oss (%)
GLM-4.7 (%)
previous_turn_restate
78.5
11.9
20.2
assistant_turn_restate
0.0
70.2
23.0
meta_planning
0.4
6.8
43.0
degenerate_short
16.6
5.1
0.1
malformed_artifact
0.8
1.0
10.8
new_task_prompt
1.4
0.8
0.9
Appendix
Table 6: Failure mode distribution across model families on the self-generated benchmark (all five datasets pooled). Qwen3.5 is dominated by prompt restatement, gpt-oss by assistant-turn restatement (copying the assistant response under the user role), and GLM-4.7 by meta-planning (continuing to plan as the assistant despite the user-role header). Each family’s dominant failure mode accounts for 43 – 78% of all generations, indicating that interaction awareness failures are systematic and family-specific rather than random.
Slice / Judge Pair
N
Gen. Agr.
Gen. κ
Label Agr.
Label κ
Full panel / nano vs mini
7815
81.7
0.480
48.0
0.396
Random / nano vs mini
1965
82.5
0.477
48.6
0.406
Balanced / nano vs mini
965
72.8
0.373
47.0
0.379
Balanced / nano vs gpt-5.4
965
73.2
0.366
44.7
0.358
Balanced / mini vs gpt-5.4
965
90.9
0.722
77.3
0.721
Appendix
Table 7: Same-family GPT-5.4 judge validation. Agreement rates are percentages; κ values are unitless. The full-panel row reruns the entire real-follow-up benchmark with gpt-5.4-mini ; the random slice is prevalence-preserving, while the balanced slice is a hard-case agreement slice.
Figure 6: Manual review interface. Screenshot of the annotation tool used for LLM judge validation. The interface presents the conversation context and the generated user turn to annotate, along with the primary-label inventory and a binary genuine-followup decision. Model identity, dataset metadata, and judge labels are hidden from annotators.
Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a multi-turn benchmark that evaluates clarification as policy behavior rather than isolated question quality. RegretBench provides a hidden-intent formulation of ambiguity, supports free-form interaction grounded in semantic-state tracking, and introduces a regret-based objective that measures how much value a model loses relative to a reference clarification policy. Experiments on open-domain QA and product recommendation scenarios show that final success alone is insufficient, as models with similar accuracy can differ substantially in efficiency, robustness to user behaviors, and stopping decisions. By jointly measuring intent resolution, interaction cost, ineffective clarification, and regret, RegretBench reveals whether models clarify usefully and efficiently. Our results show that effective clarification requires more than plausible questions: models must ask the right question at the right time and stop once the user's intended meaning is clear.
Minh Ngoc Ta, My Anh Tran Nguyen, Duong D. Nguyen +2
MBZUAI · BKAI Research Center, Hanoi University of Science and Technology · INSAIT, Sofia University "St. Kliment Ohridski"
Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.
Large language models (LLMs) define a distribution over text, which can be viewed as a probabilistic representation of uncertainty: sampling K responses yields a belief state - responses a model deems plausible. Existing work exploits this representation for narrow tasks like either decoding or selective prediction, and often requires manual interventions, not controlling generation directly. We propose Belief-Augmented Generation (BAG): grounding LLMs in their own belief state via the prompt and letting them reason over these K samples to decide on and execute a conversational strategy: clarify, abstain, or answer. In a multi-turn ambiguous question answering (QA) setting, we find that LLMs by default rarely clarify or abstain, ignoring uncertainty about the input (aleatoric) or facts (epistemic). BAG improves QA accuracy across six models and yields strategy decisions more faithful to their belief state than prompt-only baselines. Disentangling when to clarify from when to abstain, however, remains challenging.
Joris Baan, Wilker Aziz, Barbara Plank +1
University of Amsterdam · LMU Munich · European Commission, Joint Research Centre (JRC), Brussels