Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models

Sarath Shekkizhar; Romain Cosentino; Adam Earle

cs.AIApr 2, 2026

Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models

Authors: Sarath Shekkizhar, Romain Cosentino, Adam Earle

Organizations: Salesforce AI Research

Abstract

Standard LLM benchmarks evaluate the assistant turn: the model generates a response to an input, a verifier scores correctness, and the analysis ends. This paradigm leaves unmeasured whether the LLM encodes any awareness of what follows the assistant response. We propose user-turn generation as a probe of this gap: given a conversation context of user query and assistant response, we let a model generate under the user role. If the model's weights encode interaction awareness, the generated user turn will be a grounded follow-up that reacts to the preceding context. Through experiments across $11$ open-weight LLMs (Qwen3.5, gpt-oss, GLM) and $5$ datasets (math reasoning, instruction following, conversation), we show that interaction awareness is decoupled from task accuracy. In particular, within the Qwen3.5 family, GSM8K accuracy scales from $41\%$ ( $0.8$ B) to $96.8\%$ ( $397$ B-A $17$ B), yet genuine follow-up rates under deterministic generation remain near zero. In contrast, higher temperature sampling reveals interaction awareness is latent with follow up rates reaching $22\%$ . Controlled perturbations validate that the proposed probe measures a real property of the model, and collaboration-oriented post-training on Qwen3.5-2B demonstrates an increase in follow-up rates. Our results show that user-turn generation captures a dimension of LLM behavior, interaction awareness, that is unexplored and invisible with current assistant-only benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: User-turn generation as a probe of interaction awareness. Left: A conversation context consisting of system, user, and assistant turns delimited by special tokens (e.g., <|im_start|> , <|im_end|> ). Standard evaluation scores the assistant response for task accuracy. Our probe appends a <|im_start|>user header and lets model $M_{\theta}$ generate under the user role. A genuine follow-up indicates the model’s weights encode conversational awareness. Right: Genuine follow-up rate (%) vs. sampling temperature for six models across three datasets. At greedy decoding ( $T{=}0$ ), most models produce near-zero follow-ups. Increasing temperature surfaces latent interaction awareness in Qwen3.5 and GLM (e.g., Qwen3.5-27B rises from $0\%$ to $22\%$ on GSM8K), while gpt-oss remains near zero. Qwen3.5-9B starts with higher baseline follow-up on GPQA Diamond ( $13.1\%$ ) and rises further with temperature.

Panel of Figure 2 — Figure 2: Controlled perturbation examples. Left: Truncation removes the end of the assistant response, prompting the model to produce a reaction to complete the response. Right: Appending a generic question elicits a grounded critique rather than a prompt restatement. Assistant and user turns in the examples are generated by GLM-4.7. These controls demonstrate that interaction awareness can surface in specific contexts.

Figure 3: Cross-family dissociation: task accuracy vs. followup rate. Top row: task accuracy (%). Bottom row: followup rate (%). Five representative models on GSM8K, GPQA Diamond, and IFBench. Task accuracy does not predict follow-up quality: gpt-oss models produce the highest follow-up on GPQA despite lower accuracy than Qwen3.5 or GLM-4.7 . Full results including IFEval and GPQA Main in Table 5 .

Figure 4: Qwen3.5 family: interaction awareness across temperature. Genuine follow-up rate (%) vs. sampling temperature for all eight Qwen3.5 models on three datasets. At

T{=}0

(greedy), follow-up is near zero for most models despite high task accuracy (Table 3 ). Higher temperatures surface latent interaction awareness, but this does not scale monotonically with model size: Qwen3.5-9B and Qwen3.5-27B often match or exceed larger models. MoE variants ( 35B-A3B , 122B-A10B ) consistently lag their dense counterparts.

Panel of Figure 5 — Figure 5: Same question, same correct answer, different interaction awareness. Both models correctly answer a GPQA chemistry question (Answer: D). Left: Qwen3.5-9B generates a user turn that critically engages with the assistant’s reasoning about the Corey-Chaykovsky reagent and reaction conditions. Right: Qwen3.5-27B , a $3\times$ larger model, restates the original prompt verbatim, an inability to generate realistic user turn after an assistant response.

	GSM8K		GPQA Dia.
Model	Full	Trunc.	Full	Trunc.
Qwen3.5-27B	0.0	0.0	1.0	1.0
Qwen3.5-397B	0.8	4.9	2.0	2.0
gpt-oss-20b	1.2	5.8	1.0	18.7
gpt-oss-120b	0.0	24.2	20.7	65.7
GLM-4.7	1.0	55.0	2.0	39.4

Table 1: Controlled perturbations confirm the metric. Genuine-followup rate (%) under two perturbations. Left: Truncating the assistant response increases follow-up for non-Qwen models, confirming sensitivity to assistant-turn content. Right: Appending a generic question (e.g., “What do you think?”) to the assistant response. Base : unperturbed follow-up rate; +Question : follow-up rate after appending the question; % Changed : fraction of user turns whose text differs from the unperturbed generation, measuring whether the model attends to the perturbation at all. gpt-oss models respond most strongly to both perturbations; Qwen models are largely insensitive. Additional results in Appendix A .

	GSM8K	Genuine Follow-up (%)		Held-out
Model	Acc.	IFBench	GPQA-Diamond	HealthBench	Coval
Base	62.9	1.0	2.0	36.7	19.4
SFT	40.3	48.0	46.0	54.4	45.2
RL	67.4	10.0	9.1	46.5	29.0

Table 2: Post-training on Qwen3.5-2B . GSM8K accuracy for reference. Followup rates on IFBench, GPQA Diamond and held-out conversational datasets (HealthBench, Coval).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	GSM8K Acc.	GSM8K	IFEval	IFBench	GPQA Main	GPQA Dia.
Qwen3.5-0.8B	41.6	1.0	0.0	3.0	0.0	0.5
Qwen3.5-2B	62.2	0.0	0.6	0.7	0.0	0.0
Qwen3.5-4B	86.5	0.0	1.1	1.0	0.0	0.5
Qwen3.5-9B	89.5	0.5	5.4	2.0	14.7	13.1
Qwen3.5-27B	95.8	0.0	1.1	1.0	1.8	1.5
Qwen3.5-35B-A3B	95.8	0.0	0.7	0.3	0.0	0.0

Appendix

Table 3: Qwen3.5 scaling: GSM8K answer accuracy (%) and genuine-followup rate (%) across all five benchmark datasets.

	GSM8K				GPQA Diamond				IFBench
Model	$T{=}0$	$0.3$	$0.7$	$1.0$	$T{=}0$	$0.3$	$0.7$	$1.0$	$T{=}0$	$0.3$	$0.7$	$1.0$
Qwen3.5-0.8B	1.0	2.7	6.6	13.7	0.5	24.2	32.3	16.2	3.0	3.3	11.3	27.7
Qwen3.5-2B	0.0	0.0	1.7	4.7	0.0	1.5	0.0	7.6	0.7	2.0	13.3	21.7
Qwen3.5-4B	0.0	0.1	5.1	9.0	0.5	0.0	4.5	8.6	1.0	1.0	18.0	25.0
Qwen3.5-9B	0.5	0.5	3.2	7.7	13.1	15.2	16.2	19.7	2.0	3.3	10.0	36.0
Qwen3.5-27B	0.0	0.1	4.5	22.0	1.5	7.1	17.2	35.9	1.0	1.3	11.7	30.7

Appendix

Table 4: Full Qwen3.5 temperature sweep: genuine-followup rate (%) across all eight models and four temperatures on three datasets.

Model	GSM8K Acc.	GSM8K	IFEval	IFBench	GPQA Main	GPQA Dia.
Qwen3.5-9B	89.5	0.5	5.4	2.0	14.7	13.1
Qwen3.5-27B	95.8	0.0	1.1	1.0	1.8	1.5
Qwen3.5-397B-A17B	96.8	0.8	10.5	9.7	1.3	0.5
gpt-oss-20b	93.9	1.4	3.3	0.0	14.1	20.7
gpt-oss-120b	95.8	0.0	3.7	3.3	17.4	16.7
GLM-4.7	95.3	1.2	3.0	1.3	4.2	1.5

Appendix

Table 5: Cross-family genuine-followup rate (%) on the self-generated benchmark. Includes Qwen3.5-9B as a within-family outlier. GSM8K accuracy shown for reference.

Primary Label	Qwen3.5 (%)	gpt-oss (%)	GLM-4.7 (%)
previous_turn_restate	78.5	11.9	20.2
assistant_turn_restate	0.0	70.2	23.0
meta_planning	0.4	6.8	43.0
degenerate_short	16.6	5.1	0.1
malformed_artifact	0.8	1.0	10.8
new_task_prompt	1.4	0.8	0.9

Appendix

Table 6: Failure mode distribution across model families on the self-generated benchmark (all five datasets pooled). Qwen3.5 is dominated by prompt restatement, gpt-oss by assistant-turn restatement (copying the assistant response under the user role), and GLM-4.7 by meta-planning (continuing to plan as the assistant despite the user-role header). Each family’s dominant failure mode accounts for

43

–

78\%

of all generations, indicating that interaction awareness failures are systematic and family-specific rather than random.

Slice / Judge Pair	$N$	Gen. Agr.	Gen. $\kappa$	Label Agr.	Label $\kappa$
Full panel / nano vs mini	7815	81.7	0.480	48.0	0.396
Random / nano vs mini	1965	82.5	0.477	48.6	0.406
Balanced / nano vs mini	965	72.8	0.373	47.0	0.379
Balanced / nano vs gpt-5.4	965	73.2	0.366	44.7	0.358
Balanced / mini vs gpt-5.4	965	90.9	0.722	77.3	0.721

Appendix

Table 7: Same-family GPT-5.4 judge validation. Agreement rates are percentages;

\kappa

values are unitless. The full-panel row reruns the entire real-follow-up benchmark with gpt-5.4-mini ; the random slice is prevalence-preserving, while the balanced slice is a hard-case agreement slice.

Figure 6: Manual review interface. Screenshot of the annotation tool used for LLM judge validation. The interface presents the conversation context and the generated user turn to annotate, along with the primary-label inventory and a binary genuine-followup decision. Model identity, dataset metadata, and judge labels are hidden from annotators.

Explore similar work

One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
Jul 23, 2026Minh Ngoc Ta, My Anh Tran Nguyen, Duong D. Nguyen +2LLM Evaluation Clarification Question Generation
MBZUAI · BKAI Research Center, Hanoi University of Science and Technology · INSAIT, Sofia University "St. Kliment Ohridski"
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Clarify, Abstain or Answer? Strategising in Conversation with Belief-Augmented Generation
May 25, 2026Joris Baan, Wilker Aziz, Barbara Plank +1Clarification Question Generation Question Answering
University of Amsterdam · LMU Munich · European Commission, Joint Research Centre (JRC), Brussels

Jul 23, 2026J/K move · Enter open · S save

One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies

Minh Ngoc Ta, My Anh Tran Nguyen, Duong D. Nguyen, Yuxia Wang+1

MBZUAI · BKAI Research Center, Hanoi University of Science and Technology · INSAIT, Sofia University "St. Kliment Ohridski"

LLM Evaluation Clarification Question Generation LLM Decision-Making Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a multi-turn benchmark that evaluates clarification as policy behavior rather than isolated question quality. RegretBench provides a hidden-intent formulation of ambiguity, supports free-form interaction grounded in semantic-state tracking, and introduces a regret-based objective that measures how much value a model loses relative to a reference clarification policy. Experiments on open-domain QA and product recommendation scenarios show that final success alone is insufficient, as models with similar accuracy can differ substantially in efficiency, robustness to user behaviors, and stopping decisions. By jointly measuring intent resolution, interaction cost, ineffective clarification, and regret, RegretBench reveals whether models clarify usefully and efficiently. Our results show that effective clarification requires more than plausible questions: models must ask the right question at the right time and stop once the user's intended meaning is clear.

Abstract

Explore similar work

One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Clarify, Abstain or Answer? Strategising in Conversation with Belief-Augmented Generation