Signals, Evidence, and Context: User Scrutiny in Real-World LLM Conversations

Ahana Biswas; Yu-Ru Lin

cs.AINov 5, 2026

Signals, Evidence, and Context: User Scrutiny in Real-World LLM Conversations

Authors: Ahana Biswas, Yu-Ru Lin

Organizations: School of Computing and Information, University of Pittsburgh, Pittsburgh, Pennsylvania, USA

Abstract

Large language models (LLMs) often present synthesized answers without making their supporting evidence easy to inspect. We study how features of these answers relate to users' subsequent accepting, checking, and challenging in 489,283 assistant-user response pairs from ShareChat and WildChat. Broad attribution language appears in 31.9% of qualifying responses, compared with explicit URLs in 1.33%, revealing a gap between claims of grounding and preserved inspectable support. Attribution, uncertainty, and other response cues generally accompany both checking and challenging; only uncertainty shows a small statistically distinguishable pooled association favoring checking. Within Technical Help, uncertainty favors checking relative to challenging; refusals and reasoning traces favor challenging in Seeking Information. Scrutiny is concentrated in some conversations, while its apparent turn-to-turn persistence becomes much smaller when conversation-level differences and response ordering are examined separately. These results suggest that supporting verification requires attention to what evidence can be inspected, which question a user is trying to resolve, and the history of the interaction.

Figures & tables

Corpus	Pairs	Convs.	Mean	Qualifying	Scrutiny
ShareChat	311,591	44,847	6.9	250,523	30.3%
WildChat	177,692	53,160	3.3	118,121	25.2%
Total	489,283	98,007	5.0	368,644	28.7%

Table 1. Behavioral analysis sample. Mean length counts observed assistant–user response pairs per conversation. Scrutiny is the percentage of qualifying pairs labeled checking or challenging. Platform-level means and medians appear in Supplementary Section A.

Figure 1. RQ1: Response features across samples and platforms. (A) Detected feature prevalence among qualifying response pairs ( $N=368{,}644$ ). The observed attribution-to-URL ratio is approximately 24:1; an illustrative sensitivity calculation accounting for attribution-detector precision yields approximately 10:1–20:1. (B) Platform-specific prevalence among all joined response pairs, with percentages shown in each cell. The denominators differ between panels. The measures describe features preserved in the released records and do not assess whether individual claims are adequately supported.

Figure 2. RQ2: Pooled associations between response features and subsequent user behavior. Relative risk ratios for checking (blue) and challenging (orange), each compared with accepting, with conversation-clustered 95% confidence intervals. All models use the common control set. NE indicates a coefficient that could not be stably estimated under the fine model-version specification. The right column reports FDR-adjusted

q

values from covariance-aware direct checking-versus-challenging Wald tests. The direct contrast cannot be determined by visually comparing the separate confidence intervals.

Figure 3. RQ3: Context-specific coefficients and direct contrasts. (A) RRRs for checking and challenging versus accepting (108 planned tests). Columns: Other, Practical, Seeking, Self-Expression, Technical, Writing, Multimedia, ShareChat, WildChat. Bold blue: BH-FDR significant; gray: nonsignificant; dash: non-estimable;

u

: unstable. (B) Direct check/challenge ratios (95% CIs); below one favors challenging, above one favors checking. Five filled points survive the conservative 48-test BH correction. Within-stratum significance does not test differences between strata.

Figure 4. RQ4: Concentration of scrutiny and adjacent-turn behavior. (A) Percentage of scrutiny-heavy conversations by opening topic among conversations with at least five qualifying exchanges. Parentheses show eligible conversation counts; the dashed 10% line represents the top-decile benchmark by construction. Topic differences are unadjusted. (B) Row percentages for strictly adjacent next-turn outcomes, retaining accepting, checking, challenging, and not-qualifying labels.

Figure 5. RQ4: Pooled and within-conversation sequential associations. Lag estimates are shown as percentage-point associations between scrutiny at successive qualifying exchanges. Conversation fixed effects separate within-conversation associations from stable differences across conversations. Longer-panel restrictions and the split-panel jackknife assess sensitivity to dynamic-panel bias. The raw and adjusted ICCs are 0.207 and 0.184, respectively.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Labeled pairs	Conversations	Mean pairs/conv.	Qualifying	Qualifying (%)	Scrutiny (%)
ShareChat	311,591	44,847	6.9	250,523	80.4	30.3
WildChat	177,692	53,160	3.3	118,121	66.5	25.2
Total	489,283	98,007	5.0	368,644	75.3	28.7

Appendix

Table 2. Construction of the response-pair sample. Qualifying and scrutiny percentages use corpus-specific labeled pairs and qualifying pairs, respectively. Means divide labeled pairs by conversations with an observed pair; they do not represent total messages per conversation.

Corpus	Platform	Mean	Median
ShareChat	ChatGPT	7.5	3
ShareChat	Gemini	11.2	3
ShareChat	Grok	6.2	3
ShareChat	Perplexity	1.6	1
WildChat	ChatGPT interactions	3.3	2

Appendix

Table 3. Observed assistant–user response pairs per conversation by platform, in the analyzed paired data. The platform means are rounded to one decimal and medians to whole pairs, as reported in the defense summary. These numbers are not derived from the full ShareChat release counts.

Feature	Check/accept	Challenge/accept	Check/challenge	$q$
URL	NE	1.80 [1.63, 2.00]	NE	–
Markdown link	1.65 [1.24, 2.20]	1.29 [1.09, 1.53]	1.28 [0.95, 1.73]	0.209
Inspectable evidence	NE	1.71 [1.55, 1.88]	NE	–
Broad attribution	1.18 [1.04, 1.35]	1.07 [1.04, 1.10]	1.11 [0.97, 1.26]	0.209
Evidence signifier	1.27 [1.11, 1.44]	1.22 [1.06, 1.39]	1.04 [0.88, 1.23]	0.787
Uncertainty	1.10 [1.07, 1.14]	1.04 [1.01, 1.07]	1.06 [1.02, 1.10]	0.018

Appendix

Table 4. Pooled multinomial models on the qualifying sample (moved from the main paper). Entries are relative risk ratios (RRRs) with 95% confidence intervals. The

q

column reports BH-FDR correction across the nine stable direct checking-versus-challenging contrasts.

Feature	Detected qualifying pairs
URL	4,889
Inspectable evidence	5,372
Broad attribution	117,715
Markdown link	1,133
Refusal	2,159
Uncertainty	159,429

Appendix

Table 5. Observed qualifying-pair counts for selected feature-positive indicators used to interpret sparse direct contrasts. Other counts are not reported in the supplied summary exports.

Feature	Checking RRR [95% CI]	Challenging RRR [95% CI]
URL	1.73 [1.51, 1.99]	1.86 [1.68, 2.07]
Inspectable evidence	1.66 [1.47, 1.87]	1.77 [1.60, 1.95]

Appendix

Table 6. Sensitivity to coarser platform fixed effects for sparse link-based indicators.

Figure 6. Pairwise $\phi$ between the 11 retained response features. URL and inspectable evidence are strongly nested.

Previous turn	Accept	Check	Challenge	Not qualifying
Accept	64.8	8.2	10.1	16.9
Check	43.8	21.9	17.4	16.9
Challenge	39.9	10.4	34.6	15.0
Not qualifying	39.9	6.5	9.4	44.2

Appendix

Table 7. Strictly adjacent turn transitions, row percentages; all four outcomes retained.

Transition	Observed excess	Shuffled excess	Extra ordered repetition
Check $\to$ check	13.64	10.8	$\approx 2.8$
Challenge $\to$ challenge	24.56	18.3	$\approx 6.3$

Appendix

Table 8. Same-type repetition: excess over transition from accepting, with within-conversation shuffle benchmarks (percentage points).

Role	Feature	Operational meaning	Interpretive caveat
Material verification support	URL	Explicit retrievable URL in the assistant response.	A link can be present without supporting the focal claim.
Material verification support	Markdown link	Link encoded in markdown structure.	A formatting signal; not a quality judgment.
Material verification support	Inspectable evidence	Composite indicator for response-level link or preserved source material that can potentially be opened or inspected.	Shared transcripts may omit platform-native panels.
Verification signifier	Attribution language	Language invoking studies, research, reports, experts, or other external authority.	Attribution can appear without retrievable support.
Verification signifier	Evidence signifier	Wording that presents a claim as evidenced or documented.	Signals evidentiary status without necessarily exposing evidence.
Epistemic communication cue	Uncertainty	Language marking uncertainty, qualification, or limited confidence.	Surface expression does not establish calibration.

Appendix

Table 9. Detailed operationalization of Study 1 response features.

Outcome	Audited $N$	Precision
Accepting	200	0.81
Checking	200	0.89
Challenging	200	0.92
Not qualifying	50	0.92

Appendix

Table 10. Outcome-label human audit,

N=650

assistant–user pairs.

Level	Category	Precision
Intent	Asking	0.80
	Doing	0.85
	Expressing	0.81
Coarse topic	Writing	0.82
	Practical Guidance	0.83
	Seeking Information	0.91

Appendix

Table 11. Human precision of topic and intent controls in a 600-message first-turn audit.

Detector	Precision
Broad attribution language	0.67
Narrow citation-like attribution	0.70
Evidence signifier	0.68
Uncertainty cue	0.72
Confidence cue	0.85
Disclaimer	0.82

Appendix

Table 12. Manual positive-case audits of frozen response-feature lexicons (approximately 20 flagged cases per detector).

Family	Scope	Tests	Reported in
Component coefficients	Nine strata $\times$ six features $\times$ two outcomes; BH-FDR	108	Main Figure 4A; Section G
All planned direct	Nine strata $\times$ six features, including Multimedia	54	Supp. G
Primary direct	Stable non-Multimedia contrasts; BH-FDR	32	Main Figure 4B; Section G
Conservative direct	Non-Multimedia: 32 stable, two unstable, 14 NE ( $p=1$ ); BH-FDR	48	Main Figure 4B; Section G

Appendix

Table 13. RQ3 multiplicity families. Component coefficients compare each outcome with acceptance; direct contrasts compare checking with challenging.

Type	Stratum	Feature	Check RRR [95% CI]	Challenge RRR [95% CI]	Check/Challenge [95% CI]	FDR $q$ (32 stable)
Topic	Multimedia	URL	NE	NE	NE	–
Topic	Multimedia	Inspectable evidence	NE	NE	NE	–
Topic	Multimedia	Uncertainty	NE	NE	NE	–
Topic	Multimedia	Reasoning trace	NE	NE	NE	–
Topic	Multimedia	Disclaimer	NE	NE	NE	–
Topic	Multimedia	Refusal	NE	NE	NE	–

Appendix

Table 14. All S1-RQ3 stratified checking-versus-challenging contrasts. NE denotes a non-estimable or unstable specification. The

q

column reports the estimable-only correction; conservative

q

values for the significant contrasts appear in the text and Figure 4 of the main manuscript.

Stratum	Feature	Contrast	RRR	95% CI	AME (pp)	$q_{108}$	Status
Writing	URL	Check / Accept	–	–	–	1.000	singular
Writing	URL	Challenge / Accept	–	–	–	1.000	singular
Writing	Inspectable evidence	Check / Accept	–	–	–	1.000	singular
Writing	Inspectable evidence	Challenge / Accept	–	–	–	1.000	singular
Writing	Uncertainty	Check / Accept	1.18	[1.10, 1.26]	1.11	<0.001	estimable
Writing	Uncertainty	Challenge / Accept	1.09	[1.03, 1.17]	0.81	0.017	estimable

Appendix

Table 21

Predictor blocks	In-sample AUC	Held-out AUC	Held-out log loss
Response features only	0.556	0.551 (0.004)	0.788 (0.007)
Topic, intent, turn position	0.594	0.592 (0.005)	0.779 (0.008)
Task block + response features	0.607	0.603 (0.004)	0.775 (0.007)
Task block + model/system	0.610	0.605 (0.004)	0.774 (0.007)
Task + model/system + features	0.620	0.615 (0.003)	0.771 (0.007)
Full: preceding + length and truncation	0.625	0.621 (0.003)	0.769 (0.007)

Appendix

Table 17. In-sample and held-out model fit. Parentheses give the standard deviation over five grouped folds.

Figure 7. Response-pair counts, conversation counts, qualifying rates, and platform-specific pair counts. Joined sample and platform denominators.

Figure 8. Blinded audits of outcome, topic/intent, and lexicon precision. Positive-case audits estimate precision, not recall.

Figure 9. Platform-specific frequency of epistemic and attribution cues in the full joined sample.

Figure 10. Platform-specific URL and inspectable-support prevalence. Denominators are full joined response pairs.

Figure 11. Technical Help: uncertainty and the differing checking/challenging estimates.

Figure 12. Seeking Information: differential refusal and reasoning-trace associations.

Figure 13. Conversation-grouped predictive comparisons, with topic/intent/turn blocks and response-feature blocks.

Figure 14. Pooled and conversation-fixed-effect lag estimates and jackknife. This chart shows association, not causal carry-over.

Figure 15. Share of qualifying turns classified as scrutiny by normalized position within the conversation, in the at-least-five-turn sample.

Figure 16. Reasoning versus non-reasoning model family among model-identified ShareChat ChatGPT responses: uncertainty, inspectable support, and subsequent scrutiny. Descriptive, nonrandomized comparisons.

Figure 17. Endpoint contrast for model-version scrutiny rates within WildChat; complete counts for other versions not shown in the source presentation.

Figure 18. ShareChat platform composition and available preserved source-object fields.

LLM Answer Verification

Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.

See all 156 papers

Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
Sep 30, 2026Guang Yang, Homa Hosseinmardi, Fengchen Liu +1LLM Sycophancy LLM Reliability
OASIS Lab, University of California, Los Angeles · University of California, Berkeley
Human-LLM Deliberation as Interactive Proof: Conditions for Verifiability Without Transparency
Sep 21, 2026Baotong Zhang, Dean Foster, João SedocLLM Answer Verification Anytime-Valid Inference
Stern School of Business, New York University, New York, NY, USA · Amazon, New York, NY, USA
Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations
May 13, 2026Qisong He, Jinwei Hu, Xinmiao Huang +3Runtime Verification LLM Grounding
School of Computer Science & Informatics, University of Liverpool

Sep 30, 2026J/K move · Enter open · S save

Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding

Guang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian

OASIS Lab, University of California, Los Angeles · University of California, Berkeley

LLM Sycophancy LLM Reliability LLM Evaluation Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) often abandon a correct answer, or endorse a user's position, once the user pushes back. This behavior, called sycophancy, is usually reported as a single rate per model, which says little about when it happens or how a user can avoid it. We study the conditions that produce it with 103,939 graded replies from ten configurations: eight LLMs with reasoning disabled, and two of them again with maximum reasoning, all facing the same 200 items, 13 pressure conditions, and four-turn conversations, with every reply labeled by two independent LLM judges. We find that the dominant factors are how costly it is for the model to verify the user's claim, and whether a trained guardrail covers it. Removing this task factor from a logistic model costs 0.485 of McFadden

R^2

, against 0.139 for model family and 0.009 for pressure tactic. Anchored facts are almost never conceded (1.3%), while adoption on logic puzzles rises with the number of clues needed to refute the pushed answer. Personal choices are endorsed in 77.0% of conversations. Most concessions on hard items come from models that cannot reliably solve them; models that can solve them rarely give the answer up. For both models tested, maximum reasoning removes these concessions completely: adoption on deep puzzles falls from 19.2% and 12.5% to 0%. Fallacious or emotional framing adds nothing beyond plain repetition. Three human annotators agree with the judges' consensus on 118/120 calibration items. These results give practical rules for reliable use: simplify hard-to-verify problems and reason deeply, state the question rather than one's preferred answer, ask for evidence on open questions, and choose models by their measured guardrail profile.

Abstract

LLM Answer Verification

Explore similar work

Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding

Human-LLM Deliberation as Interactive Proof: Conditions for Verifiability Without Transparency

Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations