Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms

Oleg Vasilyev

cs.CLOct 29, 2026

Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms

Authors: Oleg Vasilyev

Organizations: Independent Researcher

Abstract

Chatbots are increasingly consulted about public affairs, and their judgments of policies are useful only if they are robust. We measure this robustness in the setting of an ordinary user, through consumer chat interfaces. For 30 days, five chatbot configurations (Mistral, Gemini Flash, Claude Sonnet, and Claude Opus at two effort settings) were asked twice a day to score the long-term effect on the United States of 40 actions of the last four presidential administrations and of 15 hypothetical reforms, each with a confidence and a short comment. The judgments are robust: all configurations agree on the sign and the ranking of the administrations' mean scores, and two different configurations asked on the same day give opposite signs to the same action in about 4% of cases. Responses from different days differ more than same-day responses, but the difference grows slowly over the month, except where the service itself changed. We identify three such changes, which shifted the judgments while leaving little or no trace in the interface. Treating the comments as a retrieval task, we find them specific to each item and consistent across runs and chatbots, with lexical and embedding similarities agreeing. We release the dataset of 16,500 judgments.

Figures & tables

Figure 1: All 600 prompt runs: 5 chatbot configurations

\times

2 prompts

\times

30 days

\times

2 runs a day, shown as small boxes. The setup is explained in Section 2 .

Figure 2: The 40 presidential actions by the year in which each was launched (the first year, for actions that span several years). By administration: Obama 10, Trump1 9, Biden 9, Trump2 12.

Figure 3: Mean score (left) and mean confidence (right) of each model for the actions of each administration, averaged over all days and runs. O: Obama; T1 and T2: the first and second Trump administrations; B: Biden.

Figure 4: Summary of Figure 3 . Left: for each administration, the bar spans the lowest and the highest of the five models’ mean scores, and the dark line is their average. Right: the same for the confidence, with the confidence in the hypothetical reforms (R) added on the right.

Figure 5: Distribution of the scores of each item over all models, days and runs (300 responses per item). The area of a dot is proportional to the share of the responses. The actions are in chronological order and colored by administration; the reforms are in the order of their ids. The score scales are defined in Figures 12 and .

Figure 6: Mean absolute difference between the scores of two models on the same item and day (Equation 1 , off the diagonal), and between the two runs of the same model (Equation 2 , on the diagonal), for the actions (left) and the reforms (right).

Figure 7: As Figure 6 , for the confidence of the actions and for the confidence, contestedness and irreversibility of the reforms.

Figure 8: Strong disagreements between responses on the same action on the same day: the share of the pairs of responses with opposite score signs (left), and with scores that differ by more than 1 (right). The pairs are formed as in Equations 1 and 2 ; on the diagonal, the two runs of the same model.

Figure 9: Drift of the scores (Equations 3 and 4 ), averaged over the items, as a function of the number of days

d

between the responses.

	August	January	$p$
Comment length	82.5	71.0	$<$ 0.0001
Reforms score	3.41	3.67	0.0004
Reform contestedness	3.49	3.78	$<$ 0.0001
Share of action score ${}=0$	4.6%	2.1%	0.0203

Table 1: Sonnet’s answers on the days it reported a different knowledge cutoff. August: the 6 days (6–9, 11, 12) on which Sonnet, asked daily for its knowledge cutoff, answered “End of August 2026”; January: the 23 days on which it answered “January 2026”. Day 16, when it gave both answers, is left out. Each quantity is averaged per day over items and both runs, then over the days of each group. Comment length is in characters and pools the comments on actions and reforms; reforms score is the magnitude (1–5).

p

: exact two-sided permutation test on the daily values, over all 475,020 ways of choosing 6 of the 29 days.

Figure 10: Sonnet’s daily mean comment length (left) and mean contestedness of the reforms (right). Shaded: the days on which Sonnet reported its knowledge cutoff as “End of August 2026”; lighter: day 16, when it gave both responses. Dashed: the mean over the days on which it reported “January 2026”.

	Before	After	$p$
Comment length, actions	30.8	89.5	$<$ 0.0001	†
Comment length, reforms	22.6	86.5	$<$ 0.0001	†
Action $\|$ score $\|$	3.20	2.50	$<$ 0.0001
Action confidence	4.05	2.92	$<$ 0.0001	†
Reforms score	3.80	3.16	$<$ 0.0001
Reforms confidence	3.74	2.72	$<$ 0.0001	†

Table 2: Mistral before (13–26 September, days 1–14) and after (27 September–12 October, days 15–30) its switch to the new Vibe experience. Each quantity is computed per day over items and both runs, then averaged over the days of each period. Comment length is in characters. Reforms score is the magnitude (1–5). Run-to-run difference: mean absolute difference between the two daily runs’ action scores (Eq. 2 for Mistral alone). Distance to other models: mean absolute difference between Mistral’s action score and the mean score of the other four models on the same day.

p

: two-sided permutation test on the daily values, 10 6 random splits of the 30 days. † Every day after the switch lies beyond every day before it.

Opus high
	Before 22 Sep		From
	Unusual	Other	22 Sep	$p$
Action $\|$ score $\|$	2.04	2.80	2.81	0.0004
Action confidence	2.68	3.16	3.18	0.0005
Share of score ${}=0$	8.0%	0.3%	1.0%	0.0019
Hedging comments	12.5%	6.2%	7.0%	0.0242

Table 3: Opus 5 in the week before Opus 5.5 was released. Each quantity is computed per day over items and both runs, then averaged over three groups of days: Unusual, the days 3, 4, 6, 7 and 9 (15–21 September); Other, the other days before the release (1, 2, 5, 8); and the days from the release of Opus 5.5 on 22 September on (10–30). Share of score

{}=0

and hedging comments refer to the actions; a comment counts as hedging if it contains “uncertain”, “unclear”, “contested”, “disputed”, “debat(e)”, “mixed”, “too recent” or “too early”. Reforms score is the magnitude (1–5). The unusual days were identified in the data and are shown for description;

p

tests the split fixed by the release date, all days before it (1–9) against all days from it (10–30): two-sided permutation test on the daily values, 10 6 random splits.

Figure 11: Retrieval of comments across models: mean average precision for each query model (rows) and ranked model (columns), with Jensen–Shannon (JS) and three embedding models, for the actions (top) and the reforms (bottom). Diagonal cells (outlined): query and ranked comments from different runs of the same model. Chance level is about 0.025 for the actions and 0.067 for the reforms.

Metric	$\rho$	$\tau_{b}$	Range	$\geq 0.99$
MAP			0.54–0.97	0%
R-precision	0.98–1.00	0.88–0.98	0.49–0.93	0%
MAP@ $K$	0.98–1.00	0.89–1.00	0.40–0.93	0%
nDCG@ $K$	0.99–1.00	0.92–0.98	0.57–0.95	0%
nDCG	0.98–0.99	0.91–0.97	0.83–0.99	2%
AUC	0.93–0.97	0.79–0.89	0.86–1.00	22%

Table 4: Agreement of other ranking metrics with MAP. For each similarity measure, and separately for the actions and the reforms, each metric gives a

5\times 5

matrix with one cell for each pair of a query model and a ranked model, as in Figure 11 . Its 20 off-diagonal cells (pairs of different models) are compared with those of MAP, and the ranges are over the 8 matrices (4 similarity measures, for the actions and for the reforms).

\rho

: Spearman rank correlation with MAP;

\tau_{b}

: Kendall’s tau-b with MAP, which accounts for ties (frequent for MRR). Range: smallest and largest off-diagonal value over all matrices;

\geq 0.99

: share of off-diagonal cells at 0.99 or above.

	Actions			Reforms
Model	Own	Query	Ranked	Own	Query	Ranked
M	0.88	0.84	0.73	0.85	0.79	0.69
F	0.96	0.75	0.82	0.98	0.70	0.81
S	0.96	0.86	0.88	0.92	0.82	0.80
Oh	0.97	0.84	0.86	0.95	0.80	0.82
Om	0.96	0.79	0.80	0.93	0.75	0.72

Table 5: Each model as query and as ranked model: MAP averaged over the 4 similarity measures. Own: the diagonal cell, the model’s comments retrieved by its own query from the other run. Query: mean over the other four models of the model’s row, i.e. how well its averaged comments retrieve their comments. Ranked: mean of the model’s column, i.e. how well its comments are retrieved by the other models’ queries.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 495 papers

LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces
Feb 20, 2026Peter Kirgis, Ben Hawriluk, Sherrie Feng +3Language Model Safety Evaluation LLM Safety Benchmarks
Princeton University · Phillips Exeter Academy · Independent
Before You Poll with LLMs: A Deliberative Diagnostic Framework
Sep 14, 2026Ahmed Wali, Hassaan TayyabLLM Evaluation Political Bias in Language Models
Lahore University of Management Sciences
How To Do Things With Prompts
Aug 29, 2026Kristina Šekrst, Virna KarlićLLM Prompting Human-AI Interaction

Feb 20, 2026J/K move · Enter open · S save

LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

Peter Kirgis, Ben Hawriluk, Sherrie Feng, Aslan Bilimer+2

Princeton University · Phillips Exeter Academy · Independent

Language Model Safety Evaluation LLM Safety Benchmarks LLM Auditing LLM Sycophancy Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

People increasingly hold sustained, open-ended conversations with large language models (LLMs). Public reports and early studies suggest that, in such settings, models can reinforce delusional or conspiratorial ideation or even amplify harmful beliefs and engagement patterns. We present an audit and benchmarking study that measures how different LLMs encourage, resist, or escalate disordered and conspiratorial thinking. We explicitly compare API outputs to user chat interfaces, like the ChatGPT desktop app or web interface, which is how people have conversations with chatbots in real life but are almost never used for testing. In total, we run 56 20-turn conversations testing ChatGPT-4o and ChatGPT-5, via both the API and chat interface, and grade each conversation by two research assistants (RAs) as well as by GPT-5. We document five results. First, we observe large differences in performance between the API and chat interface environments, showing that the universally used method of automated testing through the API is not sufficient to assess the impact of chatbots in the real world. Second, when tested in the chat interface, we find that ChatGPT-5 displays less sycophancy, escalation, and delusion reinforcement than ChatGPT-4o, showing that these behaviors are influenced by the policy choices of major AI companies. Third, conversations with nearly identical aggregate intensity in a behavior display large differences in how the behavior evolves turn by turn, highlighting the importance of temporal dynamics in multi-turn evaluation. Fourth, even updated models display substantial levels of negative behaviors, revealing that model improvement does not imply model safety. Fifth, the same API endpoint tested just two months apart yields a complete reversal in behavior, underscoring how transparency in model updates is a necessary prerequisite for robust audit findings.

Abstract

Language Model Robustness

Explore similar work

LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces

Before You Poll with LLMs: A Deliberative Diagnostic Framework

How To Do Things With Prompts