cs.CLFeb 20, 2026
SavePerceived Political Bias in LLMs Reduces Persuasive Abilities
Organizations: Institute of Political Science Leiden University
Abstract
Conversational AI has been proposed as a scalable way to correct public misconceptions and spread misinformation. Yet its effectiveness may depend on perceptions of its political neutrality. As LLMs enter partisan conflict, elites increasingly portray them as ideologically aligned. We test whether these credibility attacks reduce LLM-based persuasion. In a preregistered U.S. survey experiment (N=2144), participants completed a three-round conversation with ChatGPT about a personally held economic policy misconception. Compared to a neutral control, a short message indicating that the LLM was biased against the respondent's party attenuated persuasion by 28%. Transcript analysis indicates that the warnings alter the interaction: respondents push back more and engage less receptively. These findings suggest that the persuasive impact of conversational AI is politically contingent, constrained by perceptions of partisan alignment.
Figures & tables
Figure 1: A. Percent of respondents in the final sample reporting at least moderate agreement with each economic misconception. B. Mean change (and SE) in agreement with the misconception on a 0–4 scale in the control arm, by party affiliation (including those who were forced to indicate closeness to one of the two parties).
Figure 2: A: Standardized treatment effects (N=2144) for each arm relative to the no-information condition or the non-directional bias condition. Point estimates and 95% confidence intervals are from OLS models of post-treatment misconception agreement, including topic fixed effects and pretreatment agreement. B: Percent of respondents in each condition who moved from at least moderate agreement with a misconception to either no agreement with the misconception or agreement with the consensus position. C: Standardized coefficients and 95% confidence intervals for topic-by-treatment interactions from an OLS model including the heavy treatment arm, the control arm, topic fixed effects, and their interactions, as well as pretreatment agreement. See SI:A for the raw coefficients.
Figure 3: A: Perceived out-party bias by treatment and partisanship. Shows the percentage of respondents who indicated post-conversation that they believed ChatGPT had a “weak” or “strong” out-party bias when asked, “Which of the following corresponds to your beliefs about the political bias of ChatGPT—ChatGPT has … a strong left-wing bias, has a weak left-wing bias, is relatively neutral, has a weak right-wing bias, has a strong right-wing bias.” We exclude “don’t know” responses from this analysis. Points represent group means with 95% confidence intervals. Results are shown pooled across all respondents and separately by ideological proximity to Democrats or Republicans. B: Shows the CATE by party closeness from OLS models estimating post-treatment agreement with the assigned economic misconception, controlling for pretreatment agreement. Dots indicate standardized coefficients; bars indicate 95% confidence intervals.
Figure 4: A: ATEs and 95% CIs on conversation characteristics. Panel A reports coefficients from three OLS models predicting (i) respondent word count, (ii) respondent argumentativeness, and (iii) respondent dismissiveness. Argumentativeness and dismissiveness are based on LLM-judged pairwise transcript comparisons, scaled using a Bayesian Bradley–Terry model ( DiGiuseppe and Flynn, 2026 ) . These two estimates aggregate results from 10 regressions, each using a draw from the posterior distribution of the Bradley–Terry scores. All models include treatment indicators, topic fixed effects, and pretreatment agreement. B: Bradley–Terry argumentativeness point estimates with 95% credible intervals, sorted from low to high.
| Full Model | Base: Undirected | |
| Pre-Treatment Beliefs (Std) | 0.371*** (0.021) | 0.371*** (0.021) |
| Partisan Treatment (Heavy) | 0.223*** (0.056) | 0.206*** (0.056) |
| Partisan Treatment (Light) | 0.175** (0.057) | 0.158** (0.057) |
| Control (No Info) | -0.017 (0.058) | |
| Undirected Bias | 0.017 (0.058) | |
| HH-Govt Debt Analogy | -0.045 (0.063) | -0.045 (0.063) |
Table SI:1: Main Effects (Fig. 2:A)
| Full Model | Base: Undirected | |
| Partisan Treatment (Heavy) | 0.342*** (0.086) | 0.316*** (0.085) |
| Partisan Treatment (Light) | 0.269** (0.088) | 0.243** (0.087) |
| Control (No Info) | -0.026 (0.089) | |
| Undirected Bias | 0.026 (0.089) | |
| HH-Govt Debt Analogy | -0.068 (0.097) | -0.068 (0.097) |
| Zero-Sum Immigration | 0.295* (0.130) | 0.295* (0.130) |
Table SI:2: Main Effects (Untransformed, DV = 0–4)
Figure SI:5: CATE and 95% confidence intervals by alignment with or against the party position on the topic. We measure alignment as Republicans who agree with the rent control, Buy American, and trade deficits misconceptions or Democrats who agree with tax cuts as revenue-neutral, the household/government debt analogy, or immigration as zero-sum. Estimates are from an OLS model of post-treatment misconception agreement that includes treatment-by-strength interactions, topic fixed effects, and pretreatment agreement. Error bars indicate 95% confidence intervals.
Figure SI:6: Partisan treatment effects moderated by partisan strength. Marginal effects of the heavy and light partisan bias treatments on post-treatment misconception agreement (standardized) for strong versus weak partisans (including leaners). Strong partisans are those at the endpoints of the six-point party identification scale. Estimates are from an OLS model of post-treatment misconception agreement that includes treatment-by-strength interactions, topic fixed effects, and pretreatment agreement. Error bars indicate 95% confidence intervals.
Figure SI:7: Partisan treatment effects moderated by affective polarization. We measure affective polarization using an additive index of three items (five-point agree/disagree scales): (i) “My identity as a [Democrat/Republican] is connected to my core moral beliefs,” (ii) “If I found out a friend of mine was a [Republican/Democrat], I would want to stop spending time with them,” and (iii) “[Republicans/Democrats] act in ways that we [Democrats/Republicans] could never understand.” Marginal effects of the heavy and light partisan bias treatments on post-treatment misconception agreement (standardized) across levels of affective polarization (standardized additive index). Estimates are from an OLS model of post-treatment misconception agreement that includes treatment-by-affective-polarization interactions, topic fixed effects, and pretreatment agreement. Shaded regions indicate 95% confidence intervals.
Figure SI:8: Partisan treatment effects moderated by pre-treatment AI trust. A: Marginal effects of the heavy and light partisan bias treatments on post-treatment misconception agreement (standardized) across levels of pre-treatment AI trust (0–4 ordinal scale). We measured trust with the following question: “How much do you trust artificial intelligence chatbots, like ChatGPT, to give you reliable information on topics you care about?” Estimates are from an OLS model of post-treatment misconception agreement that includes treatment-by-trust interactions, topic fixed effects, and pretreatment agreement. Shaded regions indicate 95% confidence intervals. B: Distribution of pre-treatment AI trust responses (N=2,144).
Figure SI:9: Treatment effects moderated by self-reported topic knowledge. A: Marginal effects of the heavy and light partisan bias treatments by a binary knowledge indicator (high knowledge = respondents reporting at least moderate familiarity with the assigned topic (3, 4)). Sample sizes for each group are reported in parentheses. B: Marginal effects across the continuous self-reported knowledge scale (0–4). All estimates are from OLS models of post-treatment misconception agreement that include treatment-by-knowledge interactions, topic fixed effects, and pretreatment agreement. Error bars and shaded regions indicate 95% confidence intervals.
Figure SI:10: This figure shows the standardized coefficients capturing the ATEs of our treatment conditions from four OLS models, each estimating a different outcome: perceived persuasiveness of the LLM during the conversation, willingness to use AI again to challenge beliefs, trust in AI for reliable information, and support for politicians using AI to learn about policy issues. Each model includes topic fixed effects and pre-treatment agreement with the economic misconception. The Trust in AI and Support for politicians models also include a pre-treatment trust-in-AI measure as a covariate. All outcomes are measured on a 0–4 ordinal scale and then standardized.
Figure SI:11: Power for a 2-arm comparison, assuming that the covariates correlated with the outcome at R-squared = 0.50. 10,000 simulations were conducted.
More in this topic
Persuasion in Language Models
Persuasion in language models studies how they influence human and AI beliefs or decisions, and how to measure and limit undue influence.
See all 79 papers