On the Suitability of LLM-Driven Agents for Dark Pattern Audits

Chen Sun; Yash Vekaria; Rishab Nithyanand

cs.CRMar 4, 2026

On the Suitability of LLM-Driven Agents for Dark Pattern Audits

Authors: Chen Sun, Yash Vekaria, Rishab Nithyanand

Organizations: University of Iowa · University of California, Davis

Abstract

As LLM-driven agents begin to autonomously navigate the web, their ability to interpret and respond to manipulative interface design becomes critical. A fundamental question that emerges is: can such agents reliably recognize patterns of friction, misdirection, and coercion in interface design (i.e., dark patterns)? We study this question in a setting where the workflows are consequential: website portals associated with the submission of CCPA-related data rights requests. These portals operationalize statutory rights, but they are implemented as interactive interfaces whose design can be structured to facilitate, burden, or subtly discourage the exercise of those rights. We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows, structured evidence gathering, and classification of potential dark patterns. Across a set of 456 data broker websites, we evaluate: (1) the ability of the agent to consistently locate and complete request flows, (2) the reliability and reproducibility of its dark pattern classifications, and (3) the conditions under which it fails or produces poor judgments. Our findings characterize both the feasibility and the limitations of using LLM-driven agents for scalable dark pattern auditing.

Figures & tables

Figure 1. Overview of our experiment design and methodology ( cf. Section 3 ). A multi-phase workflow diagram illustrating the study design.

Pattern	Definition	Harm Mechanism Example
Adding Steps	Subverts the user’s expectation that a task will take as few steps as technologically necessary, instead introducing additional required interactions.	One identifier per submission forces users with multiple identifiers (e.g., email address, IP address, MAID) to repeat the full request process multiple times.
Conflicting Info.	Includes two or more sources of information that conflict with each other, creating uncertainty about the consequences of the user’s actions.	Different pages list incompatible official submission methods (e.g., one page states email is valid, while another states only a webform is accepted).
Creating Barriers	Subverts the user’s expectation that relevant tasks will be supported by the interface, instead preventing or unnecessarily complicating the task to disincentivize user action.	The request requires highly sensitive or burdensome materials (e.g., government-issued ID, SSN, biometric verification) that exceed what is reasonably necessary.
Feedforward Ambiguity	Subverts the user’s expectation that their choice will result in a predictable outcome, instead creating a discrepancy between provided information and the resulting action.	A link promises a general privacy or access request pathway, but the destination interface supports only a limited subset of rights or redirects users into an unrelated workflow (e.g., account creation).
Hidden Info.	Subverts the user’s expectation that relevant information will be accessible and visible, instead disguising it or framing it as irrelevant.	A section framed as comprehensive (e.g., “California Privacy Rights”) omits key submission methods that exist elsewhere without cross-reference, leading users to stop searching prematurely.
Info. Without Context	Alters relevant information or user controls in ways that limit discoverability, making it unlikely that users will find the information or action they seek.	Access submission instructions appear only in unrelated sections (e.g., “Contact Us” or “Opt-Out”) and are not referenced in the expected rights section.

Table 1. High-frequency dark pattern categories in right-to-access workflows observed in our ground truth sample of 100 data brokers.

Strategy	Class. Acc. (%)		Precision (%)		Recall (%)		F1 (%)		Expl. Acc. (%)
L1: Zero-shot	70.8		61.0		60.1		60.5		78.1
L2: Zero-shot + Role	63.6		49.9		73.3		59.4		71.1
L3: Few-shot + Role	83.5		79.9		74.0		76.9		95.8
L4: Few-shot + Role + CoT	86.7		88.0		74.4		80.7		98.5
	$\Delta$	95% CI	$\Delta$	95% CI	$\Delta$	95% CI	$\Delta$	95% CI	$\Delta$	95% CI
$\Delta$ L1 $\to$ L2 (+Role)	$-7.1^{*}$	[ $-14.7$ , $0.0$ ]	$-11.12^{*}$	[ $-21.9$ , $-0.4$ ]	$+13.3^{*}$	[ $0.0$ , $26.4$ ]	$-1.2$	[ $-11.1$ , $8.6$ ]	$-6.9$	[ $-20.6$ , $8.0$ ]

Table 2. RQ1: Performance across four prompting strategies (cf. Section 4.1 ). Classification accuracy, precision, recall, and F1-score measure the the agent’s ability to correctly label dark patterns. The explanation accuracy measures the agent’s ability to provide supporting evidence and reasoning. It is only measured when the agent has correctly identified a dark pattern. The lower section reports the difference

\Delta

between consecutive strategies, where the 95% CI is computed using 1000 resamples of 50 ground truth workflows. ∗ indicates a statistically significant change (

p<0.05

Pattern	Class.	Prec.	Rec.	F1	Expl.
	Acc. (%)	(%)	(%)	(%)	Acc. (%)
Adding Steps	96.6	95.2	90.9	93.0	100.0
Conflicting Info.	88.6	95.7	71.0	81.5	100.0
Creating Barriers	86.7	91.3	85.7	88.4	97.6
Feedforward Ambiguity	86.8	94.1	76.2	84.2	100.0
Hidden Info.	69.9	85.0	40.5	54.8	94.1

Table 3. RQ1: Performance by dark pattern category under the selected Few-shot + Role + CoT configuration (cf. Section 4.2 ).

Pattern	Ground Truth		Deployment
	Prev. (%)	95% CI	Prev. (%)	95% CI
Adding Steps	24.1	[14.9, 33.3]	15.2	[11.1, 19.4]
Conflicting Info.	26.1	[17.0, 35.3]	20.1	[15.4, 24.9]
Creating Barriers	55.6	[44.7, 66.4]	48.6	[42.9, 54.4]
Feedforward Amb.	37.8	[27.8, 47.8]	25.2	[20.0, 30.4]
Hidden Info.	21.1	[12.7, 29.5]	21.1	[16.2, 26.0]

Table 4. RQ1: Dark pattern prevalence estimates (cf. Section 4.3 ). Ground truth prevalence is computed from the 100-sample ground truth dataset. Deployment prevalence reflects observations across the remaining 356 brokers.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2. Full agent prompt (Part 1 of 3) Prompt contains role, goal and interaction plan sections.

Panel of Figure 3 — Figure 3. Full agent prompt (Part 2 of 3) Prompt contains taxonomy, evaluation instructions and decision rules sections.

Panel of Figure 4 — Figure 4. Full agent prompt (Part 3 of 3) Prompt contains evidence requirements section.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 832 papers

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
May 14, 2026William Lugoloobi, Samuelle Marro, Jabez Magomere +2Language Model Fingerprinting LLM Agent Security
Oxford Internet Institute, University of Oxford · Department of Engineering Science, University of Oxford
AI Watchdog: Agent Interfaces for Detecting and Defending Against Manipulative Dark Patterns in AI Conversations
Aug 22, 2026Rachel Poonsiriwong, Chayapatr, Archiwaranguprok +4AI Agent Monitoring Human Oversight of AI
Pub · MIT Media Lab Cambridge, MA, USA · KASIKORN Labs Nonthaburi, Thailand
Do Web Agents Investigate Before They Decide?
Feb 5, 2026Syed Nazmus Sakib, Nafiul Haque, Tapodhir Karmakar Taton +2Web Agent Benchmarks Hallucination in Language Models
Department of Robotics and Mechatronics Engineering, University of Dhaka · Department of Computer Science and Engineering, University of Dhaka

May 14, 2026J/K move · Enter open · S save

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

William Lugoloobi, Samuelle Marro, Jabez Magomere, Joss Wright+1

Oxford Internet Institute, University of Oxford · Department of Engineering Science, University of Oxford

Language Model Fingerprinting LLM Agent Security

Open paper AskPDF arXiv

As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1. We formalise this attack surface by demonstrating that classifiers trained on agent actions generalise across model sizes and families. We further show that strong classifiers can be trained from few interaction traces and that agent identity can be inferred early within an episode. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our harness and a labelled corpus of agent traces here.

Abstract

LLM Agent Reliability

Explore similar work

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

AI Watchdog: Agent Interfaces for Detecting and Defending Against Manipulative Dark Patterns in AI Conversations

Do Web Agents Investigate Before They Decide?