Promoting Critical Thinking With Domain-Specific Generative AI Provocations

Thomas Şerban von Davier; Hao-Ping Lee; Jodi Forlizzi; Sauvik Das

cs.HCMar 20, 2026

Promoting Critical Thinking With Domain-Specific Generative AI Provocations

Authors: Thomas Şerban von Davier, Hao-Ping Lee, Jodi Forlizzi, Sauvik Das

Organizations: Carnegie Mellon University Pittsburgh, PA, United States

Abstract

The evidence on the effects of generative AI (GenAI) on critical thinking is mixed, with studies suggesting both potential harms and benefits depending on its implementation. Some argue that AI-driven provocations, such as questions asking for human clarification and justification, are beneficial for eliciting critical thinking. Drawing on our experience designing and evaluating two GenAI-powered tools for knowledge work, ArtBot in the domain of fine art interpretation and Privy in the domain of AI privacy, we reflect on how design decisions shape the form and effectiveness of such provocations. Our observations and user feedback suggest that domain-specific provocations, implemented through productive friction and interactions that depend on user contribution, can meaningfully support critical thinking. We present participant experiences with both prototypes and discuss how supporting critical thinking may require moving beyond static provocations toward approaches that adapt to user preferences and levels of expertise.

Figures & tables

Case Study	Domain	Theoretical Grounding	Model Type
ArtBot	Art Interpretation	Dialogic Education ( Skidmore and Murakami, 2016 )	Local, open-weight models ( Meta, 2024 )
Privy	AI Privacy Reports	Framework Grounded Provocations ( Das et al., 2022 )	Cloud-host, proprietary models ( Wei et al., 2022 )

Table 1. This table provides an overview on the theoretical grounding underlying both systems and which specific domains each one operates under.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1. ArtBot is an LLM-powered tool that challenges participants with questions to encourage them to share their own interpretations of the artwork in the shared workspace.

Figure 2. Privy is an LLM-powered tool that guides practitioners through structured privacy impact assessments to: (i) identify relevant risks in novel AI product concepts, and (ii) propose appropriate mitigations.

Generative AI in Education

Generative AI in education uses language and multimodal models to support teaching, learning, feedback, assessment, and educational design.

See all 287 papers

Counterargument for Critical Thinking as Judged by AI and Humans
May 6, 2026Tosin Adewumi, Marcus Liwicki, Foteini Simistira Liwicki +3LLM-as-a-Judge Educational Assessment
Machine Learning Group, EISLAB, Luleå University of Technology, Sweden.
MetaCues: Enabling Critical Engagement with Generative AI for Information Seeking and Sensemaking
Mar 20, 2026Anjali Singh, Karan Taneja, Zhitong Guan +1Human-AI Interaction
The University of Texas at Austin, USA · Georgia Institute of Technology, USA
Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI
Jun 4, 2026Alexander Apartsin, Yehudit ApersteinGenerative AI in Education Human-AI Collaboration
Holon Institute of Technology · Afeka College of Engineering

May 6, 2026J/K move · Enter open · S save

Counterargument for Critical Thinking as Judged by AI and Humans

Tosin Adewumi, Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled+2

Machine Learning Group, EISLAB, Luleå University of Technology, Sweden.

LLM-as-a-Judge Educational Assessment Generative AI in Education Automated Essay Scoring

Open paper AskPDF arXiv

This intervention study investigates the use of counterarguments in writing for critical thinking by students in the context of Generative AI (GenAI). This is especially as risks of cheating and cognitive offloading exist with the use of GenAI. We presented 36 students in a particular university course with 4 carefully selected thesis statements (from a set of popular debates) to write about anyone of them. We used six established rubrics (focus, logic, content, style, correctness and reference) to conduct three human assessments (two student peer-reviews and one experienced teacher) per writeup on a 5-point Likert scale for all the qualified samples (n) of 35 submissions (after disqualifying one for irregularity). Using the same rubrics and guidelines, we also assessed the submissions using six frontier LLMs as judges. Our mixed-method design included qualitative open-ended feedback per assessment and quantitative methods. The results reveal that (1) the students' self-written counterarguments to AI-generated content contains logic, among other things, which is a key component of critical thinking, and (2) GenAI can be successfully used at scale to assess students' written work, based on clear rubrics, and these assessments generally align with human assessments as shown with Gwets AC2 inter-rater reliability values of 0.33 for all the models except one.

Abstract

Generative AI in Education

Explore similar work

Counterargument for Critical Thinking as Judged by AI and Humans

MetaCues: Enabling Critical Engagement with Generative AI for Information Seeking and Sensemaking

Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI