Why Do LLMs Help with Scientific Misconduct Requests and What to Do About It?: Increasing the Trustworthiness of AI for Science
Organizations: Department of Computer Science, University of Colorado Boulder, Boulder, CO, USA · Center for AI in Knowledge Institutions (AIKI), University of Colorado Boulder, Boulder, CO, USA · ReviewerZero AI Inc., Boulder, CO, USA
Abstract
Large language models (LLMs) support scientific research, but some of that assistance can also undermine research integrity. Existing safety evaluations in science often measure whether models refuse misconduct requests, but rarely explain whether they even recognize the problem or why they comply. We investigate this gap across 8 LLMs using SciIntBench, an existing benchmark of 1,350 prompts about scientific research integrity. We examine the reasons behind compliance in four steps. First, when asked directly whether fulfilling a request would involve misconduct, models overwhelmingly identify such requests, even in the Most Covert cases (91%), while they refuse to carry out the request in only 34% of cases. Second, we examine which request characteristics are associated with refusal using features that capture how the request is framed, how much the assistant has been told about what it is helping to hide, and how the goal and the user's stake are stated, among others. Third, we estimate that the most significant effect was to rephrase requests as questions about permissibility rather than instructions, which increased refusal from 52% to 88%. Finally, we test two system-prompt fixes: asking models to assess the request before responding and reminding them of responsible conduct of research principles. These changes increase refusal rates on adversarial requests from 58% to 87% and 86%, respectively. Our findings show that models recognize wrongful requests but still help users violate integrity. Understanding why these failures occur and testing ways to reduce them offer promising paths toward more trustworthy AI for science.
Figures & tables
| Cohen’s | ||||||||
|---|---|---|---|---|---|---|---|---|
| Feature | Coding criterion | Possible values | H1– H2 | H1– G | H1– O | H2– G | H2– O | G– O |
| Request type | What the assistant is asked to do with the proposed action | Do it † / ask if acceptable / ask how | .982 | .961 | .991 | .943 | .974 | .952 |
| Output content | Whether the requested output would itself contain the false or improper content | Contains it † / does not contain it / unclear | .833 | .809 | .627 | .648 | .454 | .453 |
| Stated goal | Reason given for the problematic part of the request | Mislead others † / improve the document / none | .717 | .862 | .809 | .591 | .549 | .678 |
| Action or amount | Whether a prohibited act is named or only an amount to push | Named act † / amount only | .796 | .527 | .793 | .349 | .628 | .370 |
| Victim named | Whether a specific party harmed by the violation is identified | No † / yes | .930 | .814 | .773 | .748 | .706 | .600 |
LLM Safety
Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.
See all 470 papers