Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineering
Organizations: Monash University Melbourne, Australia · Singapore Management University Singapore City, Singapore
Abstract
How software developers interact with Artificial Intelligence (AI)-powered tools, including Large Language Models (LLMs), plays a vital role in how these AI-powered tools impact them. While overreliance on AI may lead to long-term negative consequences (e.g., atrophy of critical thinking skills); underreliance might deprive software developers of potential gains in productivity and quality. Based on twenty-two interviews with software developers on using LLMs for software development, we propose a preliminary reliance-control framework where the level of control can be used as a way to identify AI overreliance and underreliance. We also use it to recommend future research to further explore the different control levels supported by the current and emergent LLM-driven tools. Our paper contributes to the emerging discourse on AI overreliance and provides an understanding of the appropriate degree of reliance as essential to developers making the most of these powerful technologies. Our findings can help practitioners, educators, and policymakers promote responsible and effective use of AI tools.
Figures & tables
| Type of Control | Description | Example |
| Self-Control over AI | Software developers do not just hold control over AI, but also over themselves. They can overcome any urge to use AI. | Abstaining from using AI tools when studying |
| Taking Control over AI | Software developers assume strong control over AI, aware of potential flaws in AI (e.g., data breaches). Because of this, developers are unable to fully experience the benefits offered by AI tools. | Prompting with dummy data due to privacy concerns |
| Balanced Control over AI | Software developers find the middle ground in terms of control over LLMs leveraging LLM’s automation capabilities to improve their software development speed, while mitigating losses. | Boilerplate code generation |
| Handing over the Control to AI | Software developers start to step back, allowing AI to lead development tasks (e.g., problem solving). | Vibe coding |
| Losing Control over AI | Software developers lack in understanding of the code’s structure and flow, which impacts their debugging capabilities. | AI tools adding bugs or vulnerabilities into generated code |
| Type of Reliance | Description | Example |
| Self-Reliance | Software developers having confidence in their development skills, not constantly needing to seek support to finish tasks. | Turning off GitHub Copilot and coding by yourself |
| Reliance on Colleagues | Software developers start to increase reliance on LLMs; however, they still prioritise human support or collaboration to finish tasks. | Seeking colleagues for software architecture design |
| Appropriate Reliance | Those software developers increase their use of LLMs. However, they are also constantly vigilant of potential LLM pitfalls. | Cross-checking references |
| Overreliance on AI | software developers having excessive trust or dependence towards LLM. As a result, they can feel an urge to use LLMs. | Prompting for brainstorming before trying themselves |
| Full Automation | Software developers who fully rely on LLMs’ capabilities, enabling them to move their focus to other tasks | Meeting summarisation |
| Interaction Type | Type of Control | Type of Reliance |
| Conversational assistance | Taking control over AI | Appropriate reliance |
| Selection-based enhancements | Balanced control | Appropriate reliance |
| Explicit UI actions | Taking control over AI | Appropriate reliance |
| Comment-guided prompts | Balanced control | Appropriate reliance |
| Shortcut-activated prompts | Taking control over AI | Appropriate reliance |
| Command-driven actions | Taking control over AI | Appropriate reliance |