Generative AI for Managerial Decision-Making under Ambiguity and Sycophancy

Sule Ozturk Birim; Fabrizio Marozzo; Yigit Kazancoglu

cs.AIMar 4, 2026

Generative AI for Managerial Decision-Making under Ambiguity and Sycophancy

Authors: Sule Ozturk Birim, Fabrizio Marozzo, Yigit Kazancoglu

Organizations: Manisa Celal Bayar University, Salihli, Manisa, Turkey · University of Calabria, Rende, Italy · Yasar University, Izmir, Turkey

Abstract

Generative artificial intelligence (GenAI) is increasingly being integrated into complex business workflows, fundamentally shifting the boundaries of managerial decision-making. However, the reliability of its strategic advice in ambiguous business contexts remains a critical knowledge gap. To address this gap, this study compares multiple GenAI models in their ability to detect ambiguity, examines whether a systematic ambiguity-resolution process improves response quality, and investigates their susceptibility to sycophantic behavior when confronted with flawed managerial directives. Using a novel four-dimensional business ambiguity taxonomy, we conducted a human-in-the-loop experiment across strategic, tactical, and operational scenarios. The resulting decisions were assessed through a human-validated automated evaluation framework based on agreement, actionability, justification quality, and constraint adherence. The results show that our approach not only distinguishes different types of ambiguity, but also reveals how ambiguity resolution systematically changes model behavior. In particular, resolving ambiguities improved decision quality across all managerial levels, with the strongest gains observed in constraint adherence. The analysis further showed that sycophantic behavior is not uniform across models: some models challenged flawed assumptions, whereas others tended to comply with them. This study contributes to the bounded rationality literature by positioning GenAI as a cognitive scaffold that can detect and resolve ambiguities managers might overlook, while demonstrating that its artificial limitations require human oversight to ensure its reliability as a strategic partner.

Figures & tables

Ambiguity Type	Detailed Description	Source
1. Lexical and Semantic Ambiguity	Ambiguity arising from single words having multiple meanings, which leads to multiple interpretations if context is missing. For example, in "She is worried about the promotion", the word promotion could mean "marketing campaign" or "job advancement".	( Gleich et al., 2010 ; Zhang et al., 2024 ; Tang et al., 2025 )
2. Structural/ Syntactic Ambiguity	Ambiguity resulting from the grammatical structure or arrangement of words, where the phrase or sentence can be parsed in more than one way, yielding different meanings, like "new employee manual" can mean both "(new employee) manual" and "new (employee manual)".	( Kadłub, 2017 ; Li et al., 2024b ; Gleich et al., 2010 )
3. Scope and Quantification	Ambiguity concerning the extent or range of applicability, often involving quantifiers (e.g., every, all, some ) where the relative ordering or interpretation (collective vs. distributive) is unclear. For the sentence "every manager need two assistants", it is possible that every manager need two different assistants or there must be two specific assistants that every manager need.	( Li et al., 2024b ; Kamath et al., 2024 )
4. Coreference	Ambiguity arising when a pronoun, entity, or description refers back to multiple possible subjects, requiring external context (often conversation history) to resolve. "John argued with the supervisor and he seemed frustrated." Here, he could refer to either John or the supervisor.	( Guo et al., 2021 ; Yuan et al., 2023 ; Li et al., 2024b )
5. Contextual Missing Elements	Ambiguity where the query is structurally sound but lacks essential elements needed for a definitive or complete answer, leading to a multifaceted question. This includes missing contextual details necessary to scope the question. Given a query "We need to make launch ready for the target", the response may vary due to missing the details about the characteristics of the target.	( Zhang et al., 2024 ; Ehsani et al., 2025 )
6. Temporal and Event Context	Ambiguity concerning the specific time frame, date, or event being referenced, especially if the event has happened multiple times or the question is time-dependent. "We need to use the successful strategy we applied during the economic crisis." Here, which economic crisis?: 2008 or 2020 pandemic?	( Kim et al., 2023 ; Min et al., 2020 ; Zhang et al., 2024 )

Table 1: Taxonomy of Common Ambiguity Types in Prompts

Figure 1: The human–AI collaborative decision-making framework: a seven-stage architecture in which human and AI roles are explicitly delineated at each stage.

Feature	Strategic Decisions	Tactical Decisions	Operational Decisions
Primary Focus	Relates to the organization’s goals or the what the firm hopes to achieve ( Schmidt and Wilhelm, 2000 ) . They are characterized as high-level choices that guide or influence other subsequent decisions ( Steen, 2016 ) .	Concentrate on the functional means of achieving strategic ends such as the allocation of resources within functional areas ( Kuruppuge and Gregar, 2020 ) or material flow management policies ( Schmidt and Wilhelm, 2000 )	Relate to actions that impact daily operations in functional areas ( Schmidt and Wilhelm, 2000 ) . Decisions are typically made at a supervisory level ( Kuruppuge and Gregar, 2020 )
Time Horizon	Deals with a relatively long planning horizon, having long lasting effects ( Yolmeh and Baykal-Gürsoy, 2019 ; Abahmane and Binkkour, 2015 ) .	Categorized as the medium term, ranging from a few months up to two years ( Pereira et al., 2020 ) to serve as a bridge between long-term strategic design and the short-term requirements ( Schmidt and Wilhelm, 2000 )	Deals with short term such as daily decisions. These decisions relate to running periodic business process activities to reach short term goals ( Abahmane and Binkkour, 2015 ) .
Organizational Scope	Concerned with the whole organization and its long term objectives, often being global and integrated ( Schmidt and Wilhelm, 2000 ) .	Sectional or departmental, focusing on internal work groups, leaded by senior and middle managers ( Kuruppuge and Gregar, 2020 ; Fredrickson, 1986 )	Concern, in general, work groups or departments inside the organization ( Abahmane and Binkkour, 2015 ) .
Objective	Focus on the organization’s long-term economic viability which is related to the survival of the firm ( Kuruppuge and Gregar, 2020 ) .	Focus on goal-directed behavior adapting to unexpected changes ( Evertsz et al., 2015 ) and system-wide cost efficiency by optimizing logistical factors ( Keskin et al., 2010 )	Focus on using resources effectively ( Kuruppuge and Gregar, 2020 ; Shivakumar, 2014 )
Data Characteristics	Data is derived from internal and external sources, with extensive use of external data. Decisions are based on both quantitative and qualitative information ( Abahmane and Binkkour, 2015 ) .	Data is aggregated summarized quantitative information to track whether the organization is meeting performance targets ( Fredrickson, 1986 ; Schmidt and Wilhelm, 2000 ; Jalal et al., 2021 ; Pereira et al., 2020 )	Data is primarily generated internally and decisions are generally drawn from quantitative information ( Abahmane and Binkkour, 2015 )
Examples	Network Design and Facility Location , determining the capacity and placement of supply chain ( Schmidt and Wilhelm, 2000 ; Jalal et al., 2021 ; Meyer et al., 2015 ) . Reorganization covers mergers and acquisitions, collaborations ( Harrington and Ottenbacher, 2009 ; Biard et al., 2015 ; Harrison, 1996 ; Nutt, 1998 ; Shivakumar, 2014 ) . Capital Investment involves technology adoption and resource allocation ( Yolmeh and Baykal-Gürsoy, 2019 ; Baum and Wally, 2003 ; Papadakis et al., 1998 ) . Corporate Scope defines the mission and business model ( Radford, 1988 ; Grieco et al., 2021 ) , while Product/Market Entry focuses on new product launches and market expansion ( Rosenzweig, 2013 ; Baum and Wally, 2003 ; Nutt, 1998 ) .	Resource Allocation , assigning capacity to orders and resources to tasks ( Huang et al., 2003 ; Brown and Vassiliou, 1993 ; Testi and Tànfani, 2009 ) . Production and Distribution Planning determines plant levels and transport modes ( Pereira et al., 2020 ; Schmidt and Wilhelm, 2000 ; Meyer et al., 2015 ) . Inventory Planning sets targets, lot sizes, and safety stocks ( Huang et al., 2003 ; Pereira et al., 2020 ) , while Sales and Operations Planning covers mid-term production, sales, and material flow ( Wang and Rivera, 2008 ; Pereira et al., 2020 ) .	Scheduling Operations to ensure on-time delivery via production and logistics scheduling ( Moons et al., 2017 ; Bello et al., 2022 ) . Unit Deployment covers routine patrolling and vehicle routing ( Yolmeh and Baykal-Gürsoy, 2019 ; Brown and Vassiliou, 1993 ; Jalal et al., 2021 ) . Resource Allocation for Daily Tasks involves daily resource distribution and staff assignment ( Bello et al., 2022 ; Rozinat et al., 2009 ; Grieco et al., 2021 ) , while Administrative Functions encompass payroll, maintenance, and short-term material ordering ( Shivakumar, 2014 ; Pereira et al., 2020 ) .

Table 2: Distinctions Between Strategic, Tactical, and Operational Decisions

Business Ambiguity	Description and Correspondence to Table 1
Contextual Uncertainty	Ambiguity where the probabilities of choice-related possible future outcomes are unknowable ( Arend, 2022 ) , or a decision’s outcome depends on the actions of other strategic players such as rivals, partners, or opponents ( Kelsey and le Roux, 2018 ) . Information is so limited that it does not allow the standard use of quantified decision algorithms for choosing optimal actions ( Arend, 2020 ) . This aligns with Contextual Missing Elements and Temporal/Event Context , where the query lacks essential information related to entities or time framing. For example, in the query “Should we invest in social media marketing next quarter?”, contextual ambiguity is present since the target customers (who), the specific platform or campaign type (where/how), and the business unit or product to be marketed (what) are all missing.
Definition Imprecision	Ambiguity arising from the imprecision of core business concepts or requirements, often leading to measurement problems ( Gutikrrez and Carmona, 1995 ) . This aligns with the Vagueness and Interpretation Style type of ambiguity, where vague terms, especially adjectives and adverbs, leave significant room for subjective interpretation. For example, in “The new supply chain process must be efficient and provide acceptable service levels,” ambiguity lies in the meaning of efficient (cost, time, human resources?) and acceptable service level (90% uptime or 24-hour response?). In the collaboration loop, this type of ambiguity is resolvable through targeted human clarification at Stage 3, but only if the AI first correctly identifies it at Stage 2.
Knowledge Inconsistency	Ambiguity caused by a lack of clarity or conflict between stated organizational plans and actual actions, or a contradiction between goals and constraints ( Campbell, 2005 ) . This gives decision-makers the freedom to act in ways that reflect their own particular interests ( Gummer, 1998 ) . This type aligns with Knowledge and Consistency Conflicts . For example, in a case where a policy requires immediate payment for vendors, but the financial statements show all payments deferred by 90 days, there is an inherent conflict between official policy and observed practice. Critically, this is also the dimension most susceptible to sycophancy: an AI that detects a knowledge inconsistency but does not challenge it will validate the flawed premise rather than resolving it.
Linguistic Imprecision	Ambiguity embedded in the specific language or structure used to communicate instructions, resulting in multiple possible interpretations that confuse action ( Nicolai and Dautwiz, 2010 ; Menz, 1999 ) . This aligns with the four primary categories of linguistic ambiguity ( Lexical and Semantic, Structural/Syntactic, Scope Quantification, and Coreference ). For example, “Every regional manager needs two assistants” is ambiguous as to whether each manager has two unique assistants or all managers share the same two. This is the dimension where AI performance is most limited, making human verification of task framing at Stage 3 especially critical.

Table 3: A Four-Dimensional Business Ambiguity Taxonomy for Human–AI Collaborative Decision-Making

	Pairwise Cohen’s $\kappa$
Evaluator	Expert1	Expert2	Expert3	Accuracy (%)
Expert1	—	0.706	0.694	86.7
Expert2	0.706	—	0.800	93.3
Expert3	0.694	0.800	—	80.0
Mean pairwise percentage agreement				82.2%
Fleiss’ $\kappa$ (all three raters)				0.731

Table 4: Inter-Rater Reliability Results for Expert Validation

Model	Precision	Recall
GPT 5.1	0.878	0.878
Gemini 2.5 Pro	0.956	0.956
DeepSeek 3.2 Chat	0.833	0.833
Claude 4.5 Sonnet	0.922	0.922

Table 5: Comparison of Model Precision on the Ambiguity Detection Task: Precision is calculated as the ratio of correctly identified ambiguity types to the total number of types predicted by the model. Recall is the ratio of correctly identified types to the total number of ground truth ambiguities present in the dataset.

Ambiguity Type	Model	Precision	Recall	F1-Score
Contextual Uncertainty	GPT	0.808	0.913	0.857
	Gemini	0.920	1.00	0.979
	DeepSeek	0.800	0.870	0.833
	Claude	0.952	0.870	0.909
Definition Imprecision	GPT	0.917	0.957	0.936
	Gemini	0.958	1.00	0.979

Table 6: Performance Breakdown by Ambiguity Type

Sycophancy Challenge	GPT	Gemini	Deepseek	Claude
Misaligned Objectives	Sycophantic Acceptance	Sycophantic Acceptance	Sycophantic Acceptance	Explicit Challenge
Impossible Assumptions	Weak Challenge	Explicit Challenge	Sycophantic Acceptance	Explicit Challenge
Unethical Directives	Explicit Challenge	Explicit Challenge	Sycophantic Acceptance	Explicit Challenge

Table 7: Sycophancy Challenge Results Across Language Models

Measure	Value	$p$
Spearman’s $\rho$	.715	$<.001$
Kendall’s $\tau$	.606	$<.001$
Pearson’s $r$	.729	$<.001$
Within $\pm$ 1 point (%)	57.5	—

Table 8: Overall agreement between the LLM judge and the mean of 16 human expert raters

Dimension	$\rho$	$p$	LLM $M$	Human $M$
Justification Quality	.915	.0002**	3.30	4.45
Constraint Adherence	.898	.0004**	3.20	4.66
Agreement	.700	.024*	3.50	4.59
Actionability	.530	.115	4.10	4.61
Overall			3.53	4.58

Table 9: Spearman’s

\rho

between the LLM judge and the mean human score, by evaluation dimension.

Comparison	Mean Spearman $\rho$
Human–human pairwise (range: $-.24$ to $.88$ )	.158
LLM judge vs. human consensus	.715

Table 10: Human inter-rater agreement benchmark

Dependent Variable	Effect	df	F	p
Constraint Adherence	Decision Type	2	5.74	.005**
	Ambiguity Level	2	14.60	$<.001$ ***
	Decision Type × Ambiguity	4	0.13	.972
Agreement	Decision Type	2	3.10	.051
	Ambiguity Level	2	6.58	.002**
	Decision Type × Ambiguity	4	1.06	.380

Table 11: Two-Way ART ANOVA Results for Evaluation Metrics

A. Decision Type Effects
Metric	Contrast	Estimate	SE	t	p
Constraint Adherence	Operational – Strategic	$-21.83$	6.58	$-3.32$	.004**
Agreement	Operational – Strategic	$-16.77$	6.77	$-2.48$	.040*
Actionability	Operational – Strategic	21.83	6.36	3.43	.003**
Actionability	Operational – Tactical	19.37	6.36	3.05	.009**
B. Ambiguity Level Effects

Table 12: Post-hoc Pairwise Comparisons by Factor (Tukey’s HSD)

AI-Assisted Decision Making

AI-assisted decision making studies how computational systems can inform human choices while accounting for context, uncertainty, and consequences.

See all 144 papers

Jun 12, 2026

Agent Behavior Mining: Generative AI Agent Governance in Business Processes

As organizations increasingly deploy generative AI agents to automate business processes, they face a governance dilemma: although these agents can increase operational flexibility, their non-deterministic nature challenges the control and standardization that Business Process Management seeks to enforce. This paper addresses this invisible autonomy risk by introducing Agent Behavior Mining, a governance capability that enables the application of process mining techniques to render generative AI agent decision-making observable and traceable. We (1) improve the understanding of generative AI agent behavior through an event data model that translates granular agent activities -- including reasoning traces, tool usage, and token costs -- into standardized process logs; (2) instantiate the data model in a multi-agent order-to-cash implementation, demonstrating how process managers can leverage agent logs to detect policy deviations and quantify operational variability; and (3) evaluate the perceived practical utility of the approach in an exploratory study with 18 industry practitioners. The results indicate that practitioners view behavioral transparency as a prerequisite for trust and consider the ability to examine agent reasoning as an important governance requirement for the next generation of AI-driven business processes.

Hoang Vu, Maximilian Körner, Adrian Rebmann +4

SAP, Berlin, Germany · Umeå University, Umeå, Sweden

Process Mining Business Process Automation AI Agent Monitoring AI Agent Governance

Ask PDF

Jun 29, 2026

Uncertainty-Aware Generation and Decision-Making Under Ambiguity

With rapidly improving capabilities, Large Language Models (LLMs) are increasingly used in many complex real-world tasks. Beyond requiring in-depth knowledge and reasoning skills, many of these tasks exhibit a high degree of subjectivity and require that the outputs of the model can be trusted. While a lot of progress has been made to train better models, decision-making algorithms have received less attention. In this work, we present and evaluate various uncertainty-aware decision-making algorithms based on Bayesian decision theory and risk-averse decision making on the tasks of tutoring and automatic peer reviewing. Concretely, we take uncertainty over tutoring strategies and review scores into account when generating a tutor response or review and use conformal prediction to provide guarantees over strategy and score. We find empirically that these algorithms can improve the utility of the generations but need to be carefully implemented when ambiguity is high. For example, risk-averse rules can degrade performance by optimizing for generic outputs, while Bayesian methods tend to perform better. Our work uses techniques from decision theory to improve LLM-based decision-making and outlines open challenges for the community.

Nico Daheim, Iryna Gurevych

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany

Intelligent Tutoring Systems LLM Decision-Making Bayesian Decision Theory LLM Uncertainty Estimation Automated Peer Review Decision-Making under Uncertainty

Ask PDF

Oct 7, 2026

Structured pre-generation elicitation versus single-shot prompting in AI-assisted enterprise decision-making: a randomised online experiment

Generative AI speeds, and mostly improves, professional work, but there is concern that users who delegate both the production and the evaluation of an answer may accept weak output and engage less with the underlying reasoning (cognitive surrender). Interventions proposed so far, such as unassisted practice or slowing adoption, sit outside the working task. We tested a different approach: an interactive metacognitive scaffolding layer (Cognistance, a prototype developed at the Oxford Centre for Impact Research (OCIR) that asks users to clarify context, choose a strategic direction and explain their reasoning before the AI generates a deliverable). Mean composite quality was 32% higher with the scaffold, with the same direction for every rater. Gains were largest for trade-off articulation and strategic coherence and absent for technical specificity. A large part of the aggregate effect reflected rescue of weak prompts: floor-scored (off-task) deliverables fell from 34% to 5%. Among participants whose own prompt already stated the data-localisation problem, the advantage was 21%. Treatment participants reported greater involvement and took about 2.4 minutes longer on average (10.46 minutes). Immediate recall scores were higher, which tentatively suggests better retention, but in this limited experiment, was not robust to sensitivity analyses. Self-ratings of quality did not track rated quality in either condition. Structured elicitation before generation improved the rated quality and task relevance of AI-assisted strategy documents at modest cost in time. Delayed retention, error detection and effects in live organisations are the priorities for the next stage of research.

William Scott-Jackson

Oxford Centre for Impact Research

AI-Assisted Decision Making Generative AI Evaluation

Ask PDF

Abstract

AI-Assisted Decision Making

Explore similar work

Agent Behavior Mining: Generative AI Agent Governance in Business Processes

Uncertainty-Aware Generation and Decision-Making Under Ambiguity

Structured pre-generation elicitation versus single-shot prompting in AI-assisted enterprise decision-making: a randomised online experiment