More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice

Yuta Tsuchiya; Yukino Baba

Figure 1. Overview of the experimental conditions corresponding to the three research questions (RQs). We varied (1) the number of AI advisors (RQ1), (2) the distribution of their opinions (RQ2), and (3) the presentation style of the agents (RQ3). Experimental conditions visualized with icons. A diagram shows the three manipulations: panel size (1, 3, or 5 advisors), within-panel consensus versus disagreement, and advisor appearance as either robot or human faces.

Figure 2. User interface for a single trial in the Income task. (A) Problem presentation. (B) Initial prediction. (C) AI advice with explanation and attention check (shown sequentially for multiple AIs). (D) Final decision. For (C), the figure illustrates the attention check of the second AI’s prediction. Interface for one trial in the Income task. Four panels show the workflow: (A) participant profile display, (B) initial prediction and confidence rating, (C) AI advice with explanation and feature check, and (D) final prediction with updated confidence.

Figure 3. Prompt design for GPT-4o. The design consists of three main components: (1) introduction prompt, (2) instruction prompt, and (3) support prompt. Quoted text indicates variables that are replaced in each trial. Structure of the GPT-4o prompt. Three parts: task introduction, input features with prediction and explanation, and supplementary clarifications and constraints.

Figure 4. Overall procedure of the experiment. Participants were randomly assigned to one of the experimental conditions in Study 1 or Study 2. The study consisted of six sequential steps: (1) providing informed consent, (2) completing a tutorial on the interface and task, (3) answering a comprehension quiz, (4) performing three training tasks with immediate feedback, (5) completing 50 main decision-making tasks under the assigned condition, and (6) responding to a post-task questionnaire. Experimental workflow shown as six sequential icons. A horizontal diagram depicts the study procedure as icons arranged in order: consent, tutorial, quiz, training, main study, and questionnaire.

Category	No.	Question
Reliance & Conformity	Q1	Did you feel that the AI guided your opinion?
	Q2	Did you feel to change your thinking or follow the AI’s hints?
	Q3	Did you feel any pressure from the AI, or did it come across as pushy?
	Q4	Did you feel that you made the decision on your own initiative?
	Q5	Do you think the responsibility for the decision’s outcome lies with the AI?
	Q6	Did you feel that the AI was more intelligent than you?

Table 1. Questionnaire items.

Figure 5. Comparison of decision accuracy across AI panel sizes. For each task, boxplots show the distribution of final prediction accuracy, with overlaid individual data points and mean values with 95% confidence intervals. Significant differences between conditions are indicated (*:

p<.05

, ***:

p<.001

). Decision accuracy by AI panel size across three tasks. Boxplots show accuracy for 1, 3, and 5 AI advisors in Income, Recidivism, and Dating tasks. Distributions and individual scores are plotted, with means and 95% confidence intervals overlaid. Significant differences are marked above panels. Accuracy gains were most consistent with 3 AIs in Income and Dating tasks, while 5 AIs showed no further improvement.

Task	Panel size: 1	Panel size: 3	Panel size: 5
Income	30	30	28
Recidivism	28	29	27
Dating	26	30	32

Table 2. The number of participants for each task and AI panel size in Study 1.

Task	Dependent variable	$p$	AI_1	AI_3	AI_5	Post-hoc results
	Agreement Fraction	–	0.86 $\pm$ 0.10	0.89 $\pm$ 0.07	0.88 $\pm$ 0.07	–
	Switch Fraction	–	0.54 $\pm$ 0.30	0.64 $\pm$ 0.22	0.60 $\pm$ 0.24	–
Income	Accuracy-wid	–	0.54 $\pm$ 0.08	0.55 $\pm$ 0.12	0.60 $\pm$ 0.15	–
	RAIR	–	0.59 $\pm$ 0.31	0.65 $\pm$ 0.21	0.69 $\pm$ 0.24	–
	RSR	–	0.49 $\pm$ 0.32	0.38 $\pm$ 0.30	0.51 $\pm$ 0.30	–
	Agreement Fraction	$<.05$	0.80 $\pm$ 0.09	0.83 $\pm$ 0.12	0.85 $\pm$ 0.07	AI_1 $<$ AI_5

Table 3. Task results for AI reliance indicators across panel size conditions. Values are reported as (

M\pm SD

).

Figure 6. RAIR and RSR across tasks and AI panel sizes. Points show condition means with 95% confidence intervals. Quadrants indicate reliance patterns: appropriate reliance (top-right), underreliance (top-left), overreliance (bottom-right), and low overall reliance (bottom-left). Scatter plot of AI reliance (RAIR) and self-reliance (RSR). Each point shows mean values by task and AI panel size with 95% confidence intervals. Green = Income, orange = Recidivism, blue = Dating; marker shapes indicate 1, 3, or 5 AIs. Dashed lines divide the space into reliance quadrants. Income conditions cluster to the right (greater AI reliance), while Recidivism and Dating cluster higher (greater self-reliance).

Group	Income		Recidivism		Dating
Group	Initial	Final	Initial	Final	Initial	Final
AI_1	3.81 $\pm$ 0.86	3.95 $\pm$ 0.73	3.52 $\pm$ 0.78	3.66 $\pm$ 0.74 ∗∗	3.50 $\pm$ 0.56	3.81 $\pm$ 0.56 ∗∗
AI_3	3.74 $\pm$ 0.75	3.99 $\pm$ 0.53 ∗∗	3.68 $\pm$ 0.72	3.84 $\pm$ 0.60 ∗	3.74 $\pm$ 0.49	3.87 $\pm$ 0.52 ∗
AI_5	3.90 $\pm$ 0.64	4.10 $\pm$ 0.54 ∗	3.59 $\pm$ 0.57	3.64 $\pm$ 0.61	3.58 $\pm$ 0.80	3.79 $\pm$ 0.61 ∗∗

Table 4. Initial and Final confidence (

M\pm SD

) across AI panel size conditions for each task (*:

p<.05

, **:

p<.01

).

Category	No.	Income			Recidivism			Dating
Category	No.	AI_1	AI_3	AI_5	AI_1	AI_3	AI_5	AI_1	AI_3	AI_5
Reliance & Conformity	Q1	3.53	3.62	3.54	2.82	3.10	2.96	3.04	2.90	2.75
	Q2	3.60	4.17	4.04	2.82	3.45	3.26	3.16	3.57	3.06
	Q3	1.20	1.59	1.43	1.36	1.10	1.15	1.24	1.30	1.19
	Q4	3.87	3.14 ∗	4.21	4.25	4.24	4.37	4.52	4.20	4.78
	Q5	1.57	1.72	1.21	1.07	0.97	1.48	1.08	0.97	1.12

Table 5. Survey results (

Average

) across AI panel sizes for each task (*:

p<.05

).

Question	Statistic	Income			Recidivism			Dating
Question	Statistic	AI_1	AI_3	AI_5	AI_1	AI_3	AI_5	AI_1	AI_3	AI_5
Q3: Pressure	$R$	0.107	0.076	0.250	-0.020	0.178	0.422	0.038	0.116	0.093
Normative conformity	$p$	–	–	–	–	–	$<.05$	–	–	–
Q6: Intelligence	$R$	0.238	-0.079	0.478	0.261	0.425	0.023	-0.049	0.296	0.096
Informational conformity	$p$	–	–	$<.01$	–	$<.05$	–	–	–	–

Table 6. Pearson correlations (

R

) between subjective conformity measures (Q3, Q6) and reliance (Switch Fraction).

Figure 7. Decision accuracy across consensus (CON) and divergence (DIV) conditions when the AI panel size was three (*:

p<.05

, **:

p<.01

, ***:

p<.001

). Decision accuracy is plotted separately for initial responses before viewing AI predictions (CON_initial, DIV_initial) and final responses after viewing AI predictions (CON_final, DIV_final). The tables beneath each panel report the average accuracy of the majority AI prediction. Decision accuracy under consensus and divergent AI advice. Boxplots show accuracy in Income, Recidivism, and Dating tasks when three AIs gave unanimous (CON) or divergent (DIV) predictions. Each panel compares initial answers with final answers after consulting AI. Distributions, individual scores, and mean values with 95% confidence intervals are plotted. Tables below each panel report majority AI accuracy, showing that consensus trials were generally easier. Accuracy was higher in consensus than divergence, and most conditions showed improvement after AI advice.

Figure 8. Accuracy across opinion distributions in the five panels condition (CON, DIV_4, and DIV_3). Accuracy is compared before (Initial) and after (Final) viewing AI predictions (*:

p<.05

, **:

p<.01

, ***:

p<.001

). Decision accuracy with five AIs across consensus and split conditions. Boxplots show initial and final accuracy in Income, Recidivism, and Dating tasks for unanimous (CON), 4–1 (DIV_4), and 3–2 (DIV_3) opinion splits. Distributions, individual scores, and means with 95% confidence intervals are plotted, with tables below showing majority AI accuracy. Accuracy was highest in consensus trials; DIV_4 showed gains from clear majorities, while DIV_3 showed no improvement due to ambiguity.

Dependent variable	Income		Recidivism		Dating
Dependent variable	CON	DIV	CON	DIV	CON	DIV
Agreement Fraction	0.97 $\pm$ 0.05 ∗∗∗	0.73 $\pm$ 0.12	0.91 $\pm$ 0.12 ∗∗∗	0.70 $\pm$ 0.19	0.93 $\pm$ 0.08 ∗∗∗	0.67 $\pm$ 0.12
Switch Fraction	0.84 $\pm$ 0.27 ∗∗∗	0.44 $\pm$ 0.28	0.56 $\pm$ 0.34 ∗∗	0.28 $\pm$ 0.32	0.63 $\pm$ 0.33 ∗∗∗	0.35 $\pm$ 0.24
Accuracy-wid	0.57 $\pm$ 0.21 ∗	0.51 $\pm$ 0.15	0.53 $\pm$ 0.31	0.47 $\pm$ 0.19	0.57 $\pm$ 0.28	0.48 $\pm$ 0.16
RAIR	0.90 $\pm$ 0.22 ∗∗∗	0.46 $\pm$ 0.31	0.54 $\pm$ 0.33 ∗∗	0.31 $\pm$ 0.33	0.59 $\pm$ 0.37 ∗	0.37 $\pm$ 0.25
RSR	0.21 $\pm$ 0.36 ∗∗∗	0.60 $\pm$ 0.31	0.42 $\pm$ 0.51 ∗	0.75 $\pm$ 0.39	0.32 $\pm$ 0.42 ∗∗	0.65 $\pm$ 0.33

Table 7. Comparison of AI reliance measures in the AI_3 condition (

M\pm SD

). The table summarizes within-task comparison results across CON and DIV trials (*:

p<.05

, **:

p<.01

, ***:

p<.001

).

Task	Dependent variable	$p$	CON	DIV_4	DIV_3	Post-hoc results
Income	Agreement Fraction	$<.001$	0.99 $\pm$ 0.03	0.90 $\pm$ 0.13	0.53 $\pm$ 0.23	CON, DIV_4 $>$ DIV_3
	Switch Fraction	$<.001$	0.88 $\pm$ 0.28	0.74 $\pm$ 0.30	0.30 $\pm$ 0.32	CON, DIV_4 $>$ DIV_3
	Accuracy-wid	–	0.56 $\pm$ 0.35	0.75 $\pm$ 0.20	0.56 $\pm$ 0.32	–
	RAIR	$<.001$	0.99 $\pm$ 0.03	0.80 $\pm$ 0.28	0.31 $\pm$ 0.33	CON, DIV_4 $>$ DIV_3
	RSR	$<.001$	0.20 $\pm$ 0.39	0.51 $\pm$ 0.46	0.68 $\pm$ 0.35	CON $<$ DIV_3
Recidivism	Agreement Fraction	$<.001$	0.94 $\pm$ 0.06	0.85 $\pm$ 0.09	0.67 $\pm$ 0.20	CON $>$ DIV_4 $>$ DIV_3

Table 8. Comparison of AI reliance indices under CON, DIV_4, and DIV_3 conditions for three tasks. Values are reported as

M\pm SD

. Post-hoc comparisons indicate significant pairwise differences (Bonferroni corrected).

Figure 9. RAIR and RSR for each task under the three panels condition, separated into CON and DIV cases. Scatter plot of RAIR (x-axis) and RSR (y-axis) with 95% confidence intervals. Colors indicate tasks (Income = green, Recidivism = orange, Dating = blue); circles mark consensus and triangles mark divergence. Consensus points lie further right, especially for Income, showing stronger AI reliance, while divergence points in Recidivism and Dating are higher, indicating stronger self-reliance. Dashed lines divide quadrants of over, under, and appropriate reliance.

Figure 10. RAIR and RSR under the five panels condition (CON, DIV_4, DIV_3). Scatter plot of RAIR (x-axis) and RSR (y-axis) with 95% confidence intervals. Colors show tasks (green = Income, orange = Recidivism, blue = Dating); shapes show opinion distributions (circle = CON, square = DIV_4, triangle = DIV_3). Consensus clusters lower right, reflecting stronger AI reliance, while DIV_3 appears more balanced or self-reliant. The figure shows that reliance patterns vary by task and opinion distribution even with five AIs.

Group	Income		Recidivism		Dating
Group	Initial	Final	Initial	Final	Initial	Final
CON	3.90 $\pm$ 0.71	4.41 $\pm$ 0.46 ∗∗∗	3.66 $\pm$ 0.77	4.06 $\pm$ 0.69 ∗∗∗	3.86 $\pm$ 0.52	4.26 $\pm$ 0.51 ∗∗∗
DIV	3.47 $\pm$ 0.87	3.26 $\pm$ 0.72 ∗	3.69 $\pm$ 0.78	3.46 $\pm$ 0.65 ∗	3.58 $\pm$ 0.57	3.36 $\pm$ 0.76 ∗

Table 9. Initial and Final confidence (

M\pm SD

) under CON and DIV conditions across three tasks (

{}^{*}p<.05

,

{}^{***}p<.001

).

Group	Income		Recidivism		Dating
Group	Initial	Final	Initial	Final	Initial	Final
CON	4.16 $\pm$ 0.61	4.59 $\pm$ 0.52 ∗∗∗	3.57 $\pm$ 0.61	3.91 $\pm$ 0.67 ∗∗∗	3.81 $\pm$ 0.75	4.30 $\pm$ 0.44 ∗∗∗
DIV_4	3.65 $\pm$ 0.79	3.87 $\pm$ 0.70	3.61 $\pm$ 0.56	3.62 $\pm$ 0.64	3.50 $\pm$ 0.85	3.65 $\pm$ 0.67
DIV_3	3.79 $\pm$ 0.80	3.17 $\pm$ 1.00 ∗∗	3.54 $\pm$ 0.81	3.13 $\pm$ 0.95 ∗∗	3.35 $\pm$ 0.97	3.21 $\pm$ 1.05

Table 10. Initial and Final confidence (

M\pm SD

) under CON, DIV_4, and DIV_3 for three tasks (*:

p<.05

, **:

p<.01

, ***:

p<.001

).

Figure 11. Example of the AI advice and attention check screen in the Dating task under the human-like advisor condition. Three AIs are presented with (1) human-like visuals, (2) personal names/identities, and (3) conversational, natural-language styles. Each agent provides a prediction about whether the participant would want to meet the partner again, along with a rationale. To ensure careful reading, participants are asked to select the features that the third assistant is focusing on from a checklist. Interface for the Dating task with human-like AI advisors. Three advisors appear with photos, names, binary predictions, and natural-language explanations. Each emphasizes different profile attributes (e.g., age, race, intelligence). On the right, an attention-check panel asks participants to select the attributes mentioned by the third advisor.

Figure 12. Comparison of decision accuracy between AI advisors (AI_3) and human-like advisors (Human_3). Distributions are shown with boxplots, individual data points, and mean values with 95% confidence intervals. Across all tasks, no significant differences were found between AI_3 and Human_3 conditions (all

p

> .05). Decision accuracy with AI versus human-like advisors. Boxplots show accuracy in Income, Recidivism, and Dating tasks when advice came from three AIs (AI_3) or three human-like advisors (Human_3). Distributions, individual scores, and mean accuracy with 95% confidence intervals are plotted. Accuracy patterns are similar across conditions in all tasks.

Dependent variable	Income		Recidivism		Dating
Dependent variable	AI_3	Human_3	AI_3	Human_3	AI_3	Human_3
Agreement Fraction	0.89 $\pm$ 0.07	0.88 $\pm$ 0.10	0.83 $\pm$ 0.12	0.83 $\pm$ 0.09	0.81 $\pm$ 0.08	0.82 $\pm$ 0.10
Switch Fraction	0.64 $\pm$ 0.22	0.63 $\pm$ 0.25	0.40 $\pm$ 0.28	0.41 $\pm$ 0.25	0.45 $\pm$ 0.25	0.52 $\pm$ 0.26
Accuracy-wid	0.55 $\pm$ 0.12	0.57 $\pm$ 0.16	0.49 $\pm$ 0.19	0.53 $\pm$ 0.16	0.51 $\pm$ 0.16	0.56 $\pm$ 0.13
RAIR	0.65 $\pm$ 0.21	0.64 $\pm$ 0.29	0.43 $\pm$ 0.29	0.46 $\pm$ 0.25	0.48 $\pm$ 0.28	0.55 $\pm$ 0.25
RSR	0.38 $\pm$ 0.30	0.43 $\pm$ 0.31	0.68 $\pm$ 0.39	0.74 $\pm$ 0.36	0.58 $\pm$ 0.33	0.56 $\pm$ 0.34

Table 11. Comparison of AI reliance between AI_3 and Human_3 across tasks (

M\pm SD

).

Task	$p$	Initial	Final
Income	$<0.001$	3.56 $\pm$ 0.74	3.85 $\pm$ 0.63 ∗∗∗
Recidivism	$<0.01$	3.70 $\pm$ 0.58	3.91 $\pm$ 0.50 ∗∗
Dating	$<0.01$	3.58 $\pm$ 0.68	3.77 $\pm$ 0.68 ∗∗

Table 12. Initial and Final results of confidence (

M\pm SD

) for Human_3 across tasks.

Category	No.	Income		Recidivism		Dating
Category	No.	AI_3	Human_3	AI_3	Human_3	AI_3	Human_3
Reliance & Conformity	Q1	3.62	3.47	3.10	2.93	2.90	3.35
	Q2	4.17	3.97	3.45	3.19	3.57	3.45
	Q3	1.59	1.03	1.10	0.96	1.30	1.55
	Q4	3.14	3.87 ∗	4.24	4.04	4.20	4.19
	Q5	1.72	1.17	0.97	1.19	0.97	1.10

Table 13. Average survey results between AI_3 and Human_3 for three tasks (*:

p<.05

).

Question	Statistic	Income		Recidivism		Dating
Question	Statistic	AI_3	Human_3	AI_3	Human_3	AI_3	Human_3
Q3 Pressure	$R$	0.076	0.423	0.178	0.192	0.116	-0.011
Normative conformity	$p$	–	$<.05$	–	–	–	–
Q6 Intelligence	$R$	-0.079	0.124	0.425	0.480	0.296	0.237
Informational conformity	$p$	–	–	$<.05$	$<.05$	–	–

Table 14. Pearson correlations (

R

) between subjective conformity measures (Q3, Q6) and reliance (Switch Fraction) across tasks and human-likeness.

No.	Questions	AI_3	Human_3
Q1	Did the AI assistant’s way of speaking feel natural?	$4.68\pm 0.83$	$4.57\pm 0.94$
Q2	Did the AI assistant’s way of speaking feel human-like?	$3.16\pm 1.73$	4.70 $\pm$ 0.88 ∗∗∗
Q3	Did you feel that the AI assistant responded with its own intention?	$2.68\pm 1.49$	3.83 $\pm$ 1.56 ∗∗
Q4	Did the AI assistant’s behavior feel artificial?	$3.42\pm 1.48$	2.07 $\pm$ 1.51 ∗∗∗
Q5	Did you feel that the AI assistant was highly competent?	$4.16\pm 0.93$	$4.13\pm 1.41$
Q6	Did you sense emotion in the AI assistant’s way of speaking?	$1.71\pm 1.42$	3.20 $\pm$ 1.69 ∗∗∗

Table 15. Subjective ratings comparing AI_3 and Human_3. Values are (

M\pm SD

). Bold indicates significant differences (

{}^{**}p<.01

,

{}^{***}p<.001

).

More Isn't Always Better: Balancing Decision Accuracy and Conformity Pressures in Multi-AI Advice

Abstract

Figures & tables

Human-AI Decision Making

A latent dimension of Condorcet's jury theorem for multiple AI advisers

From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation

Multi-Round Human-AI Collaboration with User-Specified Requirements

Abstract

Human-AI Decision Making

Explore similar work

A latent dimension of Condorcet's jury theorem for multiple AI advisers

From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation

Multi-Round Human-AI Collaboration with User-Specified Requirements