Human and LLM Facilitation in Collaborative Design Meetings: A Computational Behavioral Comparison
Authors: Mohammed Azeez Khan, Aaron Dsouza, Ashutosh Mishra, Ummul Faiz Z. Bibi, Neha K. Nair, Amar Kumar Behera
Organizations: Department of Computer Science and Engineering, National Institute of Technology Warangal, Warangal, India · Department of Electronics and Communication Engineering, National Institute of Technology Warangal, Warangal, India · Department of Computer Science and Engineering, Nawab Shah Alam Khan College of Engineering and Technology, Hyderabad, India · Department of Physics, National Institute of Technology Warangal, Warangal, India · Department of Design, Indian Institute of Technology Kanpur, Kanpur, India
Large language models (LLMs) are increasingly being explored as facilitators of group discussion and collaborative work, but the behavioral differences between LLM-generated and human facilitation remain insufficiently characterized. We compare human Project Manager facilitation turns from scenario-based collaborative design meetings with Llama 3.1 8B Instruct responses across 199 paired contexts from the AMI Meeting Corpus. The analysis covers five confirmatory dimensions: contextual semantic distance, a lexical elaboration proxy, empathyand-hedge lexical density, topic divergence, and phase-vocabulary alignment. An exploratory directiveness measure failed domain validation and is not interpreted. The sampled human Project Manager turns were higher on contextual semantic distance (r = 0.644) and on empathyand-hedge lexical density (r = 0.203). The second contrast is carried mainly by hedge terms, so we read it as lexical hedging rather than psychological empathy. Llama-generated facilitation was higher on topic divergence (r = 0.703), the lexical elaboration proxy (r = 0.288; lengthconfounded), and instruction-conditioned phase-vocabulary alignment (r = 0.382). A secondary no-phase check left the Llama-higher phase contrast in place. The system prompt still requested a facilitation move, so that check is not evidence of spontaneous phase awareness. These differences should not be interpreted as a ranking of facilitation quality. The resulting behavioral profile provides a reference point for later comparisons of LLM facilitators, human facilitators, prompts, models, and collaborative settings.
Figures & tables
Figure 1: Primary effect sizes at T=0.7 ( n=199 pairs). Points are the paired effect size r , signed so that negative values mean the human turn is higher and positive values mean the Llama response is higher. Intervals are the 95% bootstrap confidence intervals on r , given the same sign. Directiveness is omitted because it is unvalidated.
Dimension
Observed difference
What it supports
What it does not establish
Contextual semantic distance
Human higher
Greater distance in all-MiniLM-L6-v2 space
General semantic superiority
Lexical elaboration
Llama higher
Greater surface lexical elaboration
Conceptual specificity
Empathy/hedging
Human higher
A lexical hedge/empathy difference
Psychological empathy
Topic divergence
Llama higher
Greater LDA topic-distribution divergence
Creativity
Phase alignment
Llama higher
Greater phase-vocabulary alignment
Spontaneous phase awareness
Table 1: What the five confirmatory contrasts support. Directions are human versus Llama 3.1 8B Instruct at T=0.7 .
Source
N
Words
SD
Types
Sents
Human PM
199
23.5
12.7
725
1.5
LLM T=0.3
199
40.5
7.9
1031
1.8
LLM T=0.7
199
38.5
7.5
1125
1.6
LLM T=1.0
199
37.6
8.5
1196
1.5
Table 2: Descriptive statistics for human PM turns and Llama 3.1 8B Instruct responses. Types are unique lowercased word tokens. Mean sentences count terminal punctuation.
Dimension
H Mdn
L Mdn
r
95% CI
pBonf
Sig.
Contextual distance
0.742
0.593
0.644
[0.56, 0.72]
< 0.001
Lexical elaboration
0.397
0.416
0.288
[0.16, 0.40]
< 0.001
Empathy/hedge density
0.000
0.000
0.203
[0.07, 0.34]
0.021
Topic divergence
0.540
0.744
0.703
[0.63, 0.76]
< 0.001
Phase-vocabulary alignment a
0.000
0.000
0.382
[0.25, 0.50]
< 0.001
Directiveness (unvalidated) b
0.021
0.036
0.026
[0.00, 0.17]
0.718
ns
Table 3: Paired Wilcoxon signed-rank tests: human PM turns versus Llama 3.1 8B Instruct ( T=0.7 , n=199 pairs). pBonf=min(1,5praw) for the five confirmatory dimensions and is compared with 0.05 (equivalently, raw p with 0.010 ). Effect size r=∣Z∣/n . The 95% confidence interval is a bootstrap with 2,000 resamples.
Figure 2: Score distributions for human facilitation and Llama 3.1 8B Instruct at T=0.7 . Horizontal lines are medians. Hedge/Empathy is lexical density, not a psychological rating. Directiveness is exploratory and unvalidated and is not a confirmatory dimension.
Figure 3: Phase-vocabulary means at T=0.7 . Removing the explicit phase label did not eliminate the Llama-higher contrast. The label-present versus no-label difference was not statistically significant after correction. The system prompt still requested a design-facilitation move.
Contrast
Means
r
95% CI
Raw p
Sec. p
Human vs. no label
0.0050 vs. 0.0115
0.294
[0.16, 0.42]
< 0.001
< 0.001
Label vs. no label
0.0149 vs. 0.0115
0.123
[0.01, 0.26]
0.082
0.163
Table 4: Secondary phase-label sensitivity at T=0.7 ( n=199 matched contexts). The two raw p -values are Bonferroni-adjusted together. This family is separate from the five confirmatory dimensions. Effect size r=∣Z∣/n , with the same 2,000-resample bootstrap as the primary tests.
Dimension
χ2
p
pBonf
Sig.
Contextual distance
0.44
0.802
1.000
—
Lexical elaboration
4.04
0.133
0.664
—
Empathy/hedge density
2.21
0.332
1.000
—
Topic divergence
5.46
0.065
0.327
—
Phase-vocabulary alignment
5.61
0.060
0.302
—
Directiveness (exploratory)
0.85
0.654
0.654
—
Table 5: Friedman tests of scores across T∈{0.3,0.7,1.0} within the same 199 contexts. Bonferroni correction uses the five confirmatory dimensions.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Spearman correlations among the six scores ( N=796 ), including the unvalidated directiveness score.
Figure 5: Exploratory PCA of behavioral scores for human turns and Llama responses at all three temperatures, fit on all 796 turns. The axes are principal components of the score matrix, not a validated style space.
Figure 6: Descriptive Spearman correlations between decoding temperature and scores on 597 stacked responses. These are not the within-context Friedman tests.
Figure 7: Mean scores for human facilitation and Llama responses at T=0.7 . Each panel starts at zero and uses its own scale. Printed numbers are the raw means. Hedge/Empathy is lexical density. Directiveness is exploratory and unvalidated and is not a confirmatory dimension.
Dimension
Prim. r
Len. r
Part. ρ
ME β
pBonf
Note
Contextual distance
0.644
0.514
− 0.137
− 0.143
< 0.001
Stable
Topic divergence
0.703
0.712
− 0.027
+0.199
< 0.001
Stable
Lexical elaboration
0.288
0.807
− 0.596
+0.017
< 0.001
Length-confounded
Phase alignment a
0.382
0.487
− 0.059
+0.010
< 0.001
Instruction-conditioned
Empathy/hedge density
0.203
0.231
+0.068
− 0.011
0.021
Hedge proxy
Directiveness (exploratory)
0.026
—
—
+0.005
0.718
Unvalidated
Appendix
Table 6: Robustness checks: length-matched pairs ( n=73 ), Spearman correlation of the paired difference with human word count, and a mixed-effects estimate with a random intercept for context. Primary r is from the paired Wilcoxon test ( n=199 ).
LLM agents are increasingly used for collaborative problem solving and human-group simulation. This makes outcome-only evaluation insufficient: if LLM groups are used as models of human groups, we need to know whether they succeed or fail through human-like deliberative mechanisms. We compare human group chats with matched LLM deliberation traces on Wason-style deductive reasoning, then test whether the same process signatures generalize to analogical, abductive, and analytical tasks. Humans and LLMs show the same assembly bonus asymmetry: discussion improves the average member more often than the best initial member. Initial-answer diversity accounts for the effect of model heterogeneity, increasing movement in both corrective and destructive directions. The main differences are process-level. Compared with humans, LLM groups follow majorities more often, surface less unique information, and converge earlier; correct minority signals succeed mainly when re-expressed early. Interventions motivated by human group-decision research yield modest improvements in collective outcomes, but do not remove the coordination bottleneck. Together, these results suggest that LLM groups can reproduce some outcome-level patterns of human deliberation while diverging in the mechanisms that generate assembly bonus and process loss, with implications for group simulation and human-AI collaboration.
Ala N. Tak, Teruhisa Misu, Kumar Akash +3
USC Institute for Creative Technologies, Playa Vista, California, USA · Honda Research Institute USA, Inc., San Jose, California, USA
When multiple large language models interact in a shared conversation, do they develop differentiated social roles or converge toward uniform behavior? We present a controlled experimental platform that orchestrates simultaneous multi-agent discussions among 7 heterogeneous LLMs on a unified inference backend, systematically varying group composition, naming conventions, and prompt structure across 12 experimental series (208 runs, 13,786 coded messages). Each message is independently coded on six behavioral flags by two LLM judges from distinct model families (Gemini 3.1 Pro and Claude Sonnet 4.6), achieving mean Cohen's kappa = 0.78 with conservative intersection-based adjudication. Human validation on 609 randomly stratified messages confirmed coding reliability (mean kappa = 0.73 vs. Gemini). We find that (1) heterogeneous groups exhibit significantly richer behavioral differentiation than homogeneous groups (cosine similarity 0.56 vs. 0.85; p < 10^-5, r = 0.70); (2) groups spontaneously exhibit compensatory response patterns when an agent crashes; (3) revealing real model names significantly increases behavioral convergence (cosine 0.56 to 0.77, p = 0.001); and (4) removing all prompt scaffolding converges profiles to homogeneous-level similarity (p < 0.001). Critically, these behaviors are absent when agents operate in isolation, confirming that behavioral diversity is a structured, reproducible phenomenon driven by the interaction of architectural heterogeneity, group context, and prompt-level scaffolding.
This study examines whether LLM-generated replies remain semantically consistent when the underlying LLM changes. Using messages from real collaborative conversations, we compared the semantic similarity of generated replies across LLMs under two conditions: with and without preceding chat history. Results show that model choice and conversational context both affect response similarity and alignment with human replies. These findings indicate that prompting and conversational context alone may not be sufficient to preserve response consistency across LLMs, highlighting the need for infrastructure and design strategies that can maintain stable and comparable responses amid the rapid and continuous evolution of LLMs.