Sophistication in GenAI Use: Field Evidence from a Large Firm

Nicholas J. Hallman; Zachary T. Kowaleski; Anu Puvvada; Jaime J. Schmidt

cs.AIAug 27, 2026

Sophistication in GenAI Use: Field Evidence from a Large Firm

Authors: Nicholas J. Hallman, Zachary T. Kowaleski, Anu Puvvada, Jaime J. Schmidt

Organizations: University of Texas at Austin · KPMG LLP

Abstract

We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm. Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses from nearly 4,000 back-office employees across 15 functional areas over eight months in 2025. We document three main findings. First, senior employees exhibit more sophisticated genAI use, consistent with domain expertise complementing genAI capabilities. Second, sophistication varies considerably across functions and is highest in Strategy, Digital Innovation, and Project Management, three groups that share a focus on firmwide strategic initiatives and organizational change. Third, we observe neither improvements in sophistication over time nor lasting improvements following formal AI training, suggesting that sophisticated use can be difficult to change. Together, our study provides measures of and insights into sophisticated genAI use that managers can use to improve outcomes and that researchers can use in future research.

Figures & tables

Figure 1: Usage and Intensity

Figure 2: Usage and Intensity (continued)

Figure 3: Use Cases by Group

Figure 4: Use Cases by Group (continued)

Figure 5: Evaluation of Use

Figure 6: Evaluation of Use (continued)

Figure 7: Bivariate Associations

Figure 8: Bivariate Associations (continued)

Metric	Count
Total back-office employees	5,191
Employees with aIQ Chat use	3,925
Employees in Copilot dataset	4,392
Employees with Copilot use	3,962
Employees who used BOTH tools	2,861

Table 1: Sample Construction and Scale

Category	N	Pct of Convs
WRITING	116,216	0.733
KNOWLEDGE	36,458	0.230
TEXT ANALYSIS	17,008	0.107
TOOL GUIDE	15,634	0.099
CODING DATA	11,556	0.073
PERSONAL	3,026	0.019

Table 2: Use Cases

	Conversations		Employee-Months
	N = 158,496		N = 17,671
Variable	Mean	Std. Dev.	Mean	Std. Dev.
Prompt Input-Output Measures:
1ST PROMPT LANG COMPLEXITY	2.586	0.724	2.474	0.442
1ST PROMPT COMPLEXITY	2.305	0.665	2.272	0.372
STRUCTURE	1.738	0.734	1.629	0.404

Table 3: Style and Technique Measures

Variable	CLARITYZ	CLARITYZ	ANY STRAT	ANY STRAT	USE DIV	USE DIV
$\overline{\textit{1ST PROMPT LEN}}$	0.000***	0.000***	0.000***	0.000**	0.000***	0.000***
	(0.000)	(0.000)	(0.000)	(0.000)	(0.000)	(0.000)
$\overline{\textit{ROUNDS}}$	0.259***	0.257***	0.042***	0.044***	0.014***	0.023***
	(0.020)	(0.022)	(0.005)	(0.005)	(0.005)	(0.005)
CHAT DAYS	0.005	0.015	-0.043***	-0.018***	0.103***	0.144***
	(0.017)	(0.019)	(0.004)	(0.004)	(0.004)	(0.005)

Table 4: Multivariate Regressions




Sophisticated use measures
CLARITYZ	Composite (z-scored) measure of prompt clarity for employee $i$ in month $t$ . Constructed by first z-scoring, within the sample of active employee-months, the employee-month averages of six LLM-assigned continuous prompt-quality ratings ( 1ST PROMPT LANG COMPLEXITY , 1ST PROMPT COMPLEXITY , STRUCTURE , PROMPT SOPH , SPECIFICITY , and FORMAT CLARITY ) and three LLM-assigned binary prompt features ( CONSTRAINTS , STRUCT OUTPUT , and ACCEPT CRITERIA ). The composite is the equally weighted average of these nine z-scores and is then z-scored again within active employee-months. Higher values indicate clearer, more structured prompts.
ANY STRAT	Composite measure of deliberate prompting strategy use for employee $i$ in month $t$ . Constructed as the proportion of the employee-month’s conversations in which at least one of five prompting strategies equals one ( ROLE PLAY , FEW SHOT , CHAIN OF THOUGHT , SELF CHECK , or INTERACTIVE REFINE ).
USE DIV	Measure of how broadly an employee’s LLM conversations span different use-case categories in a given month. We compute it as a normalized Shannon-entropy index across that employee-month’s totals for the eight broad use-case categories listed below, counting each conversation once in every broad category to which it is assigned: for each category, we convert its monthly total into a share of the employee’s total across all categories, then compute entropy and normalize it to range from 0 to 1. The measure equals 0 when all activity falls in a single category (or when the employee-month has a total of zero across categories) and approaches 1 when activity is evenly spread across categories.

Table 13

Month	N	CHAT OR COP	CHAT USE	COP USE	$\overline{\textit{CHAT DAYS}}$	$\overline{\textit{COP DAYS}}$	$\overline{\textit{CHAT CONVS}}$
Sample Mean	32,849	0.839	0.437	0.780	2.714	5.832	8.563
2025-01	3,928	-0.060	-0.051	-0.070	-0.464	-1.360	-0.117
2025-02	4,101	-0.034	-0.016	-0.042	-0.191	-0.611	0.342
2025-03	4,105	0.006	0.008	0.003	0.193	-0.169	0.506
2025-04	4,078	0.020	0.012	0.023	0.180	0.323	0.480
2025-05	4,064	0.033	0.020	0.038	0.328	0.583	0.135

Table B1: Usage and Intensity

Functional Area	N	CHAT OR COP	CHAT USE	COP USE	$\overline{\textit{CHAT DAYS}}$	$\overline{\textit{COP DAYS}}$	$\overline{\textit{CHAT CONVS}}$
Sample Mean	32,849	0.839	0.437	0.780	2.714	5.832	8.563
Accounting & Finance	2,967	-0.057	-0.039	-0.043	-0.447	-0.768	-1.637
Administrative Services	2,995	0.004	0.031	0.005	0.048	-0.215	0.118
Affiliate Operations	657	-0.033	-0.122	-0.001	-0.877	-0.981	-0.776
Communications	827	0.088	0.115	0.110	0.847	2.034	0.481
Compliance	865	-0.105	-0.092	-0.189	-0.362	-1.308	1.343

Table B1: Usage and Intensity (continued)

Month	N	WRITING	CODING DATA	TEXT ANALYSIS	KNOWLEDGE	TOOL GUIDE	IDEATION	PERSONAL	OTHER
Sample Mean	158,496	0.733	0.073	0.107	0.230	0.099	0.018	0.019	0.012
2025-01	15,992	0.025	-0.014	-0.002	0.003	-0.011	0.004	-0.003	-0.001
2025-02	19,775	0.013	-0.009	0.007	0.011	-0.004	0.006	-0.004	-0.000
2025-03	21,641	-0.003	0.001	0.004	0.020	-0.002	0.002	-0.003	0.002
2025-04	22,144	-0.007	0.007	0.000	-0.011	0.004	-0.001	0.000	0.000
2025-05	20,270	-0.000	0.006	0.016	0.005	-0.001	-0.003	0.001	-0.001

Table B2: Use Cases by Group

Functional Area	N	WRITING	CODING DATA	TEXT ANALYSIS	KNOWLEDGE	TOOL GUIDE	IDEATION	PERSONAL	OTHER
Sample Mean	131,736	0.756	0.052	0.104	0.240	0.091	0.020	0.021	0.012
Accounting & Finance	8,759	-0.124	0.021	-0.032	0.040	0.059	-0.009	-0.000	-0.002
Administrative Services	14,172	0.050	-0.029	-0.018	-0.058	-0.020	-0.003	0.020	0.004
Affiliate Operations	1,628	0.083	-0.008	-0.032	-0.095	-0.029	-0.009	-0.011	-0.003
Communications	4,273	0.118	-0.039	-0.004	-0.031	-0.071	0.009	-0.010	-0.006
Compliance	2,991	0.051	-0.036	-0.031	0.039	-0.029	-0.009	-0.005	-0.002

Table B2: Use Cases by Group (continued)

Group	N	CLARITYZ	ANY STRAT	USE DIV
Sample Mean	17,671	0.000	0.058	0.316
2025-01	1,799	-0.078	-0.003	-0.023
2025-02	2,048	-0.031	0.007	0.005
2025-03	2,191	0.017	0.008	0.012
2025-04	2,262	-0.019	0.006	0.007
2025-05	2,257	0.017	-0.011	0.001

Table B3: Evaluation of Use

Group	N	CLARITYZ	ANY STRAT	USE DIV
Sample Mean	14,449	0.002	0.061	0.322
Accounting & Finance	1,192	-0.311	-0.010	-0.019
Administrative Services	1,407	-0.159	-0.013	-0.015
Affiliate Operations	207	-0.139	-0.013	-0.050
Communications	461	0.343	0.007	-0.024
Compliance	299	-0.074	0.014	-0.016

Table B3: Evaluation of Use (continued)

		CLARITYZ		ANY STRAT		USE DIV
Predictor	N	beta	p	beta	p	beta	p
Ambition
$\overline{\textit{1ST PROMPT LEN}}$	17,671	0.000	0.000	0.000	0.000	0.000	0.000
Persistence
$\overline{\textit{ROUNDS}}$	17,671	0.279	0.000	0.021	0.000	0.054	0.000
$\textit{ROUNDS}\geq 2$	17,671	0.323	0.000	0.024	0.000	0.080	0.000

Table B4: Bivariate Associations

		CLARITYZ		ANY STRAT		USE DIV
Predictor	N	beta	p	beta	p	beta	p
Ambition
$\overline{\textit{1ST PROMPT LEN}}$	17,671	0.000	0.000	0.000	0.012	0.000	0.000
Persistence
$\overline{\textit{ROUNDS}}$	17,671	0.289	0.000	0.032	0.000	0.058	0.000
$\textit{ROUNDS}\geq 2$	17,671	0.244	0.000	0.028	0.000	0.053	0.000

Table B4: Bivariate Associations (continued)

AI and Labor

AI and labor studies how artificial intelligence changes jobs, wages, skills, hiring, and the organization and quality of work.

See all 89 papers

Generative AI and the Productivity Divide: Human-AI Complementarities in Education
May 18, 2026Lihi Idan, Bharat AnandHuman-AI Complementarity AI in Education
Industrial and Systems Engineering Department, Texas A&M University · Leonard N. Stern School of Business, New York University
How Organizations Use AI: Evidence from ChatGPT
Aug 12, 2026Aaron Chatterji, David Holtz, Neel Rakholia +2Generative AI Adoption Organizational AI Adoption
OpenAI · Columbia Business School · Wharton School, U. Pennsylvania
Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis
Mar 5, 2026Benjamin M. Chen, Hong BaoGenerative AI Adoption Generative AI in Education
Associate Professor and Gallant Ho Outstanding Young Professor, University of Hong Kong Faculty of Law. · PhD student, University of Hong Kong Faculty of Law.

May 18, 2026J/K move · Enter open · S save

Generative AI and the Productivity Divide: Human-AI Complementarities in Education

Lihi Idan, Bharat Anand

Industrial and Systems Engineering Department, Texas A&M University · Leonard N. Stern School of Business, New York University

Human-AI Complementarity AI in Education AI Literacy Generative AI in Education Human-AI Collaboration

Open paper AskPDF arXiv

Generative Artificial Intelligence (GenAI) is transforming how firms create, process, and apply knowledge, yet little is known about the heterogeneity of its productivity effects across users. We report results from a randomized controlled experiment in which participants-analogs of early-career knowledge workers-were assigned to self-study a technical domain using either traditional resources or large-language-model (LLM) assistance. On average, GenAI access significantly increased task performance, but the distribution of gains was highly uneven. Improvements were not predicted by GPA or prior knowledge, but by AI Interaction Competence (AIC) -- the ability to elicit, filter, and verify model outputs. High-AIC participants realized outsized gains; low-AIC participants saw limited or even negative marginal returns. A scaffolding intervention (conceptual maps) reduced outcome variance, indicating that standardized workflows can mitigate inequality in AI-mediated performance. We interpret these findings through the lens of human-AI complementarities: GenAI raises mean productivity while introducing a new axis of capability inequality. Managerially, firms should pair GenAI access with short AIC micro-training and simple standard operating procedures to capture value consistently and avoid uneven adoption outcomes.

Abstract

AI and Labor

Explore similar work

Generative AI and the Productivity Divide: Human-AI Complementarities in Education

How Organizations Use AI: Evidence from ChatGPT

Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis