In your own words: computationally identifying interpretable themes in free-text survey data
Authors: Jenny S Wang, Aliya Saperstein, Emma Pierson
Organizations: Technology and Operations Management Unit, Harvard Business School, Boston, Massachusetts, USA. · Department of Sociology, Stanford University, Palo Alto, CA, USA. · Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, USA.
Free-text survey responses can provide nuance often missed by structured questions, but remain difficult to statistically analyze. To address this, we introduce In Your Own Words, a computational framework for exploratory analyses of free-text survey data that identifies structured, interpretable themes in free-text responses, facilitating systematic analysis. To illustrate the benefits of this approach, we apply it to a new dataset of free-text descriptions of race, gender, and sexual orientation from 1,004 U.S. participants. The themes our approach produces on this dataset are more coherent and interpretable than those produced by past computational methods. The themes have three practical applications in survey research. First, they can suggest structured questions to add to future surveys by surfacing salient constructs - such as belonging and identity fluidity - that existing surveys do not capture. Second, the themes reveal heterogeneity within standardized categories, explaining additional variation in health, well-being, and identity importance. Third, the themes illuminate systematic discordance between self-identified and perceived identities, highlighting mechanisms of misrecognition that existing measures do not reflect. More broadly, our framework can be deployed in a wide range of survey settings to identify interpretable themes from free text, complementing existing qualitative methods.
Figures & tables
Figure 1: Overview of data collection (a) and computational framework (b – e). (a) : To create the dataset analyzed via the In Your Own Words framework, we ask participants to describe their race, gender, and sexual orientation in free text. (b) : We convert the free-text responses into embedding vectors that capture their semantic meaning but are not readily interpretable. (c) : We then use a sparse autoencoder (SAE) to extract more interpretable dimensions from the embeddings; each dimension captures a recurring pattern in how identity is expressed, such as references to cultural heritage, language, or childhood experiences. (d) To produce a text interpretation of each dimension, we prompt a large language model (LLM) to identify the common theme among the free-text responses that score highly along that dimension. (e) : We then use an LLM to annotate each free-text response for whether it contains each theme. Themes described in the illustration are abbreviated for space.
Figure 2: Free-text themes produced by our computational framework . Each row shows one free-text theme, and colored bars indicate the standardized category of respondents whose free-text responses contain the theme. While some themes are predominantly associated with a single category (e.g., “mentions never questioning sexual orientation” is mostly expressed by straight respondents) many themes cut across categories within each identity axis. Very rare standardized categories (with fewer than 10 respondents) are excluded. For space, free-text themes are abbreviated, and only a subset of themes are shown; the full text of all themes is provided in Figures S7 - S9 . Section D.2 describes the standardized categories in more detail.
Free-Text Question
In at least 2–3 sentences, how would you describe your race ? For example, you could discuss specific traditions or customs, cultural practices or norms, languages or dialects spoken in your family, or ways race and/or ethnicity has influenced your life experiences.
In at least 2–3 sentences, how would you describe your gender ? For example, you could discuss ways your gender identity influences your life or significant milestones or moments in your life related to your gender identity.
In at least 2–3 sentences, how would you describe your sexual orientation ? For example, you could discuss ways your sexual orientation influences your life, significant milestones or moments in your life related to your sexual orientation, or any differences between your romantic and sexual identities.
Table 1: Race, Gender, and Sexual Orientation Free-Text Questions
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. S1 : Self-described identity: Distribution of word counts in participants’ free-text responses about their race, gender, and sexual orientation. Dashed lines indicate median word count for each identity axis. Annotated quotes highlight the richness in information the median respondent is able to provide.
Fig. S2 : Perceived identity: Distribution of word counts in participants’ free-text responses about their perceived race, gender, and sexual orientation. Dashed lines indicate median word count for each identity axis. Annotated quotes highlight the richness in information the median respondent is able to provide.
Category
Multiselect (%)
Race and/or Ethnicity
American Indian or Alaska Native
6.8%
Asian
15.4%
Black or African American
22.2%
Hispanic or Latino
16.8%
Middle Eastern or North African
4.4%
Appendix
Table S1 : Demographic breakdown of the full sample (N = 1,004). Percentages indicate the share of participants who selected each identity category. Because race and sexual orientation were multiselect items, totals for these categories exceed 100%.
Fig. S3 : Respondents with minority identities report greater added value from free text and write longer responses than individuals with non-minority identities. Across both panels, reference (non-minority) categories are White for race, both Cisgender Man and Cisgender Woman for gender, and Straight or heterosexual for sexual orientation; all comparisons are computed within identity axis. Left: Odds ratios from identity-specific logistic regressions predicting answering “Yes” to: Do you feel that your free-text response provides important details about your [identity] that were not captured by the multiple-choice questions? Right: Percent change in free-text word count relative to the corresponding reference category (within each identity axis). Categories with ≤ 15 responses are omitted from category-specific regressions due to unstable estimates but are included in the “Minority” vs. “Not Minority” regression.
Fig. S4 : Prompt used to interpret high-activating responses within an SAE dimension, adapted from Movva et al. ( 2025 ). Unlike the original general-purpose prompt, this version is tailored for identity-specific responses and provides specific formatting suggestions to standardize the output features.
Fig. S5 : Prompt used to annotate whether a response satisfies a theme. The structure is similar to Movva et al. ( 2025 ), but we replace their general-purpose examples with identity-specific few-shot cases to better capture the nuances of race, gender, and sexual orientation.
Model
Prompt
Race κ
Gender κ
Sexual Orient. κ
Human Pos. (%)
LLM Pos. (%)
Δ Pos. (pp)
GPT-4.1-mini
Baseline
0.754
0.552
0.609
13.0
16.3
+3.3
GPT-4.1
Baseline
0.685
0.559
0.741
13.0
10.2
-2.8
GPT-4.1-mini
Identity-Specific
0.771
0.544
0.666
13.0
13.1
+0.1
GPT-4.1
Identity-Specific
0.683
0.547
0.778
13.0
9.0
-4.0
Appendix
Table S2 : Annotation agreement and positive annotation rates across annotation model configurations. Cohen’s Kappa ( κ ) compares each model’s annotations to a human annotator. Human Pos. and LLM Pos. report average positive annotation rates expressed as percentages, where the positive rate denotes the fraction of responses labeled as activating a given theme in response to the annotation prompt (Figure S5 ). Δ Pos. reports the difference between model and human positive rates in percentage points. We select the (GPT-4.1-mini, Identity-Specific) configuration for subsequent analyses, as it achieves reasonably high agreement across identity axes while most closely matching the human positive annotation rate.
Fig. S6 : Reliability of GPT-4.1-mini across identity axes. Distribution of Cohen’s Kappa ( κ ) scores for the GPT-4.1-mini and identity-specific prompt configuration used in our final analyses. Each bar represents a unique theme within the race, gender, and sexual orientation categories. The median score of 0.65 (horizontal line) indicates substantial agreement with human labels.
Race Theme
Fidelity
1
mentions celebrating Christian holidays
0.78
2
uses the phrase ‘I would describe my race’ or a close variation of it
0.91
3
mentions speaking English or American English as a primary language
0.90
4
mentions being first, second, or third generation American or immigrant
0.75
5
mentions pride in Black or African American identity
0.88
6
uses the phrase ‘I am’ followed by a race or ethnicity descriptor without elaboration
0.75
Appendix
Table S3 : Computationally identified free-text themes for SAE dimensions trained on race-related free-text response embeddings and their associated interpretation fidelity (F1 score). Low-fidelity themes (F1 <0.50 ) and themes reflecting writing style rather than semantic content are excluded from analysis; excluded themes are indicated with strikethroughs.
Gender Theme
Fidelity
1
uses the exact word ‘male’
0.91
2
mentions childhood experiences related to gender roles or expectations
0.78
3
mentions responsibility to provide for family
0.55
4
mentions use of multiple pronouns or experimenting with pronouns
0.46
5
mentions physical appearance or specific physical traits
0.38
6
mentions giving birth or having children as a significant aspect of their identity
0.86
Appendix
Table S4 : Computationally identified free-text themes for SAE dimensions trained on gender-related free-text response embeddings and their associated interpretation fidelity (F1 score). Low-fidelity themes (F1 <0.50 ) and themes reflecting writing style rather than semantic content are excluded from analysis; excluded themes are indicated with strikethroughs.
Sexual Orientation Theme
Fidelity
1
refuses to provide a clear or specific answer about sexual orientation
0.64
2
explicitly states attraction to ‘men’ without additional context or qualifiers
0.91
3
mentions demisexuality or being demisexual
0.89
4
mentions being married
0.91
5
explicitly self-describes as asexual or aromantic
0.99
6
explicitly self-describes as straight or heterosexual without additional context or elaboration
0.80
Appendix
Table S5 : Computationally identified free-text themes for SAE dimensions trained on sexual orientation-related free-text response embeddings and their associated interpretation fidelity (F1 score). Low-fidelity themes (F1 <0.50 ) and themes reflecting writing style rather than semantic content are excluded from analysis; excluded themes are indicated with strikethroughs.
Fig. S7 : Some identity themes align with standardized racial categories, but many cut across categories . Each row represents a theme, with colored bars representing the standardized race categories of respondents whose free-text responses contain the theme. Themes are ordered by the maximum proportion of responses (among those mentioning the theme) that come from any single race category—in other words, by the highest value among the race-specific proportions for each theme. Thus, themes at the top cut across categories and are not strongly associated with any particular category, while themes at the bottom are predominantly expressed by respondents from specific race categories.
Fig. S8 : Some identity themes align with standardized gender categories, but many cut across categories . Each row represents a theme, with colored bars representing the standardized gender categories of respondents whose free-text responses contain the theme. Themes are ordered by the maximum proportion of responses (among those mentioning the theme) that come from any single gender category—in other words, by the highest value among the gender-specific proportions for each theme. Thus, themes at the top cut across categories and are not strongly associated with any particular category, while themes at the bottom are predominantly expressed by respondents from specific gender categories.
Fig. S9 : Some identity themes align with standardized sexual orientation categories, but many cut across categories . Each row represents a theme, with colored bars representing the standardized sexual orientation categories of respondents whose free-text responses contain the theme. Themes are ordered by the maximum proportion of responses (among those mentioning the theme) that come from any single sexual orientation category—in other words, by the highest value among the sexual orientation-specific proportions for each theme. Thus, themes at the top cut across categories and are not strongly associated with any particular category, while themes at the bottom are predominantly expressed by respondents from specific sexual orientation categories.
Race Theme
R2
mentions being born in the United States or America
0.01
mentions celebrating Christian holidays
0.03
mentions speaking English or American English as a primary language
0.03
mentions traditional or cultural food as an important aspect of their identity
0.04
explicitly states a lack of cultural or ethnic identity or traditions
0.05
mentions being first, second, or third generation American or immigrant
0.05
Appendix
Table S6 : Fraction of variance in each race theme ( R2 ) explained by the standardized categories, sorted by ascending R2 .
Gender Theme
R2
mentions childhood experiences related to gender roles or expectations
0.00
uses the word ‘cisgender’
0.01
discusses how gender impacts societal roles, opportunities, or treatment
0.01
explicitly states being heterosexual
0.02
mentions roles or responsibilities within a family or household
0.02
mentions how gender identity influences decisions, safety, or interactions in life
0.02
Appendix
Table S7 : Fraction of variance in each gender theme ( R2 ) explained by the standardized categories, sorted by ascending R2 .
Sexual Orientation Theme
R2
mentions being married
0.02
discusses how sexual orientation influences personal growth, relationships, or key life milestones
0.03
mentions that their sexual orientation does not significantly impact their life
0.03
discusses how their sexual orientation influences their personal relationships or life milestones
0.04
mentions ‘opposite sex’ or ‘opposite gender’
0.05
explicitly states attraction to ‘men’ without additional context or qualifiers
0.05
Appendix
Table S8 : Fraction of variance in each sexual orientation theme ( R2 ) explained by the standardized categories, sorted by ascending R2 .
Race
Gender
Sexual Orientation
R2
0.15
0.12
0.22
Adjusted R2
0.14
0.11
0.22
R2 (McFadden)
0.31
0.18
0.26
R2 (Cox-Snell)
0.13
0.11
0.20
Appendix
Table S9 : Median R2 values between themes and standardized categories under alternative metrics. We assess how much variation in theme activation is explained by standardized categories. Standard and adjusted R2 are computed from linear probability models with binary theme activation as the outcome, while McFadden and Cox-Snell R2 are computed from logistic regressions with the same outcome. All values are reported as medians across themes within each identity axis.
Identity
Outcome
Relative R2 Increase ( × )
Race
Mental Health
1.5 × (0.04 → 0.06) ∗
Physical Health
1.9 × (0.02 → 0.05) ∗
Life Satisfaction
1.9 × (0.01 → 0.03)
Everyday Discrimination
0.8 × (0.02 → 0.02)
Income
1.5 × (0.02 → 0.04)
Identity Importance
1.1 × (0.30 → 0.33) ∗∗∗
Appendix
Table S10 : Free-text themes reveal within-group heterogeneity. For each identity-outcome pair (rows) we compute the adjusted R2 for two regression models: one that predicts the outcome from the standardized categories alone; and one that additionally includes free-text themes. Each row reports the ratio of adjusted R2 from the two models, with numbers greater than one indicating that the free-text themes increase adjusted R2 ; the adjusted R2 for both models are reported in parentheses. Stars denote whether there is a statistically significant increase in predictive power; * indicates p<0.05 ; ** p<0.01 ; *** p<0.001 ; all p-values are computed using F-tests for improvement in model fit with Benjamini-Hochberg multiple hypothesis correction ( FDR=0.05 ) Benjamini and Hochberg (1995) .
Fig. S10 : Regression coefficients for free-text race themes, when controlling for standardized race categories, in regressions to predict the three outcomes—race importance, physical health, and mental health—for which free-text themes significantly improved model fit after multiple hypothesis correction. We estimate models for each theme individually because themes may be collinear, complicating interpretation of multivariate models. Both point estimates and 95% confidence intervals are plotted (uncorrected for multiple hypotheses). We report HC3 heteroskedasticity-consistent standard errors. Themes are ordered by the estimated coefficient for Race Importance (largest to smallest).
Fig. S11 : Regression coefficients for free-text gender themes, when controlling for standardized gender categories, in regressions to predict the four outcomes—gender importance, physical health, mental health, and life satisfaction—for which free-text themes significantly improved model fit after multiple hypothesis correction. We estimate models for each theme individually because themes may be collinear, complicating interpretation of multivariate models. Both point estimates and 95% confidence intervals are plotted (uncorrected for multiple hypotheses). We report HC3 heteroskedasticity-consistent standard errors. Themes are ordered by the estimated coefficient for Gender Importance (largest to smallest).
Fig. S12 : Regression coefficients for free-text sexual orientation themes, when controlling for standardized sexual orientation categories, in regressions to predict the two outcomes—sexual orientation importance and mental health—for which free-text themes significantly improved model fit after multiple hypothesis correction. We estimate models for each theme individually because themes may be collinear, complicating interpretation of multivariate models. Both point estimates and 95% confidence intervals are plotted (uncorrected for multiple hypotheses). We report HC3 heteroskedasticity-consistent standard errors. Themes are ordered by the estimated coefficient for Sexual Orientation Importance (largest to smallest).
Perceived Race Theme
Fidelity
1
mentions connection (or lack thereof) to European heritage or traditions
0.548
2
mentions being black or African
0.916
3
mentions feeling disconnected from or not fully immersed in their cultural heritage
0.804
4
mentions being perceived as white
0.800
5
mentions equality or equal treatment
0.436
6
mentions uncertainty or lack of knowledge about how others perceive their race/ethnicity
0.630
Appendix
Table S11 : Computationally identified free-text themes for SAE dimensions trained on perceived race-related free text response embeddings and their associated fidelity (F1 score).
Perceived Gender Theme
Fidelity
1
mentions being perceived with mixed or multiple gender identities
0.810
2
mentions societal or cultural expectations related to gender
0.679
3
discusses societal perceptions or stereotypes specifically about women
0.561
4
mentions societal perceptions or treatment of women
0.749
5
mentions discomfort or negative emotions regarding the topic
0.316
6
expresses uncertainty or avoidance in response
0.480
Appendix
Table S12 : Computationally identified free-text themes for SAE dimensions trained on perceived gender-related free text response embeddings and their associated fidelity (F1 score).
Perceived Sexual Orientation Theme
Fidelity
1
avoids explicitly identifying or discussing specific sexual orientations
0.486
2
expresses indifference or lack of concern about others’ perceptions of their sexual orientation
0.748
3
mentions connection or comparison between sexual orientation and gender identity
0.699
4
mentions how others assume they are heterosexual due to appearances or societal norms
0.652
5
mentions being perceived as straight due to being married to a man
0.828
6
mentions being perceived as lesbian or bisexual
0.732
Appendix
Table S13 : Computationally identified free-text themes for SAE dimensions trained on perceived sexual orientation-related free text response embeddings and their associated fidelity (F1 score).
Table S14 : Top Words for LDA Model Learned from Race Free-Text Responses. Themes were learned using Gibbs sampling on raw text with stopwords and lemmatization intact. The “# Responses” column indicates the number of responses where that theme was the primary (highest probability) assignment. “Top 10 Race Words” displays the most probable terms for each theme after a post-training filter removed the top 50 high-frequency stopwords for interpretability.
# Responses
Top 10 Gender Words
1
13
can, family, women, men, our, hard, they, when, believe, you
Table S15 : Top Words for LDA Model Learned from Gender Free-Text Responses. Themes were learned using Gibbs sampling on raw text with stopwords and lemmatization intact. The “# Responses” column indicates the number of responses where that theme was the primary (highest probability) assignment. “Top 10 Gender Words” displays the most probable terms for each theme after a post-training filter removed the top 50 high-frequency stopwords for interpretability.
# Responses
Top 10 Sexual Orientation Words
1
339
asexual, feel, really, like, sexual, attraction, think, if, ’ve, can
2
1
rainbow, christian, no, he, focusing, sunglasses, pins, covered, telling, human
3
0
base, sexual, attracted, straight, heterosexual, orientation, men, women, people, like
4
1
no, back, problem, effeminate, ’re, hope, nowadays, fits, gay, identity
5
0
intimacy, getting, trauma, sexuality, labels, beyond, must, last, kept, earlier
6
0
base, sexual, attracted, straight, heterosexual, orientation, men, women, people, like
Appendix
Table S16 : Top Words for LDA Model Learned from Sexual Orientation Free-Text Responses. Themes were learned using Gibbs sampling on raw text with stopwords and lemmatization intact. The “# Responses” column indicates the number of responses where that theme was the primary (highest probability) assignment. “Top 10 Sexual Orientation Words” displays the most probable terms for each theme after a post-training filter removed the top 50 high-frequency stopwords for interpretability.
# Responses
Theme Representation
1
178
Black American Identity and Heritage
2
64
Asian American Identity and Cultural Practices
3
44
European Ancestry of White Americans
4
39
Cultural Heritage and Ethnic Identity in American Families
5
39
White American English Speakers
6
32
racial and ethnic self-identification
Appendix
Table S17 : BERTopic Representations Learned from Race Free-Text Responses. The “# Responses” column counts responses assigned to each unique cluster. Because the underlying clustering is noise-aware, 289 responses failing to meet the density threshold for any theme (representing roughly 29% of the corpus) are classified as noise and excluded from this table for clarity.
# Responses
Theme Representation
1
237
Gender Identity and Expression
2
67
Impact of Gender Identity on Women’s Life Experiences
3
63
Cisgender Female Identity
4
59
Gender Identity and Biological Sex
5
54
Male Identity and Biological Sex
6
48
Female Gender Identity
Appendix
Table S18 : BERTopic Representations Learned from Gender Free-Text Responses. The “# Responses” column counts responses assigned to each unique cluster. Because the underlying clustering is noise-aware, 175 responses failing to meet the density threshold for any theme (representing roughly 17% of the corpus) are classified as noise and excluded from this table for clarity.
# Responses
Theme Representation
1
96
Heterosexual Attraction to Men
2
87
Asexual and Aromantic Experiences
3
48
Bisexual Attraction to Men and Women
4
37
Heterosexual Identity
5
33
Demisexual and Emotional Attraction
6
33
Influence of Heterosexuality on Personal Life
Appendix
Table S19 : BERTopic Representations Learned from Sexual Orientation Free-Text Responses. The “# Responses” column counts responses assigned to each unique cluster. Because the underlying clustering is noise-aware, 299 responses failing to meet the density threshold for any theme (representing roughly 30% of the corpus) are classified as noise and excluded from this table for clarity.
Level
Count
Theme / Subtheme Name & Description
[1]
989
Cultural Identity: Mentions the significance of cultural heritage and personal identity in shaping experiences and perspectives.
[2]
1
└ Race and Ethnicity: How individuals identify with race/ethnicity, including cultural heritage and experiences.
[2]
1
└ Language and Communication: Role of language spoken at home and its impact on cultural connection.
[2]
1
└ Cultural Practices and Traditions: Cultural practices (food, celebrations, customs) shaping identity.
[1]
2
Agriculture: Mentions the context of living in a rural farming community.
Appendix
Table S20 : TopicGPT Hierarchical Themes Learned from Race Free-Text Responses. [1] denotes top-level theme; [2] indicates subthemes grouped under the preceding themes. Counts indicate the number of responses in which each theme was identified; responses may contribute to multiple themes.
Level
Count
Theme / Subtheme Name & Description
[1]
981
Gender Identity: Discusses the experiences and perspectives shaped by one’s gender identity, including the diversity of expressions and identities beyond traditional norms.
[2]
1
└ Cisgender: Refers to individuals whose gender identity matches the sex they were assigned at birth.
[2]
1
└ Transgender: Refers to individuals whose gender identity differs from the sex they were assigned at birth.
[2]
1
└ Non-binary: Refers to gender identities that do not fit within the traditional binary of male and female.
[2]
1
└ Gender Fluid: Refers to a gender identity that is not fixed and can change over time or depending on the situation.
[2]
1
└ Agender: Refers to individuals who do not identify with any gender.
Appendix
Table S21 : TopicGPT Hierarchical Themes Learned from Gender Free-Text Responses. [1] denotes top-level themes; [2] indicates subthemes grouped under the preceding theme. To improve readability, only themes appearing in ten or more responses are shown individually; all themes with fewer than ten occurrences are grouped under “Other Themes.” Counts indicate the number of responses in which each theme was identified; responses may contribute to multiple theme.
Level
Count
Theme / Subtheme Name & Description
[1]
240
Sexuality: Discusses the complexities of sexual orientation and the nuances of attraction and relationships.
[2]
1
└ Heterosexuality: Refers to attraction to the opposite gender.
[2]
1
└ Homosexuality: Refers to attraction to the same gender.
[2]
1
└ Bisexuality: Refers to attraction to both men and women.
[2]
1
└ Pansexuality: Refers to attraction to people regardless of gender.
[2]
1
└ Asexuality: Refers to little or no sexual attraction to others.
Appendix
Table S22 : TopicGPT Hierarchical Themes Learned from Sexual Orientation Free-Text Responses. [1] denotes top-level themes; [2] indicates subthemes grouped under the preceding theme. To improve readability, only themes appearing in 100 or more responses are shown individually; all themes with fewer than 100 occurrences are grouped under “Other Themes.” Counts indicate the number of responses in which each theme was identified; responses may contribute to multiple themes.
Race Theme
Fidelity
1
mentions whiteness or being white
0.359
2
self-describes as white
0.990
3
mentions cultural traditions or practices
0.940
4
explicitly self-describes as Black or African American
0.910
5
mentions collective or community-oriented values
0.699
6
mentions speaking an Asian language
0.723
Appendix
Table S23 : Computationally identified free-text themes for SAE dimensions ( M=16 , K=4 ) trained on race-related free text response embeddings and their associated fidelity (F1 score).
Gender Theme
Fidelity
1
uses the term ‘female’ or ‘females’ explicitly to describe gender
0.397
2
mentions physical appearance or traits about self
0.407
3
is a single word response
0.438
4
mentions being nonbinary or genderfluid
0.910
5
includes a definitive statement about gender identity without discussing ambiguity or questioning
0.561
6
self-describes as cisgender male or man
0.950
Appendix
Table S24 : Computationally identified free-text themes for SAE dimensions ( M=16 , K=4 ) trained on gender-related free text response embeddings and their associated fidelity (F1 score).
Sexual Orientation Theme
Fidelity
1
expresses uncertainty or refusal to describe their sexual orientation
0.615
2
mentions experimentation or exploration of sexuality
0.486
3
mentions fluidity or changes in sexual orientation over time
0.370
4
discusses how sexual orientation influences relationships or life experiences
0.815
5
mentions bisexual or queer as part of their sexual orientation
0.890
6
mentions comfort or natural feeling about sexual orientation
0.525
Appendix
Table S25 : Computationally identified free-text themes for SAE dimensions ( M=16 , K=4 ) trained on sexual orientation-related free text response embeddings and their associated fidelity (F1 score).
Race Theme
Fidelity
1
mentions African American identity or culture
0.769
2
self-describes using a single concise phrase or label without additional context or elaboration
0.679
3
mentions specific ancestry percentages or fractions
0.584
4
mentions lack of specific traditions or customs related to their race or ethnicity
0.767
5
mentions Arabic language or dialect
0.507
6
mentions indigenous roots or ancestry
0.686
Appendix
Table S26 : Computationally identified free-text themes for SAE dimensions ( M =64, K =4) trained on race-related free text response embeddings and their associated fidelity (F1 score).
Gender Theme
Fidelity
1
mentions using multiple pronouns or being comfortable with multiple pronouns
0.893
2
mentions racial or ethnic identity explicitly
0.630
3
explicitly self-describes as cisgender
0.626
4
mentions being born male and still identifying as male
0.867
5
mentions how gender impacts their life
0.452
6
discusses challenges or discrimination faced in professional or societal contexts due to gender
0.636
Appendix
Table S27 : Computationally identified free-text themes for SAE dimensions ( M =64, K =4) trained on gender-related free text response embeddings and their associated fidelity (F1 score).
Sexual Orientation Theme
Fidelity
1
mentions pansexual or pan-romantic
0.958
2
discusses how sexual orientation shapes identity, relationships, or personal growth
0.357
3
mentions attraction to or relationships with nonbinary or transgender individuals
0.838
4
uses the term ‘heterosexual’
0.889
5
mentions that sexual orientation does not significantly influence their life
0.844
6
mentions questioning or uncertainty about their sexual or romantic orientation
0.508
Appendix
Table S28 : Computationally identified free-text themes for SAE dimensions ( M =64, K =4) trained on sexual orientation-related free text response embeddings and their associated fidelity (F1 score).
Thematic analysis (TA) is widely used in health research to extract patterns from patient interviews, yet manual TA faces challenges in scalability and reproducibility. LLM-based automation can help, but existing approaches produce codebooks with limited generalizability and lack analytic auditability. We present an automated TA framework combining iterative codebook refinement with full provenance tracking. Evaluated on five corpora spanning clinical interviews, social media, and public transcripts, the framework achieves the highest composite quality score on four of five datasets compared to six baselines. Iterative refinement yields statistically significant improvements on four datasets with large effect sizes, driven by gains in code reusability and distributional consistency while preserving descriptive quality. On two clinical corpora (pediatric cardiology), generated themes align with expert-annotated themes.
Seungjun Yi, Joakim Nguyen, Huimin Xu +9
Department of Biomedical Engineering, University of Texas at Austin, Austin, TX, USA · Department of Computer Science, University of Texas at Austin, Austin, TX, USA · School of Information, University of Texas at Austin, Austin, TX, USA +6
Social scientists are increasingly turning to unstructured datasets to unlock new empirical insights, e.g., estimating descriptive statistics of or causal effects on quantitative measures derived from text, audio, or video data. In many settings, unsupervised analysis is of primary interest, in that the researcher does not want to (or cannot) manually pre-specify all important aspects of the unstructured data to measure; they are interested in "discovery." This paper proposes a general and flexible framework for pursuing such discovery from unstructured data in a statistically principled way. The framework leverages recent methods from the literature on AI interpretability to map unstructured data points to high-dimensional, sparse, and interpretable "concept embeddings"; computes statistics from these concept embeddings for testing interpretable, concept-by-concept hypotheses; performs selective inference on these hypotheses using algorithms validated by new results in high-dimensional central limit theory, producing a selected set ("discoveries"); and both generates and evaluates human-interpretable natural language descriptions of these discoveries. The proposed framework has few researcher degrees of freedom, is robust to data snooping and other post-selection inference concerns, and facilitates fast and inexpensive sensitivity analysis and replication. Applications to recent descriptive and causal analyses of unstructured data in empirical economics are explored.
Topic modeling in applied psychology increasingly spans two methodological traditions: probabilistic bag-of-words models and newer embedding-based approaches. Yet many evaluations of these methods rely on longer and cleaner benchmark corpora, leaving less guidance for short, open-ended survey responses. This paper compares Structural Topic Models (STM), a probabilistic topic model, and BERTopic, an embedding-based model, for analyzing open-ended survey responses. We evaluated three STM conditions and five BERTopic conditions, varying typographical correction, stemming, embedding choice, and contextual augmentation, a strategy we introduced to provide additional semantic context for very short responses. Results indicate that BERTopic consistently produced higher topic coherence than STM, with contextual augmentation yielding the strongest performance gains. In contrast, higher-dimensional embeddings alone did not improve coherence and were associated with greater data loss. Qualitative evaluation showed that BERTopic generated more interpretable and stable topics, while STM topics were often broader and more mixed. However, STM provides stronger support for inferential covariate analysis, whereas BERTopic covariate comparisons are primarily descriptive. These findings suggest that STM and BERTopic offer complementary strengths. We conclude with practical guidance for selecting and combining topic modeling approaches in applied social science research.
Yan Jiang, Sihong Liu, Philip A. Fisher
Stanford Center on Early Childhood, Stanford University