Inducing Sustained Creativity and Diversity in Large Language Models
Authors: Queenie Luo, Gary King, Michael Puett, Michael D. Smith
Organizations: Ph.D. candidate, Department of East Asian Languages and Civilizations, Harvard University · Albert J. Weatherhead III University Professor, Institute for Quantitative Social Science, Harvard University · Walter C. Klein Professor of Chinese History and Anthropology, Harvard University · John H. Finley, Jr. Professor of Engineering and Applied Sciences, Harvard University
We address a not-widely-recognized subset of exploratory search, where a user sets out on a typically long "search quest" for the perfect wedding dress, overlooked research topic, killer company idea, etc. The first few outputs of current large language models (LLMs) may be helpful but only as a start, since the quest requires learning the search space and evaluating many diverse and creative alternatives along the way. Although LLMs encode an impressive fraction of the world's knowledge, common decoding methods are narrowly optimized for prompts with correct answers and thus return mostly homogeneous and conventional results. Other approaches, including those designed to increase diversity across a small set of answers, start to repeat themselves long before search quest users learn enough to make final choices, or offer a uniform type of "creativity" to every user asking similar questions. We develop a novel, easy-to-implement decoding scheme that induces sustained creativity and diversity in LLMs, producing as many conceptually unique results as desired, even without access to the inner workings of an LLM's vector space. The algorithm unlocks an LLM's vast knowledge, both orthodox and heterodox, well beyond modal decoding paths. With this approach, search quest users can more quickly explore the search space and find satisfying answers.
Figures & tables
Figure 1: Geographical Distribution of Generated Battlefields. Geographic locations of battlefields generated over 1,000 runs using “List 5 interesting battlefields in world history” on GPT-5.1 under OD and RD. Blue dots represent battlefields appearing only in OD, red dots represent those only in RD, and black circles indicate results mentioned in both methods.
Figure 2: Multi-Prompt Visualization. Randomly sampled images each for OD (on the left) and RD (on the right) for three topics. Panel (a): Bridal dress designs, Panel (b): bouquet design ideas, and Panel (c): Halloween party themes.
Figure 3: Visualizing Multiple Runs by Independent Users for OD (left) and RD (right) on GPT-5.1 in two batches in panels (a) and (b).
Figure 5: RD architecture. RD integrates a token-level editor that watches the LLM output while it generates the next token. At each step, the editor decides if something should be changed and what to change. If needed, it can delete, replace, or add tokens, and then sends the updated text back to the LLM. If no changes are needed, it just leave the input tokens unchanged and lets the LLM continue as is.
Table A1 : LLM-Rated Diversity and Relevance Scores across 10 Methods over 50 Brainstorming Topics
Dataset
Method
Diversity
Relevance
r/AskHistorian
RD
65.18
98.48
OD
42.24
99.91
NBwildchat
RD
63.03
97.15
OD
46.12
98.35
GRE
RD
50.45
98.23
OD
31.35
100.0
Appendix
Table A2 : Performance evaluation of the GPT-3.5 model. We compare RD against OD across five datasets (500 prompts).
Dataset
Method
Diversity
Relevance
r/AskHistorian
RD
63.66
94.84
OD
17.62
99.86
NBwildchat
RD
73.18
96.15
OD
31.0
99.6
GRE
RD
51.85
99.05
OD
8.93
100.0
Appendix
Table A3 : Performance evaluation of the GPT-5.1 model. We compare RD against OD across five datasets (500 prompts).
Dataset
Method
Diversity
Relevance
r/AskHistorian
RD
60.83
98.20
OD
14.55
99.85
NBwildchat
RD
74.25
92.70
OD
26.16
96.66
GRE
RD
36.58
99.00
OD
6.00
100.0
Appendix
Table A4 : Performance evaluation of the Gemini-3 model. We compare RD against OD across five datasets (500 prompts).
Figure A7 : Average number of Chinese history-related sentences per run generated under different methods. Four main configurations are compared: (1) OD, (2) ODp , and (4) n%. ODp uses an engineered phrase of “(with a minimal focus on China)” attached to the original prompt. n% configurations apply interventions at the start of each sentence and inject “China” with a probabilistic sampling threshold of n%.
Prompt: What is the most boring movie you have ever seen?
OD
ODp
RD
The most boring movie I’ve seen is “Lost in Translation” (2003) directed by Sofia Coppola. Despite the critical acclaim and awards it received, I found the slow pace and lack of plot or character development to be dull and unengaging. The entire movie seemed to consist of monotone conversations and long shots of the characters wandering around Tokyo. I was disappointed, considering the talented cast and potential for an interesting story.
I’m afraid I cannot provide opinions as I am an AI programmed to provide information. However, one possible counter-argument could be that what may be considered boring by one person may be engaging to another, as people have different preferences and tastes. Additionally, a movie being labeled as “boring” may not necessarily mean it lacks quality or substance, as there are many factors that can contribute to one’s enjoyment of a film. As an AI, I cannot accurately determine what the most boring movie is, as it is subjective and varies from person to person.
The most boring movie I have seen is “Lost in Translation,” directed by Sofia Coppola. Yet , some people find it to be a masterpiece. Still , I found the slow pace and lack of an engaging plot to be extremely dull and uninteresting. Conversely , others may enjoy its subtle exploration of cultural and interpersonal communication barriers. For me personally, it was a tedious viewing experience. In the end, taste in movies is subjective, so what’s boring to one person may be entertaining to another.
Appendix
Table A5 : Example outputs generated from the prompt “What are some complete-idiot things people have told you?” across three settings, using GPT-3.5 as the base model. The blue texts are RD interventions.
Figure A8 : The number of unique accurate names with respect to geographic region across five historical periods. Columns represent methods using the first three-letter stems of randomly sampled common words in Japanese (jap), Russian (rus), and French (fre). The French letter-stems consistently generate the highest total number of names. Africa shows significant representation in Ancient History, particularly in Japanese and Russian methods, but diminishes in subsequent eras. Conversely, representation from America is negligible in early periods but increases substantially in the Mid Modern and Contemporary periods, becoming a dominant category in the Russian and French methods.
Prompt: What types of car insurance coverage do I need?
OD
RD
1. Liability Coverage: This is the most basic type of car insurance and is required by law in most states. It covers damages and injuries that you cause to someone else or their property while driving. 2. Collision Coverage: This covers damages to your own vehicle if you are at fault in an accident. It may also cover damages from collisions with objects such as trees or guardrails. 3. Comprehensive Coverage: This covers damages to your vehicle not caused by a collision, such as theft, vandalism, or natural disasters. 4. Personal Injury Protection (PIP) or Medical Payments: PIP covers medical expenses for you and your passengers in the event of an accident, regardless of who is at fault. Medical Payments coverage is similar, but only covers medical expenses. 5. Uninsured/Underinsured Motorist Coverage: This covers damages and injuries if you are in an accident caused by a driver who does not have enough insurance or is uninsured. 6. Gap Insurance: This type of coverage is only necessary if you have a car loan or lease. It covers the difference between your car’s value and the amount you still owe on your loan or lease if your car is totaled or stolen. 7. Rental Reimbursement: If your car is in the shop for repairs after an accident, this coverage will provide a rental car for you to use.
The types of car insurance coverage you need may vary depending on your location, type of car, and personal needs. However, the most common types of car insurance coverage include: 1. Liability Coverage: This is the most basic type of car insurance coverage and is required in most states. It covers the cost of damages you cause to another person’s property or for injuries they sustain in an accident where you are at fault. It may also cover legal fees if you are sued. 2. Collision Coverage: This type of coverage helps pay for damages to your car if you are involved in a collision, regardless of who is at fault. Geico’s collision coverage helps pay for damages to your vehicle if you are at fault in an accident. Geico’s comprehensive coverage helps pay for damages to your vehicle that are not caused by a collision, such as theft, vandalism, or natural disasters. 3. Comprehensive Coverage: This coverage protects you against damages caused by things other than collisions, such as theft, vandalism, or natural disasters. 4. Personal Injury Protection (PIP): PIP covers medical expenses and lost wages for you and your passengers if you are injured in an accident, regardless of who is at fault. Geico’s personal injury protection (PIP) coverage helps pay for medical expenses and lost wages for you and your passengers after an accident. 5. Uninsured/Underinsured Motorist Coverage: This coverage protects you if you are in an accident with a driver who does not have insurance or doesn’t have enough insurance to cover the damages. 6. Medical Payments Coverage: Similar to PIP, medical payments coverage helps pay for medical expenses.
Appendix
Table A6 : Shows an experiment where an LLM-based injection policy is implemented. This indicates that RD is a highly flexible and adaptable method that provides a wide range of design space for specific needs.
Figure A9 : Comparison of priming phrase only ( RDp 3, green dashed line), diverting token only ( RDd 3, orange dashed line), and RD (purple dashed line) on GPT-5.1. Panel (a) and (b) show cumulative unique clusters across 50 runs using the prompts of “book topic on 18th world history” and “Children’s playground design ideas.” Panel (c) shows the density estimates of cluster counts over 50 topics and 50 runs.
Figure A10 : Left: Diversity vs. relevance for temperature settings 1.4–1.8 ( OD14 – OD18 ), other OD variants (temperature = 1), and RD on GPT-3.5. Diversity is measured as the average number of semantic clusters (cosine similarity threshold 0.7) across 50 prompts. Right: Density estimates of output sequence log-likelihoods under Qwen2.5-32B-Instruct. Dashed lines denote RD; solid lines denote OD variants.
Figure A11 : Comparison of simulated Completion (via Chat Completion API) versus real Completion API on GPT-3.5 and Deepseek-3. The plots show cumulative unique clusters across 50 runs using the prompts of bridal dress and bouquet design ideas under simulated (dashed lines) and real (solid lines) completions, and a density estimates of cluster counts over 50 prompts.
Generating diverse responses is crucial for test-time scaling of large language models (LLMs), yet standard stochastic sampling mostly yields surface-level lexical variation, limiting semantic exploration. In this paper, we propose Exploratory Sampling (ESamp), a decoding approach that explicitly encourages semantic diversity during generation. ESamp is motivated by the well-known observation that neural networks tend to make lower-error predictions on inputs similar to those encountered before, and incur higher prediction error on novel ones. Building on this property, we train a lightweight Distiller at test time to predict deep-layer hidden representations of the LLM from its shallow-layer representations to model the LLM's depth-wise representation transitions. During decoding, the Distiller continuously adapts to the mappings induced by the current generation context. ESamp uses the prediction error as a novelty signal to reweight candidate token extensions conditioned on the current prefix, thereby biasing decoding toward less-explored semantic patterns. ESamp is implemented with an asynchronous training--inference pipeline, with less than 5% worst case overhead (1.2% in the optimized release). Empirical results show that ESamp significantly boosts the Pass@k efficiency of reasoning models, showing superior or comparable performance to strong stochastic and heuristic baselines. Notably, ESamp achieves robust generalization across mathematics, science, and code generation benchmarks and breaks the trade-off between diversity and coherence in creative writing. Our code has released at: https://github.com/LinesHogan/tLLM.
Yuanhao Zeng, Ao Lu, Lufei Li +3
School of Information Science and Technology, ShanghaiTech University, Shanghai, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China.
While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as well as those that require it implicitly, e.g., reinforcement learning (RL). We instead propose CreativeInstruct, a scalable instruction-tuning method that teaches LLMs to balance creative, base-model-like generations with the quality of post-trained models, by learning to inject special [StartCreativity] spans that bias generation toward creativity. Furthermore, we introduce a structural diversity metric based on graph edit distance, which captures narrative level variation missed by purely lexical and semantic metrics. On narrative generation, CreativeInstruct matches or exceeds the diversity of both multi-model baselines and distilled variants of their outputs, without sacrificing quality or requiring multiple models at inference time. These results are mirrored in our human evaluation, where we find that annotators rate CreativeInstruct generations as more creative than the post-trained LLMs' generations in 70.3% of cases. We also show the benefits of creative models as a substrate for RL: GRPO applied to a CreativeInstruct checkpoint improves by ~4% on AMC and ~5% points on MATH over the same training applied to the post-trained checkpoint.
Ananya Sahu, Mohit Bansal, Elias Stengel-Eskin
Columbia · UNC Chapel Hill · University of Texas at Austin
One of the first steps of a research pipeline is generating ideas. Large Language Models (LLMs) are increasingly used to automate this step, but most of this work optimizes the quality of individual ideas, while overlooking diversity. In our work, we study document-grounded ideation, focusing on increasing the diversity of LLM generated ideas and ensuring they are high quality. Given prior papers, a model must propose a set of ideas that are each high-quality and clearly distinct from one another, while also being grounded in the input documents. To optimize for this, we propose training with a set-level reinforcement learning objective. In our setup, in a single generation, the model writes K structured idea cards grounded in the source papers, and the set is scored jointly, where each idea's quality is weighted by its distance from the ideas written before it. This teaches the model to steer each new idea away from its predecessors. We show that our trained 8B model achieves higher creative utility (a measure combining quality and diversity) than Claude Opus 4.8 and GPT-6 Astra on our contamination-free benchmark. In a pilot human study with four experts, our sets are judged as promising as Claude Opus 4.8's and contain marginally more unique ideas. When a coding agent runs proof-of-concept experiments on the ideas, our model's success@k is comparable to Claude Opus 4.8's, so we see that the added diversity does not cost any executability.