Dynamic Context Evolution for Scalable Synthetic Data Generation

Ryan Lingo; Rajeev Chhajer

cs.CLApr 8, 2026

Dynamic Context Evolution for Scalable Synthetic Data Generation

Authors: Ryan Lingo, Rajeev Chhajer

Organizations: Honda Research Institute, USA, Inc.

Abstract

Large language models produce repetitive output when prompted independently across many batches, a phenomenon we term cross-batch mode collapse: the progressive loss of output diversity when a language model is prompted repeatedly without access to its prior generations. Practitioners have long mitigated this with ad hoc deduplication and seed rotation, but no principled framework exists. We introduce Dynamic Context Evolution (DCE), comprising three mechanisms: (1) verbalized tail sampling (the model labels each idea with a guess about how obvious it is, and obvious ideas are discarded), which filters high-probability candidates via model self-assessment; (2) semantic memory, which maintains a persistent embedding index to reject near-duplicates across batches; and (3) adaptive prompt evolution, which reconstructs the generation prompt each batch using memory state and rotating diversity strategies. In experiments across three domains (sustainable packaging concepts, educational exam questions, and creative writing prompts) and two model families (gpt-5-mini and claude-haiku-4-5), a component ablation across 2-3 random seeds per method shows that DCE achieves 0.0 +/- 0.0% collapse versus 5.6 +/- 2.0% for naive prompting, while producing 17-18 HDBSCAN clusters per seed versus naive's volatile 2-17, indicating reliably richer conceptual structure. These results are validated with an independent embedding model (all-MiniLM-L6-v2) and hold across sensitivity sweeps of the VTS threshold tau and dedup threshold delta. Deduplication and prompt evolution are individually insufficient but jointly effective, at approximately $0.50 per 1,000 candidates using only standard API calls, with no fine-tuning or custom architectures required.

Figures & tables

Panel of Figure 1 — Figure 1: System architecture of the DCE generation loop. The pipeline flows left to right: the prompt constructor assembles each batch’s prompt from phase/strategy settings and semantic memory state, the LLM generates candidates, and successive filters (VTS, dedup) gate acceptance into the memory bank. The feedback loop along the bottom returns memory state (recent ideas, dense regions, category gaps) to the prompt constructor for the next batch.

Figure 3: Prompt evolution: batch 10 (left) vs. batch 190 (right). The summary callouts below each panel highlight the key quantitative difference: early prompts have sparse memory and prioritize broad exploration, while late prompts have rich category distributions, identify dense regions to avoid, and target specific gaps.

Method	Multiplicative	Additive	Geometric	Rank
Naive	32.7	64.0	56.9	1 / 2 / 1
Prompt evo + dedup	31.4	64.9	55.9	2 / 1 / 2
DCE (full)	28.7	63.7	53.4	3 / 3 / 3
VTS only	28.0	62.7	52.8	4 / 4 / 4
Dedup only	27.2	62.3	52.0	5 / 5 / 5
Prompt evo only	24.5	61.4	49.2	6 / 6 / 6

Table 1: EDV retention (%) under three formulations (packaging domain, seed 42, MiniLM embeddings). Rankings are nearly identical: VTS + dedup is the lowest-ranked method in all three, and the top methods cluster within a narrow range. Minor rank swaps occur at the top (naive and prompt evo + dedup exchange positions 1–2 under the additive form), but the overall ordering is stable.

Panel of Figure 5 — Figure 5: The EDV filtering space. Each candidate is scored on depth (how surprising, determined by VTS) and breadth (how distant from memory, determined by deduplication). Only candidates that pass both filters (top-right quadrant) contribute to EDV. The quadrant labels show which filter rejects candidates in each region.

Panel of Figure 6 — Figure 6: Collapse characterization: naive prompting vs. DCE (200 batches, 10-batch rolling averages). Left: naive novelty decays steadily after batch ${\sim}$ 60 (annotated), while DCE maintains elevated novelty throughout. Right: cluster count saturates early under naive prompting but continues growing under DCE, indicating sustained conceptual exploration.

Multi-seed methods (mean $\pm$ std)
Method	Seeds	EDV Retention	Collapse
Naive	3	20.4 $\pm$ 6.1%	5.6 $\pm$ 2.0%
VTS only	2	22.4 $\pm$ 1.3%	3.2 $\pm$ 1.2%
VTS + dedup	3	22.1 $\pm$ 2.7%	0.0 $\pm$ 0.0%
DCE (full)	3	23.6 $\pm$ 3.5%	0.0 $\pm$ 0.0%
Single-seed ablations

Table 2: Ablation results on the packaging domain across 200 batches. The top four methods report mean

\pm

std over 2–3 random seeds (see Seeds column); ablation-only methods (below the rule) use a single seed. EDV retention is the ratio of last-batch to first-batch EDV. Methods using dedup achieve 0% collapse consistently; methods without it do not. DCE achieves both the highest mean EDV retention and zero collapse across all seeds.

Figure 7: EDV over 200 batches (10-batch rolling averages; raw values shown faintly), split by deduplication status. Left: methods without dedup all show declining EDV; the smoothed DCE reference (gray dashed) is clearly above all of them by batch 50. Right: methods with dedup achieve 0% collapse; DCE (thick line) maintains the highest EDV throughout, confirming that prompt evolution adds value beyond dedup alone.

Method	Clusters@50	Clusters@100	Clusters@200
Naive	2	8	2
VTS only	2	6	14
VTS + dedup	5	5	2
DCE (full)	3	8	18
Dedup only	5	2	9
Prompt evo only	3	8	17

Table 3: HDBSCAN cluster counts at batch milestones (MiniLM embeddings, min_cluster_size

{}=5

). By batch 200, DCE and methods with prompt evolution maintain 17–19 clusters, while naive generation collapses to just 2. Cluster counts are sensitive to min_cluster_size : at

\{5,7,10\}

DCE produces more clusters than naive, but at

\texttt{min\_cluster\_size}{=}3

naive exceeds DCE (32.7 vs. 19.7), likely because fine-grained clustering fragments naive’s repetitive output into many small groups. Absolute counts across

\{3,5,7,10\}

: DCE 19.7, 17.7, 11.7, 5.0; naive 32.7, 11.0, 8.0, 4.3. Across seeds 42, 123, 456, DCE produces 18, 18, 17 clusters respectively (naive: 2, 14, 17), with DCE showing notably lower variance.

Method	EDV ret (OpenAI)	EDV ret (MiniLM)
Naive	27.0%	32.7%
VTS only	23.8%	28.0%
VTS + dedup	18.3%	19.7%
DCE (full)	28.5%	28.7%

Table 4: EDV validation: original OpenAI embeddings vs. independent MiniLM embeddings (seed 42; Table 2 reports multi-seed means). The bottom two ranks are stable across embeddings (VTS + dedup lowest, VTS only third), but DCE and naive swap at the top: DCE leads under OpenAI embeddings while naive leads under MiniLM.

	Dedup accept	Dedup reject
VTS accept ( $P<0.10$ )	144 (14.4%)	4 (0.4%)
VTS reject ( $P\geq 0.10$ )	826 (82.6%)	26 (2.6%)

Table 5: VTS vs. dedup decision analysis on naive generation data (1,000 ideas). VTS and dedup target largely non-overlapping populations: 96.9% of VTS-rejected ideas are semantically novel (would pass dedup), confirming that VTS filters on predictability, not redundancy.

Domain	Method	Ideas accepted	EDV retention	Collapse rate
Packaging	Naive	1,000	27.0%	4.0%
Packaging	VTS + dedup	943	18.3%	0.0%
Packaging	DCE	950	28.5%	0.0%
Education	Naive	1,000	16.8%	34.0%
Education	VTS + dedup	845	25.0%	0.0%
Education	DCE	919	26.2%	0.0%

Table 6: Cross-domain results (200 batches each, seed 42; Table 2 reports multi-seed means). DCE achieves 0% collapse across all three domains. The educational domain exhibits the most severe naive collapse (34%), making DCE’s intervention most valuable there. Creative writing falls between the other two domains in natural redundancy.

Model	Method	Ideas accepted	Dedup rejection rate
GPT-5-mini	Naive	1,000	—
GPT-5-mini	VTS + dedup	943	5.7%
GPT-5-mini	DCE	950	5.0%
Claude Haiku 4.5	Naive	1,000	—
Claude Haiku 4.5	VTS + dedup	699	30.1%
Claude Haiku 4.5	DCE	890	11.0%

Table 7: Model comparison on the packaging domain (200 batches). Claude Haiku has substantially higher dedup rejection rates than GPT-5-mini, indicating it generates more repetitive content. DCE’s benefit is even more pronounced: it reduces rejection from 30.1% to 11.0%, a 19-point improvement.

Split (explore/exploit)	EDV retention	Collapse rate
0/100 (all exploitation)	22.2%	0.0%
20/80	30.0%	0.0%
30/70	23.2%	0.0%
40/60 (default)	27.1%	0.0%
50/50	27.7%	0.0%
60/40	26.9%	0.0%

Table 8: Sensitivity to the exploration-exploitation split. All configurations achieve 0% collapse, and EDV retention varies within a moderate range (22–30%), indicating that DCE is robust to this parameter.

$\tau$	Ideas accepted	EDV retention	Collapse rate
0.05	564	31.4%	0.0%
0.10 (default)	950	28.5%	0.0%
0.20	941	27.0%	0.0%

Table 9: Sensitivity to the VTS threshold

\tau

. All values achieve 0% collapse. Lower

\tau

is more aggressive (fewer ideas pass), while higher

\tau

admits more ideas at the cost of slightly reduced depth.

$\delta$	Ideas accepted	EDV retention	Collapse rate
0.80	869	27.5%	0.0%
0.85 (default)	950	28.5%	0.0%
0.90	994	21.4%	0.0%
0.95	999	25.5%	2.5%

Table 10: Sensitivity to the dedup threshold

\delta

. Stricter thresholds (lower

\delta

) reject more ideas, improving diversity but reducing dataset size. Relaxed thresholds (higher

\delta

) retain more ideas while still preventing the worst collapse.

Figure 8: Diversity-quantity tradeoff across

\delta

values. Bars show accepted ideas (left axis); markers show EDV retention at each tested threshold (right axis). The dashed line marks the 1,000-idea baseline. At

\delta=0.95

, collapse appears (2.5%), marking the upper bound of safe relaxation.

Strategy	Accept %	Mean EDV	Batch Novelty
Gap targeting	90.8%	0.265	0.277
Assumption inversion	94.0%	0.280	0.293
Cross-industry stimulus	97.2%	0.299	0.312
Constraint variation	98.0%	0.305	0.319

Table 11: Per-strategy performance in the packaging domain (seed 42, MiniLM embeddings). Constraint variation and cross-industry stimulus produce the highest novelty; gap targeting produces the lowest but serves a complementary role by steering toward underrepresented categories.

		Controlled (top- $K$ )		Unconstrained
Domain	Method	Train/Val	F1	Train/Val	F1
Packaging	Naive	245/62	17.1%	800/200	15.2%
Packaging	VTS + dedup	160/40	15.8%	754/189	6.2%
Packaging	DCE	152/38	19.3%	760/190	30.5%
Education	Naive	355/89	83.8%	688/172	33.2%
Education	VTS + dedup	147/37	43.5%	508/127	7.2%

Table 12: Downstream classification (DeBERTa-base, 3 epochs). Left columns: category-controlled (top-

K

); right columns: unconstrained. With class count held constant, DCE produces the best classifier in packaging and writing. In unconstrained mode, DCE doubles naive F1 on packaging (30.5% vs. 15.2%). Education is limited by training set size; relaxing

\delta

to 0.90 resolves this (44.9% F1).

Figure 9: UMAP ( McInnes et al., 2018 ) projection of generated ideas under naive prompting (left) vs. DCE (right), colored by batch number on a diverging scale (blue = early, red = late). Density contours show the spatial extent of early-batch (blue, batches 1–50) and late-batch (red, batches 151–200) subsets. Under naive prompting, the contours overlap heavily, confirming mode collapse. Under DCE, the contours cover distinct territory, indicating sustained exploration. Star markers show subset centroids; $d$ is the Euclidean distance between them in UMAP space.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Component	Parameter	Value
Generator (primary)	Model	gpt-5-mini-2025-08-07
Generator (primary)	Batch size	5
Generator (primary)	Temperature	(not configurable)
Generator (cross-model)	Model	claude-haiku-4-5-20251001
Generator (cross-model)	Max tokens	4096
Generator (cross-model)	Structured output	JSON schema in system prompt

Appendix

Table 13: Complete hyperparameter listing.

Repetition in Language Generation

See all 23 papers

Aug 24, 2026

Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time

Large Language Models generate text autoregressively, but open-ended generation is prone to verbatim looping, in which models repeat spans already present in context. Standard defenses such as repetition, presence, and frequency penalties and n-gram blocking act on token recurrence rather than the sequential structure of a loop, and often suppress looping only at strengths that also degrade formatting or fluency. We propose Don't Repeat Yourself (DRY), a sampling-time logit adjustment that penalizes a candidate token only when generating it would extend the current suffix into an exact continuation of a span seen earlier in the context. Sequence breakers protect chat templates and formatting tokens. Across models from 1.5B to 120B parameters, nine prompt families, and a 600-pair human study, DRY reduces suffix-extension rate by 47% while improving lexical diversity. An intervention-matched placebo produces no comparable reduction, identifying suffix matching as the operative mechanism. On AWQ-quantized 70B and 120B models, DRY reduces loop rate by roughly half while preserving MT-Bench, MMLU, and GSM8K performance, whereas standard alternatives lose measurable ground. DRY has been adopted by popular open-source LLM inference frameworks including llama.cpp, ExLlamaV2, and text-generation-webui, highlighting its practical impact on text generation.

Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder +2

Thoughtworks · New York University

Repetition in Language Generation

Ask PDF

Oct 1, 2025

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling, a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., "Generate 5 jokes about coffee and their corresponding probabilities"). Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

Jiayi Zhang, Simon Yu, Derek Chong +4

Northeastern University · Stanford University · West Virginia University

LLM Prompting Model Collapse Diverse Text Generation Language Model Post-Training Output Diversity in Language Models

Ask PDF

Oct 1, 2026

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations that scales with model capability. Across open-ended domains (in-the-wild chat, creative writing, planning for image generation, and protein design), Gacha Decoding significantly outperforms existing generation diversity approaches at equal quality (up to 2.4x Vendi over the next-best prior approach), reaching the same number of high-quality modes with over an order of magnitude fewer samples (11.0x) and discovering novel modes that no other approach surfaces. Our key insight is to treat diversity as an instruction-following problem: rather than relying on the LM's token entropy, we combine its instruction-following capability with randomness from an external RNG tool to scalably identify and realize distinct modes of the response space. This approach of "planning with dice" enables Gacha to invert the long-observed tension between diversity and model capability. As the underlying LM becomes a better instruction follower, diversity under Gacha Decoding consistently improves--even as its token entropy and diversity under prior approaches decline. Together, our results highlight that instruction following, rather than token entropy alone, can drive generation diversity.

Scott Geng, Yufei Zhang, Joseph Lee +3

University of Washington · Meta Superintelligence Labs

Language Model Decoding Diverse Text Generation Output Diversity in Language Models

Ask PDF

Abstract

Repetition in Language Generation

Explore similar work

Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following