One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction

Hongliang Li; Lu Wang; Yong Xu; Hanyang Chen; Zhitao Hou; Xiaoting Qin; Song Ge; Qingwei Lin

cs.AISep 18, 2026

One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction

Authors: Hongliang Li, Lu Wang, Yong Xu, Hanyang Chen, Zhitao Hou, Xiaoting Qin, Song Ge, Qingwei Lin,

Organizations: Microsoft · Beijing Jiaotong University

Abstract

Large language models (LLMs) are increasingly deployed for enterprise information extraction (IE), where the same document must be reorganized differently for each user. Existing prompt optimization methods, however, rely on a single prompt optimized against a global objective, which is misaligned with the inherent user heterogeneity of real workplaces. We formulate enterprise IE as per-user prompt adaptation under interaction feedback and propose Self-Meta-Evolve, a hierarchical framework that maintains a dedicated prompt for each user and continuously refines it through a dual-loop process: an inner loop that edits structured prompts based on persona-conditioned feedback, and an outer loop that evolves the meta-prompt itself by distilling successful editing patterns. To enable scalable training and evaluation, we release a persona-driven IE benchmark of 292 simulated enterprise users, paired with a reproducible persona-generation pipeline grounded in O*NET occupational taxonomies. On this benchmark, Self-Meta-Evolve achieves a 74.58% success rate, outperforming the strongest prompt-optimization baseline by 13.56 absolute points, and reaches 52.54% within only two iterations. A double-blind human study with twenty real professionals further confirms that prompts adapted by our framework win against static baselines in 71% of pairwise comparisons.

Figures & tables

Figure 1: Static prompts fail under user-specific extraction objectives. Conventional systems apply a shared prompt to all users, often producing outputs that do not match heterogeneous role-specific preferences. Our framework maintains a dedicated prompt for each user and updates it from interaction feedback, enabling personalized extraction over time.

Figure 2: Architecture of Self-Meta-Evolve. The inner loop iteratively refines a user’s structured extraction prompt based on AI-User feedback, while the outer loop evolves the meta-prompt by distilling successful editing trajectories across users.

Method	SR $\uparrow$	ML $\downarrow$	SR@2	SR@4	SR@8
ProTeGi	0.610	0.754	0.390	0.475	0.542
OPRO	0.458	1.124	0.271	0.339	0.424
EvoPrompt	0.407	1.286	0.220	0.305	0.356
Bandit-UCB	0.542	0.941	0.322	0.424	0.492
APE	0.339	1.423	0.203	0.271	0.305
MetaSPO	0.373	1.331	0.237	0.305	0.339

Table 1: Main comparison on the held-out evaluation set (

N=59

). SR@

t

denotes cumulative success rate at step

t

. Self-Frozen disables the outer loop.

Backbone	ProTeGi	Ours	$\Delta$
GPT-5.5	64.41	77.97	13.56
GPT-5.1	61.02	74.58	13.56
GPT-4.1	52.54	64.41	11.87
Claude-Opus-4.8	62.71	76.27	13.56
Claude-3.7-Sonnet	54.24	69.49	15.25

Table 2: Additional backbone evaluation: SR (%) on the held-out set (

N=59

);

\Delta

is the gain in percentage points.

Figure 3: Optimization trajectories. Self-Meta-Evolve maintains higher SR and lower loss across 20 steps. Markers at

t=2,4,8

correspond to Table 1 .

Method	Tok/Iter $\downarrow$	Total (M) $\downarrow$	Tok/%SR $\downarrow$
ProTeGi	6.5K	0.195	3.19K
OPRO	15.8K	0.474	10.35K
EvoPrompt	18.5K	0.555	13.64K
APE	4.8K	0.144	4.24K
Self-Frozen	8.2K	0.246	4.27K
Self-Meta-Evolve	12.4K	0.372	4.98K

Table 3: Efficiency: API cost per iteration and per absolute %SR over 30 iterations.

Method	STEM		Hum.		$\Delta$
Method	SR	ML	SR	ML	$\downarrow$
ProTeGi	0.620	0.742	0.540	0.915	8.0
OPRO	0.480	1.095	0.340	1.380	14.0
Bandit-UCB	0.560	0.960	0.420	1.225	14.0
APE	0.380	1.435	0.220	1.760	16.0
Self-Meta-Evolve	0.760	0.492	0.720	0.574	4.0

Table 4: Cross-persona generalization (

N=50

per group).

\Delta

: drop from STEM to Humanities.

Variant	ML $\downarrow$	SR $\uparrow$	SR@2	SR@4	SR@8
Full	0.512	0.746	0.525	0.610	0.678
$-$ Outer Loop	0.865	0.576	0.407	0.492	0.525
$-$ Success Buf.	0.685	0.661	0.458	0.542	0.593
$-$ Edit Plan	0.621	0.695	0.492	0.576	0.627

Table 5: Ablation on the held-out evaluation set.

#Personas	50	100	150	200	250	292
SR	0.542	0.627	0.699	0.729	0.742	0.746
ML	0.901	0.732	0.611	0.548	0.524	0.512

Table 6: Persona scaling. Returns saturate around 200–250 personas; 292 sits above the knee.

$(\lambda_{r},\lambda_{d})$	(0.1,0.1)	(0.1,0.15)	(0.1,0.2)	(0.1,0.25)
SR (Ours)	0.737	0.741	0.746	0.744
SR (ProTeGi)	0.604	0.607	0.610	0.609

Table 7: Sensitivity to feedback weights. Rankings and absolute SR are stable in the studied range.

Dimension	Real	Synth.	$\Delta$	$p$
Role consistency	4.78	4.74	$-$ 0.04	0.68
Technical depth	4.62	4.65	$+$ 0.03	0.74
Format realism	4.88	4.82	$-$ 0.06	0.52
Language naturalness	4.70	4.68	$-$ 0.02	0.81
Overall	4.75	4.72	$-$ 0.03	0.69

Table 8: Persona realism (5 evaluators, 100 samples; independent

t

-test). Cohen’s

\kappa=0.71

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Prompt used in Stage 2 to expand each O*NET role into a persona.

Parameter	Value
Max optimization iterations $T$	30
Max inner-loop steps per persona	8
Success buffer size $K$	20
Meta-evolution frequency	every 5 iterations
Temperature	0.0
Max generation tokens	2048

Appendix

Table 9: Hyperparameter configuration.

Panel of Figure 5 — Figure 5: Abridged initial extraction prompt.

Panel of Figure 6 — Figure 6: AI-User feedback prompt.

Panel of Figure 7 — Figure 7: Inner-loop edit-plan prompt.

Panel of Figure 8 — Figure 8: Outer-loop meta-evolution prompt.

Figure 9: Distribution of human perception in the AI-User authenticity test (

N=100

Panel of Figure 10 — Figure 10: Example synthetic enterprise artifacts.

Panel of Figure 11 — Figure 11: Output JSON schema.

Dataset	Initial	Ours	$\Delta$
CUAD	35.0	55.0	20.0
SciREX	30.0	60.0	30.0
Aggregate	32.5	57.5	25.0

Appendix

Table 10: Public-document pilot: SR (%) over 20 documents

\times

5 objectives per dataset.

\Delta

is in percentage points.

Method	SR (%)	ML
Initial Prompt	88.0	0.220
Self-Frozen	90.0	0.185
Self-Meta-Evolve	91.5	0.176

Appendix

Table 11: Additional evaluation with a shared extraction objective.

Automatic Prompt Optimization

Automatic prompt optimization uses model or task feedback to revise instructions for stronger, more reliable language-model performance.

See all 103 papers

Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks
Apr 13, 2026Yuqing Yang, Tengxiao Liu, Wang Bill Zhu +3LLM Input Optimization LLM Memory
University of Southern California · University of California, Santa Barbara
POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM Personalization Personalized Language Models
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses
Mar 11, 2026Minki Hong, Eunsoo Lee, Sohyun Park +1LLM Prompting LLM Evaluation
Department of Computer Science and Artificial Intelligence, Dongguk University, South Korea

Apr 13, 2026J/K move · Enter open · S save

Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks

Yuqing Yang, Tengxiao Liu, Wang Bill Zhu, Taiwei Shi+2

University of Southern California · University of California, Santa Barbara

LLM Input Optimization LLM Memory Persistent Memory for Language Models Automatic Prompt Optimization Language Model Self-Improvement LLM Evaluation

Open paper AskPDF arXiv

As LLM-based assistants become persistent and personalized, they must extract and retain useful information from past conversations as memory. However, the types of information worth remembering vary considerably across tasks. We formalize the heterogeneous memory extraction task and introduce BEHEMOTH, a benchmark that repurposes 18 existing datasets spanning personalization, problem-solving, and agentic tasks, using a downstream utility-driven metric for systematic evaluation. Our empirical analysis confirms that no single static extraction prompt dominates across all task categories, and that existing self-evolving prompt optimization frameworks, originally designed for homogeneous distributions, degrade when training tasks are heterogeneous. To address this, we propose CluE, a cluster-based self-evolving strategy that groups training examples into clusters by extraction scenarios, analyzes each cluster independently, and synthesizes cross-cluster insights to update the extraction prompt. Experiments on BEHEMOTH show that CluE generalizes effectively across heterogeneous tasks (

+

9.04% relative gain), consistently outperforming prior self-evolving frameworks.

Abstract

Automatic Prompt Optimization

Explore similar work

Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses