PersonaTrace: Synthesizing Realistic Digital Footprints with LLM Agents
Organizations: Apple · Harvard University
Abstract
Digital footprints (records of individuals' interactions with digital systems) are essential for studying behavior, developing personalized applications, and training machine learning models. However, research in this area is often hindered by the scarcity of diverse and accessible data. To address this limitation, we propose a novel method for synthesizing realistic digital footprints using large language model (LLM) agents. Starting from a structured user profile, our approach generates diverse and plausible sequences of user events, ultimately producing corresponding digital artifacts such as emails, messages, calendar entries, reminders, etc. Intrinsic evaluation results demonstrate that the generated dataset is more diverse and realistic than existing baselines. Moreover, models fine-tuned on our synthetic data outperform those trained on other synthetic datasets when evaluated on real-world out-of-distribution tasks.
Figures & tables
| Dataset | Pairwise Corr. (↓) | Remote-Clique (↑) | Entropy (↑) | Avg. #Links | Avg. Length | |
|---|---|---|---|---|---|---|
| FinePersonas-Email | 0.2333 | 0.7680 | 2.8080 | 0.0000 | 681.50 | |
| IWSPA-2023-Adversarial | 0.4079 | 0.5919 | 2.5094 | 0.0000 | 276.48 | |
| LLM-Gen Phishing | 0.3094 | 0.6906 | 2.6969 | 0.2522 | 685.58 | |
| Synthetic-Satellite-Emails | 0.5416 | 0.4586 | 2.8218 | 0.0000 | 1050.76 | |
| Synthetic | PersonaTrace (Proposed) | 0.2093 | 0.7898 | 2.8305 | 0.2532 | 1437.87 |
| Enron | 0.2798 | 0.7218 | 2.7110 | 0.0000 | 2002.57 | |
| Dataset | Tone | Fluency | Coherence | Informativeness | Engagement | Overall | |
|---|---|---|---|---|---|---|---|
| Synthetic | FinePersonas-Email | 4.52 | 4.92 | 4.56 | 3.98 | 3.98 | 4.39 |
| IWSPA-2023-Adversarial | 1.59 | 1.91 | 1.67 | 1.31 | 1.22 | 1.53 | |
| LLM-GenPhishing | 3.41 | 4.89 | 4.67 | 3.13 | 2.78 | 3.70 | |
| Synthetic-Satellite-Emails | 4.82 | 4.96 | 4.86 | 4.91 | 3.65 | 4.64 | |
| PersonaTrace (Proposed) | 4.95 | 4.99 | 4.99 | 4.92 | 4.09 | 4.79 | |
| Enron | 4.19 | 4.73 | 4.47 | 4.28 | 2.71 | 4.07 | |
| Training Set | Enron | Human-Gen Phishing | Private | Private w/o Spam | W3C-Emails | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Email Categorization | ||||||||||
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | |
| None | 0.2741 | 0.0722 | 0.0818 | 0.0236 | 0.0011 | 0.0022 | 0.0025 | 0.0028 | 0.0011 | 0.0004 |
| FinePersonas-Email | 0.5908 | 0.1810 | 0.2241 | 0.0755 | 0.0015 | 0.0027 | 0.0020 | 0.0022 | 0.5401 | 0.1093 |
| IWSPA-2023-Adversarial | 0.0010 | 0.0038 | 0.0007 | 0.0004 | 0.0012 | 0.0023 | 0.0035 | 0.0039 | 0.0011 | 0.0022 |
| LLM-Gen Phishing | 0.4046 | 0.1095 | 0.1363 | 0.0376 | 0.0008 | 0.0016 | 0.0015 | 0.0017 | 0.1909 | 0.0523 |
| Training Set | DailyDialog | Persona-Chat | Private | u-sticker | ||||
|---|---|---|---|---|---|---|---|---|
| Acc | BertS | Acc | BertS | Acc | BertS | Acc | BertS | |
| None | 0.1202 | 0.0237 | 0.1072 | 0.1521 | 0.0958 | 0.1102 | 0.0933 | 0.0849 |
| DailyConversations | 0.1091 | 0.1867 | 0.1070 | 0.2195 | 0.0961 | 0.2576 | 0.0933 | 0.2976 |
| DiaSynth | 0.1182 | 0.1334 | 0.1089 | 0.2036 | 0.0961 | 0.1698 | 0.0875 | 0.1666 |
| PersonaBench | 0.1253 | 0.0328 | 0.1064 | 0.1663 | 0.0960 | 0.1246 | 0.0930 | 0.0986 |
| Synthetic-Persona-Chat | 0.1101 | 0.2143 | 0.0982 | 0.2272 | 0.0955 | 0.2954 | 0.0893 | 0.3306 |
| Task | w/o Agents | w/ Agents | |
|---|---|---|---|
| Acc | 0.0063 | 0.2733 | |
| Email Categorization | F1 | 0.0057 | 0.0813 |
| ROUGE | 0.0376 | 0.1527 | |
| Email Drafting | BertS | 0.3012 | 0.4590 |
| ROUGE | 0.0413 | 0.2500 | |
| Question Answering | BertS | 0.2880 | 0.4405 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Synthetic Data Generation
Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.
See all 623 papers