DiRAFT: Privacy-Aware Financial Transcript Synthesis from Aggregate Discourse Priors
Organizations: The Vanguard Group Malvern, Pennsylvania, USA
Abstract
Using customer conversations to train and evaluate financial language models raises confidentiality concerns because non-public financial facts can remain after direct identifiers are masked. Synthetic transcripts offer an alternative data source, but their discourse fidelity and residual confidentiality risks require assessment. We introduce DiRAFT (Discourse Realization from Aggregate First-order Transitions), which extracts financial concept graphs and records the order in which concepts are first introduced. It pools adjacent concept transitions across training records and samples new graph plans from aggregate transition, starting-concept, and plan-length priors. A discourse layer guides text realization without supplying an individual source passage or graph. We assess privacy-related risks through two complementary evaluations: corpus-level structural overlap and source-grounded checks for specific non-public investor facts. The structural assessment covers two financial corpora and three generation backends. The confidential-fact assessment uses human review of 1,400 investor-call outputs across seven strategies and two backends. We also evaluate transition fidelity, content recoverability, and trajectory diversity. Stochastic sampling provides a favorable balance between transition fidelity and structural overlap, with consistent advantages over greedy and top-3 on investor calls. The confidential-fact assessment likewise detects fewer matches for stochastic sampling than these alternatives on both backends covered by the fact assessment. These complementary assessments provide empirical evidence on residual confidentiality risks without establishing a formal guarantee.
Figures & tables
| Privacy | Transition fidelity | |||||
|---|---|---|---|---|---|---|
| Flagged structural overlap (%) | Detected confidential-fact matches (%) | Weighted transition JSD | ||||
| Strategy | Opus 4.6 | GPT-5.4-mini | Opus 4.6 | GPT-5.4-mini | Opus 4.6 | GPT-5.4-mini |
| Stochastic | 2.1 | 7.2 | 2.0 (0.6–7.0) | 2.0 (0.6–7.0) | 0.563 | 0.478 |
| Greedy | 20.3 | 53.2 | 16.0 (10.1–24.4) | 17.0 (10.9–25.5) | 0.710 | 0.722 |
| Top-3 | 13.1 | 38.4 | 9.0 (4.8–16.2) | 13.0 (7.8–21.0) | 0.649 | 0.621 |
| Uniform-random | 0.6 | 0.4 | 1.0 (0.2–5.4) | 0.0 (0.0–3.7) | 0.931 | 0.988 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Illustrative confidential fact | Candidate search anchors |
|---|---|---|
| Portfolio positions and allocations | The client holds 1,250 units of Fund A, representing 38% of the portfolio. | Product names, verified aliases, share classes, quantities, percentages |
| Account balances and available cash | The client’s account balance is 65,000 in available cash. | Amounts, account labels, balance, available cash, cash reserve |
| Trading instructions and intentions | The client intends to sell $75,000 of Bond B after 15 October while retaining Fund A. | Instruments, sell, redeem, retain, amounts, execution dates, conditions |
| Transaction history | The client redeemed $40,000 from Fund A on 3 September and transferred the proceeds. | Product, redemption, transfer, proceeds, amount, transaction date |
| Liquidity needs and payment obligations | The client needs $85,000 for a home closing in October. | Amount, October/Oct, home/house, closing, withdrawal, payment deadline |
| Borrowing and pledged assets | The client has a $310,000 loan secured by a position in Fund A. | Loan balance, collateral, pledged assets, secured lending, product names |
| Strategy | Opus 4.6 | GPT-5.4-mini |
|---|---|---|
| Stochastic | 2.0 (0.6–7.0) | 2.0 (0.6–7.0) |
| Greedy | 16.0 (10.1–24.4) | 17.0 (10.9–25.5) |
| Top-3 | 9.0 (4.8–16.2) | 13.0 (7.8–21.0) |
| Uniform-random | 1.0 (0.2–5.4) | 0.0 (0.0–3.7) |
| Co-occurrence | 2.0 (0.6–7.0) | 3.0 (1.0–8.5) |
| Source-graph conditioning | 20.0 (13.3–28.9) | 23.0 (15.8–32.2) |
| Opus 4.6 | GPT-5.4-mini | |||||
|---|---|---|---|---|---|---|
| Strategy | Confirmed | No match | Uncertain | Confirmed | No match | Uncertain |
| Stochastic | 2 | 98 | 0 | 2 | 96 | 2 |
| Greedy | 16 | 84 | 0 | 17 | 80 | 3 |
| Top-3 | 9 | 89 | 2 | 13 | 86 | 1 |
| Uniform-random | 1 | 98 | 1 | 0 | 98 | 2 |
| Co-occurrence | 2 | 96 | 2 | 3 | 95 | 2 |
| Dataset | Backend | Strategy | Avg Entity Jaccard | Avg Transition Jaccard | Avg LCT | Flagged (%) |
|---|---|---|---|---|---|---|
| Investor calls | Opus 4.6 | Stochastic | 0.213 | 0.037 | 0.38 | 2.1 |
| Greedy | 0.314 | 0.072 | 0.62 | 20.3 | ||
| Top-3 | 0.296 | 0.057 | 0.51 | 13.1 | ||
| Uniform-random | 0.140 | 0.009 | 0.10 | 0.6 | ||
| Co-occurrence | 0.205 | 0.029 | 0.16 | 3.3 | ||
| Source-graph conditioning | 0.290 | 0.041 | 0.69 | 14.3 |
| 1 | Extract concept graphs and ordered concept sequences from using |
|---|---|
| 2 | Estimate transition prior , start prior , and length prior |
| 3 | Estimate aggregate script priors from |
| 4 | |
| 5 | for : |
| 6 | Sample and |
| 7 | Initialize content plan |
| Control | Description | Setting options |
|---|---|---|
| Communicative framing (sampled from empirical priors) | ||
| Call goal | Dialogue purpose | Initial consultation, follow-up, portfolio review, specific inquiry, relationship check |
| Speaker pattern | Dialogue driver | Financial Advisor-led, Investor-led, balanced |
| Tone | Communication style | Exploratory, informational, transactional, consultative |
| Structural extent | ||
| Turn budget | Length and turn allocation | Learned from empirical data |
| Investor calls | S&P 500 earnings calls | |||||
| Method | Opus | GPT | Haiku | Opus | GPT | Haiku |
| Transition fidelity (weighted Jensen–Shannon divergence) | ||||||
| Stochastic sampler | 0.563 | 0.478 | 0.482 | 0.592 | 0.665 | 0.727 |
| Greedy sampler | 0.710 | 0.722 | 0.697 | 0.719 | 0.696 | 0.749 |
| Top-3 sampler | 0.649 | 0.621 | 0.611 | 0.653 | 0.674 | 0.726 |
| Uniform-random baseline | 0.931 | 0.988 | 0.945 | 1.000 | 1.000 | 1.000 |
| Investor calls | S&P 500 earnings calls | |||||
| Method | Opus | GPT | Haiku | Opus | GPT | Haiku |
| Node recoverability (macro-averaged node F1) | ||||||
| Stochastic sampler | 0.912 | 0.986 | 0.958 | 0.886 | 0.974 | 0.940 |
| Greedy sampler | 0.909 | 0.992 | 0.964 | 0.963 | 0.995 | 0.981 |
| Top-3 sampler | 0.914 | 0.986 | 0.962 | 0.900 | 0.984 | 0.936 |
| Uniform-random baseline | 0.918 | 0.990 | 0.963 | 0.964 | 0.994 | 0.982 |
| Planned | Recovered | |||
|---|---|---|---|---|
| Method | Shared | Opus | GPT | Haiku |
| Stochastic sampler | 0.937 | 0.971 | 0.964 | 0.969 |
| Greedy sampler | 0.284 | 0.722 | 0.417 | 0.654 |
| Top-3 sampler | 0.467 | 0.807 | 0.623 | 0.756 |
| Uniform-random baseline | 1.000 | 0.995 | 1.000 | 0.990 |
| Co-occurrence sampling | 0.998 | 0.990 | 0.996 | 0.990 |
| Method / Text source | Backend | Concept | Transition | ||
|---|---|---|---|---|---|
| Precision | Recall | Precision | Recall | ||
| Original real | — | 94.6 | 92.3 | 91.3 | 88.1 |
| Stochastic | Opus | 95.0 | 95.5 | 91.8 | 91.6 |
| Greedy | Opus | 95.6 | 94.9 | 91.9 | 91.7 |
| Top-3 | Opus | 94.2 | 92.2 | 91.0 | 88.6 |
| Uniform-random | Opus | 92.4 | 91.8 | 88.2 | 87.9 |
| Backend | Original preferred | Tie | Shuffled preferred | Tie-adjusted preference (95% CI) |
|---|---|---|---|---|
| Opus 4.6 | 97/200 (48.5%) | 65/200 (32.5%) | 38/200 (19.0%) | 64.75% [58.00%, 71.25%] |
| GPT-5.4-mini | 90/200 (45.0%) | 67/200 (33.5%) | 43/200 (21.5%) | 61.75% [55.25%, 68.00%] |
| Metric | Opus 4.6 | GPT-5.4-mini |
|---|---|---|
| Annotator 1: Original / Tie / Shuffled | 51 / 28 / 21 | 47 / 35 / 18 |
| Annotator 2: Original / Tie / Shuffled | 46 / 37 / 17 | 43 / 32 / 25 |
| Annotator 1: Tie-adjusted preference | 65.0% | 64.5% |
| Annotator 2: Tie-adjusted preference | 64.5% | 59.0% |
| Observed agreement | 71/100 (71.0%) | 61/100 (61.0%) |
| Expected chance agreement | 37.39% | 35.91% |
| Baseline | Stochastic preferred | Tie | Baseline preferred | Tie-adjusted preference (adjusted interval) |
|---|---|---|---|---|
| Scenario-only prompting | 119/200 (59.5%) | 63/200 (31.5%) | 18/200 (9.0%) | 75.25% [68.50%, 81.50%] |
| Source-graph conditioning | 112/200 (56.0%) | 66/200 (33.0%) | 22/200 (11.0%) | 72.50% [65.50%, 79.00%] |
| Co-occurrence | 97/200 (48.5%) | 57/200 (28.5%) | 46/200 (23.0%) | 62.75% [54.50%, 70.75%] |
| Metric | Scenario-only prompting | Source-graph conditioning | Co-occurrence |
|---|---|---|---|
| Annotator 1: S / T / B | 59 / 33 / 8 | 57 / 32 / 11 | 52 / 27 / 21 |
| Annotator 2: S / T / B | 60 / 30 / 10 | 55 / 34 / 11 | 45 / 30 / 25 |
| Annotator 1: Tie-adjusted preference | 75.5% | 73.0% | 65.5% |
| Annotator 2: Tie-adjusted preference | 75.0% | 72.0% | 60.0% |
| Exact agreement | 57/100 (57%) | 55/100 (55%) | 59/100 (59%) |
| Expected chance agreement | 46.10% | 43.44% | 36.75% |
| Annotator 2 | ||||
|---|---|---|---|---|
| Baseline | Annotator 1 | Stochastic | Tie | Baseline |
| Scenario-only prompting | Stochastic | 41 | 16 | 2 |
| Tie | 17 | 12 | 4 | |
| Baseline | 2 | 2 | 4 | |
| Source-graph conditioning | Stochastic | 37 | 17 | 3 |
| Tie | 16 | 13 | 3 | |
| Method | Measure | Before | After | Approximate increase (pp) |
|---|---|---|---|---|
| DiRAFT | Full-path reproduction | 0.00 | 0.20 | +0.20 |
| Source-path conditioning | Full-path reproduction | 0.00 | 1.25 | +1.25 |
| DiRAFT | Transition coverage | 0.00 | 0.25 | +0.25 |
| Source-path conditioning | Transition coverage | 0.00 | 1.25 | +1.25 |
| DiRAFT | Concept coverage | 8.50 | 9.34 | +0.84 |
| Source-path conditioning | Concept coverage | 8.50 | 9.54 | +1.04 |
| Method | Measure | Before | After | Approximate increase (pp) |
|---|---|---|---|---|
| DiRAFT | Full-path reproduction | 0.02 | 0.12 | +0.10 |
| Source-path conditioning | Full-path reproduction | 0.02 | 0.83 | +0.81 |
| DiRAFT | Transition coverage | 0.35 | 0.43 | +0.08 |
| Source-path conditioning | Transition coverage | 0.35 | 1.08 | +0.73 |
| DiRAFT | Concept coverage | 7.25 | 7.55 | +0.30 |
| Source-path conditioning | Concept coverage | 7.25 | 8.28 | +1.03 |
Synthetic Data Generation for Language Models
Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.
See all 304 papers