OSynth: A Longitudinal, Context-Sensitive Evaluation of Meeting Understanding

Ian Porada; Suraj Tripathi; Chaitanya Animesh; Leda Sari; Chaitanya Narisetty; Austin R. Matthews; Bohan Yu; Ramesh L. Thulasiram

cs.CLOct 21, 2026

OSynth: A Longitudinal, Context-Sensitive Evaluation of Meeting Understanding

Authors: Ian Porada, Suraj Tripathi, Chaitanya Animesh, Leda Sari, Chaitanya Narisetty, Austin R. Matthews, Bohan Yu, Ramesh L. Thulasiram,

Organizations: Otter.ai Mountain View, California, USA

Abstract

Meeting notetakers are increasingly deployed in settings where longitudinal context is available including calendar metadata, prior meetings, emails, slides, and speaker enrollments; however, meeting corpora remain limited in terms of size, domain coverage, and available context. We present OSynth, a large-scale synthetic dataset of simulated organizations comprising 570 hours of meeting audio together with associated artifacts and a latent ground-truth state. In a pairwise comparison, human raters distinguish OSynth meetings from real meeting transcripts with only 56% accuracy based on excerpts of 150-300 words. Across state-of-the-art LLMs, additional context yields significant gains on several entity-sensitive dimensions, demonstrating limitations of contemporary approaches and the potential usefulness of OSynth in developing better-contextualized meeting-understanding systems.

Figures & tables

Figure 1: Each OSynth organization, termed a workspace , is generated from a single shared latent state from which all artifacts (audio, transcripts, etc.) are derived. Meeting transcripts are adversarially refined to improve realism, and in experiments multimodal large language models (MLLMs) improve significantly in certain dimensions when given access to broader workspace context (L3) as compared to a single meeting’s audio (L1).

Corpus	Hours	Domain	Reference	Contextual artifacts
ICSI	72	Academic	gold + recorded audio	prior meetings (recurring groups)
AMI	100	Product design	gold + recorded audio	slides (subset), whiteboard video, prior meetings (scenario subset)
AliMeeting	119	Multi-domain (zh)	gold + recorded audio	—
ELITR-Minuting	80	Technical	gold, no audio	minutes
MeetingBank	3,579	City council	ASR + recorded audio	agenda, prior meetings
NOTSOFAR-1	32	Office meetings	gold + recorded audio	—

Table 1: OSynth compared to existing meeting corpora. Reference encodes the transcription process ( gold vs. ASR ) and audio source ( recorded vs. synthetic ); OSynth is the only synthetic corpus in this table and is validated against real meeting transcripts by the human-discrimination study in § 3.5 .

Figure 2: Overview of the OSynth synthetic data pipeline. A hierarchical LLM pipeline samples a workspace-level latent state, expands it into meetings and surrounding artifacts, renders the artifacts into audio and text, and maintains a rolling WorkspaceState so that every observable can be scored against an exact reference. Artifacts emit WorkspaceEvents that are considered when generating subsequent artifacts. Audio is synthesized turn-by-turn with voice cloning and the final audio channels are simualted (virtual, hybrid, or in-person acoustics) using pyroomacoustics ( Scheibler et al., 2018 ) .

Synthetic source	Trials	Accuracy
Haiku (w/o obfuscation)	54	79.6%
OSynth	378	56.3%
Chance baseline	—	50%

Table 2: Human discrimination accuracy in the 2AFC realism study. The OSynth condition is relatively closer to chance, supporting OSynth as a more credible proxy for real workspace meetings.

System	Ctx.	WER $\downarrow$	Num. rec. $\uparrow$	Term rec. $\uparrow$	AI desc. $\uparrow$	Assignee $\uparrow$	Spk. cons. $\uparrow$
OpenAI	L1	0.162 ±.014	0.680 ±.056	0.684 ±.040	3.49 ±.06	0.485 ±.040	0.220 ±.045
	L2	0.155 ±.015	0.695 ±.056	0.708 ±.040	3.60 ±.06	0.504 ±.040	0.219 ±.048
	L3	0.144 ±.015	0.883 ±.038	0.778 ±.036	3.96 ±.05	0.690 ±.035	0.476 ±.067
Gemini	L1	0.099 ±.015	0.960 ±.022	0.809 ±.031	3.55 ±.05	0.069 ±.024	0.265 ±.050
	L2	0.099 ±.015	0.963 ±.021	0.796 ±.032	3.60 ±.05	0.071 ±.023	0.254 ±.049
	L3	0.095 ±.015	0.975 ±.017	0.841 ±.028	3.80 ±.06	0.630 ±.040	0.349 ±.068

Table 3: Main results across context levels L1 to L3 on the evaluation set. Values are means with 95% bootstrap percentile interval (1,000 resamples). Bold marks the best context level for each system. Both systems improve mostly monotonically on the entity-sensitive dimensions (number recall, technical-term recall, assignee accuracy, speaker consistency). The OpenAI stack uses gpt-4o-transcribe-diarize , gpt-audio , and gpt-5.5 ; the Gemini stack uses gemini-3-flash and gemini-3.1-pro-preview .

Metric	Type	L1	L3	$\Delta$
Number recall	iso.	0.680 ±.056	0.883 ±.038	+0.20
Technical-term rec.	iso.	0.684 ±.040	0.778 ±.036	+0.09
AI assignee accuracy	iso.	0.485 ±.040	0.690 ±.035	+0.20
Spk.-consistency rate	iso.	0.220 ±.045	0.476 ±.067	+0.26
LLM-judge faithfulness	std.	3.10 ±.09	3.40 ±.10	+0.30
LLM-judge coherence	std.	4.60 ±.06	4.91 ±.04	+0.31

Table 4: L1 to L3 gain on isomorphic latent-state metrics versus standard LLM-as-judge summarization metrics for the OpenAI system on the evaluation set including 95% bootstrap percentile intervals (1,000 resamples). The Type column marks each row as either iso. (isomorphic, scored by exact comparison against OSynth ’s latent state) or std. (standard meeting-summarization metric, scored by LLM-as-judge over the produced artifact). Gains concentrate on the entity-sensitive isomorphic dimensions (assignee accuracy, number recall, speaker consistency), which standard summary-side LLM-as-judge scores systematically compress.

Metric	L1	L3	$\Delta$
LLM-judge faithfulness	3.50	3.63	+0.13
LLM-judge coherence	5.00	5.00	+ 0.00

Table 5: Anchoring study on eight real AMI meetings (series ES2002 and ES2003), Gemini system, judged summary metrics only.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Workspaces	160
Meetings	2,126
Hours of audio	570
Meetings / workspace (mean)	13.3
Personas / workspace (mean)	12.9
Email threads / workspace (mean)	4.3
Shared documents / workspace (mean)	1.5

Appendix

Table 6: OSynth corpus statistics. Each workspace bundles meeting audio, aligned transcripts, downstream notetaking artifacts, inter-meeting workspace artifacts, and per-speaker enrollments derived from a shared latent state. Workspaces are generated with varying workspace size and time horizons, thus subsamples of the data follow distinct distributions in terms of length and content.

Date	Title	Dur	Att
05-18 09:00	Executive Leadership Sync	60	5
05-18 09:15	Engineering Daily Standup	15	4
05-18 11:00	Project Brindle Kickoff	45	4
05-19 09:15	Engineering Daily Standup	15	4
05-19 10:00	Marcus–Simone 1:1 ( $\star$ )	30	2
05-19 10:00	ISO 20022 Migration Sync	60	5

Appendix

Table 7: Three-day slice of the example workspace meeting calendar around the focus meeting (marked with

\star

). Dur is scheduled duration in minutes; Att is attendee count. The full workspace contains 28 meetings across the two-week observation window.

Configuration	Num. rec.	Term rec.	Assignee
L2 (no L3 streams)	0.637	0.723	0.524
+ enrollments only	0.605	0.657	0.698
+ workspace text only	0.975	0.739	0.752
L3 (both streams)	0.973	0.760	0.792

Appendix

Table 8: OpenAI-stack ablation on the L2

\to

L3 increment, computed on the earlier 50-meeting evaluation subset. Per-speaker voice enrollments (routed to SID) and workspace text artifacts (member list, emails, slides, agendas) contribute differently across dimensions: workspace text accounts for essentially the entire number-recall gain, while enrollments and workspace text contribute complementarily on assignee accuracy. Number- and term-recall figures use exact matching; assignee accuracy uses partial-credit substring matching.

Term	GT	L1	L2	L3
BLE	BLE	VLE	PLE	BLE ✓
ArcBus	ArcBus	ARC plus	Arctus	ArcBus ✓

Appendix

Table 9: Progressive resolution of TERM errors ( BLE , ArcBus ) across L1, L2, and L3 from a single passage. L2’s self-biased prior outputs recover some tokens ( PMIC , SN7142 ) but also introduce confident hallucinations ( Arctus ); the explicit L3 jargon list recovers the remainder.

	Meeting $n$	Meeting $n{+}1$
Title	Marcus–Devon 1:1	BDW Sandbox Scope
Date	2026-05-19	2026-05-21
L1 label	Bevan	Devin
L3 label	Devon Ashby	Devon Ashby

Appendix

Table 10: Cross-meeting labels for canonical person p8 ( Devon Ashby ) in two near-consecutive meetings in the same workspace, under OpenAI L1 versus L3. L1 hallucinates a different phonetically plausible English given name in each meeting because per-meeting diarization clusters do not persist and the L1 SID stage has no workspace name list to anchor on; L3 fixes both meetings to the same canonical identity via the workspace name list and enrollment-based SID against the persona’s prompt clip.

Measure	Prompts	OSynth	$\Delta$
Active-speech WPM	134.5	180.9	$+34.5\%$
Pause ratio	47.9%	18.9%	$-60.5\%$
Pauses $>$ 200 ms / min	44.5	15.6	$-65.0\%$

Appendix

Table 11: Pacing of the voice-cloning prompts versus the rendered OSynth turns, from NeMo forced alignments macro-averaged per speaker. The generated turns are faster and markedly more uniform in pausing.

Figure 11: Web interface used in the 2AFC human-discrimination study (§ 3.5 ). Each trial displays a pair of length-matched transcript excerpts (one real, one synthetic), with speaker labels anonymized to generic identifiers and a word-count badge above each excerpt; the annotator selects which excerpt is LLM-generated using the two-button control at the bottom. The transcript textboxes are scrollable.

Metric	Type	L1	L3	$\Delta$
Number recall	iso.	0.913	0.983	+0.07
Technical-term recall	iso.	0.756	0.802	+0.05
AI assignee accuracy	iso.	0.101	0.666	+0.57
Spk.-consistency rate	iso.	0.326	0.401	+0.08
LLM-judge faithfulness	std.	3.02	2.88	$-$ 0.14
LLM-judge coherence	std.	4.74	4.88	+0.14

Appendix

Table 12: Gemini-stack analog of Table 4 : L1

\to

L3 gain on isomorphic latent-state metrics versus standard LLM-as-judge summarization metrics, on a smaller 50-meeting subset of the evaluation data. Same qualitative pattern as OpenAI (isomorphic gains compressed by standard summary-side scores), with a single non-monotonic cell: LLM-judge faithfulness dips slightly at L3 despite uniform improvement on every isomorphic dimension. We explain this as a scoring asymmetry of standard LLM-judge protocols against Gemini’s workspace-canonical-name expansions (§ 5.2 ). Assignee accuracy here is scored under exact match; the OpenAI counterpart in Table 4 uses partial-credit substring matching.

Metric	L1	L2	L3
WER $\downarrow$	0.559	0.557	0.638
Number recall $\uparrow$	0.259	0.259	0.397
Technical-term recall $\uparrow$	0.364	0.361	0.302
AI description quality $\uparrow$	2.08	1.88	2.13
Assignee accuracy $\uparrow$	0.012	0.017	0.465
Speaker consistency $\uparrow$	0.244	0.215	0.478

Appendix

Table 13: Open-source SpeechBrain cascade on a 27-meeting single-workspace subset of the evaluation set used in Table 3 , and reported as a reproducible open-source reference point only; the subset is smaller than the 238-meeting main evaluation, so these numbers are not directly comparable cell-for-cell. The cascade trails both state-of-the-art systems on every metric reported here. Enrollment-driven metrics (assignee accuracy, speaker consistency) jump sharply at L3 once SID is enabled, but WER and technical-term recall regress at L3 because the L3 vocabulary-correction LLM pass silently summarizes around the cascade’s high-WER upstream output (see prose above).

Metric	L1	L3	$\Delta$
Faithfulness	4.00 ±.10	4.43 ±.10	+0.43
Coherence	4.99 ±.02	5.00 ±.00	+0.01
AI description quality	3.87 ±.05	4.05 ±.04	+0.18

Appendix

Table 14: Judged metrics recomputed with Qwen3-235B-Instruct as the judge, on the same 238-meeting evaluation set and with the same prompts as Table 4 (95% bootstrap percentile interval, 1,000 resamples). The direction of the L1

\to

L3 effect is preserved; coherence is saturated at the 5.00 ceiling under this prompt, which also truncates its interval.

Asset	Type	License	Use in OSynth
Qwen3-TTS ( Qwen Team, 2026 )	Model	Apache 2.0	Multi-speaker TTS with voice cloning (§ 3.2 )
SpeechBrain toolkit	Software	Apache 2.0	Open-source cascade baseline (Appendix V )
vad-crdnn-libriparty	Model	Apache 2.0	Voice-activity detection in the cascade
asr-wav2vec2-commonvoice-14-en	Model	Apache 2.0	Acoustic posteriors in the cascade
spkrec-ecapa-voxceleb	Model	Apache 2.0	Diarization and SID in the cascade
pyctcdecode	Software	Apache 2.0	CTC beam-search decoding with hot-word biasing (L3)

Appendix

Table 15: Licenses of the open-source models, software packages, and datasets used in the construction and evaluation of OSynth . SpeechBrain pretrained checkpoints listed individually inherit the Apache 2.0 license of the SpeechBrain toolkit and are distributed via the corresponding Hugging Face repositories. For NOTSOFAR-1, we follow the CC BY 4.0 license stated in the official GitHub repository.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 625 papers

Rethinking Meeting Effectiveness: A Benchmark and Framework for Temporal Fine-grained Automatic Meeting Effectiveness Evaluation
Apr 19, 2026Yihang Li, Chenhui ChuAutomated Evaluation
Kyoto University
Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting
Sep 3, 2026Muneeb Khan, Frederic Kirstein, Terry Ruas +1LLM Agent Evaluation Conversational Agents
University of Göttingen, Germany
Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding
Aug 25, 2026Quanwei Tang, Dong Zhang, Shoushan Li +1Audio QA Audio Understanding
School of Computer Science & Technology, NLP Lab, Soochow University, China · Jiangsu Key Lab of Language Computing, Suzhou.

Apr 19, 2026J/K move · Enter open · S save

Rethinking Meeting Effectiveness: A Benchmark and Framework for Temporal Fine-grained Automatic Meeting Effectiveness Evaluation

Yihang Li, Chenhui Chu

Kyoto University

Automated Evaluation

Open paper AskPDF arXiv

Evaluating meeting effectiveness is crucial for improving organizational productivity. Current approaches rely on post-hoc surveys that yield a single coarse-grained score for an entire meeting. The reliance on manual assessment is inherently limited in scalability, cost, and reproducibility. Moreover, a single score fails to capture the dynamic nature of collaborative discussions. We propose a new paradigm for evaluating meeting effectiveness centered on novel criteria and temporal fine-grained approach. We define effectiveness as the rate of objective achievement over time and assess it for individual topical segments within a meeting. To support this task, we introduce the AMI Meeting Effectiveness (AMI-ME) dataset, a new meta-evaluation dataset containing 2,459 human-annotated segments from 130 AMI Corpus meetings. We also develop an automatic effectiveness evaluation framework that uses a Large Language Model (LLM) as a judge to score each segment's effectiveness relative to the overall meeting objectives. Through substantial experiments, we establish a comprehensive benchmark for this new task and evaluate the framework's generalizability across distinct meeting types, ranging from business scenarios to unstructured discussions. Furthermore, we benchmark end-to-end performance starting from raw speech to measure the capabilities of a complete system. Our results validate the framework's effectiveness and provide strong baselines to facilitate future research in meeting analysis and multi-party dialogue. Our dataset and code will be publicly available. The AMI-ME dataset and the Automatic Evaluation Framework are available at https://github.com/ku-nlp/AMI-ME.

Abstract

Synthetic Data Generation

Explore similar work

Rethinking Meeting Effectiveness: A Benchmark and Framework for Temporal Fine-grained Automatic Meeting Effectiveness Evaluation

Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting

Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding