CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

Sil Hamilton; Albert Yu Sun; Oscar J. Romero; Carl-Leander Henneking; David Mimno; Bishan Yang; Igor Labutov

cs.AIAug 27, 2026

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

Authors: Sil Hamilton, Albert Yu Sun, Oscar J. Romero, Carl-Leander Henneking, David Mimno, Bishan Yang, Igor Labutov

Organizations: Epiq AI Labs · Cornell University

Abstract

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.

Figures & tables

Figure 1: CorporateBench advances the state of the art in multi-document QA benchmarking when compared against nine previous benchmarks in evidence and question counts. CB achieves a high ratio of 87.6 documents per question, with the closest benchmark (EKRAG) achieving a ratio of 1.5 ( Yu et al., 2025 ) . A high ratio indicates that questions require synthesizing information across many documents, which better reflects the complexity of real enterprise knowledge work.

Figure 2: CorporateBench contains four synthetic companies spanning different industries, headcounts, and business models. The figure shows a vertical slice of a hypothetical company and its social network at a given time.

Panel of Figure 3 — Figure 3: An email generated for CorporateBench . Employees, topic, and evidence strings are in bold.

Company	Zenith	Streamvibe	Biocure	Pound
Size	Small (S)	Medium (M)	Large (L)	Extra Large (XL)
Industry	Tech	Media	Pharmacare	Finance
Entity Types
Task	103	1,134	6,479	52,504
Meeting	76	890	6,445	57,265
Employee	12	122	1,110	10,210

Table 1: Summary statistics for each of the four synthetic companies generated for CorporateBench . Note that each company represents an additional order of magnitude over the previous in all respects.

Figure 4: Topic classification

F_{1}

scores across number of few-shot examples provided in the prompt (for LLMs) or labeled training examples (for TF-IDF+LR) for Biocure (L) and Pound (XL) datasets.

		Entities			Relationships			Temporal Relationships †
Size	Model	Precision	Recall	F 1	Precision	Recall	F 1	Precision	Recall	F 1
S	Haiku 4.5	.866	.786	.824	.751	.966	.845	.453	.488	.470
	GPT-5 Nano	.926	.624	.746	.795	.285	.419	.317	.092	.142
	Gemini 2.5 F-L	.846	.792	.818	.677	.969	.797	.441	.444	.442
M	Haiku 4.5	.911	.721	.805	.782	.595	.676	.427	.209	.281
	GPT-5 Nano	.902	.645	.752	.569	.165	.256	.273	.046	.078

Table 3: KB Evaluation Results. Bold denotes best score out of the three models.

Method	Model	KB QA				Topic QA				Integrated QA
Method	Model	S	M	L	XL	S	M	L	XL	S	M	L	XL
RAG	Haiku 4.5	.265	.273	.189	.164	.650	.563	.418	.464	.332	.277	.327	.424
	Sonnet 4.5	.322	.311	.198	.215	.656	.535	.423	.492	.439	.250	.272	.401
	GPT-5 Nano	.527	.501	.312	.244	.672	.479	.385	.456	.511	.475	.468	.510
	GPT-5.1	.470	.440	.276	.269	.634	.485	.406	.456	.469	.564	.560	.566
	Gemini 2.5 F-L	.230	.200	.199	.147	.572	.401	.374	.376	.338	.218	.313	.295

Table 4: KB QA, Topic QA, and Integrated QA Results. Bold denotes best model performance for the particular QA task using the respective methods (RAG or KB).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Industry	Size	Model	Story
Zenith	Tech	Small	B2C	Nestled in the heart of California’s tech scene, Zenith Labs is a nimble, people-first B2C company crafting sleek laptops, phones, and desktops for everyday users who crave simplicity without sacrificing style. With few employees and a laid-back, startup vibe, Zenith Labs punches above its weight in the consumer electronics space, offering intuitive, affordable devices that blend minimalist design with reliable performance. Their audience? Young professionals, students, and creatives who want tech that feels personal—not corporate. Zenith Labs’ motto "Keep it crisp" reflects their mission to strip away the noise and deliver clean, user-friendly experiences. Their goal is to become the go-to brand for people who want smart tech that doesn’t try too hard.
Streamvibe	Media	Medium	B2C	A creative powerhouse in the entertainment industry, StreamVibe Studios produces original streaming content, podcasts, and digital campaigns for the cord-cutting generation. With talented creators, producers, and marketers spread across Los Angeles and Austin, StreamVibe has carved out a niche creating viral web series, documentary films, and branded content for Gen Z and millennial audiences. Known for their "content first, suits second" mentality, StreamVibe attracts top creative talent with flexible work arrangements, profit-sharing, and a culture that celebrates bold storytelling and authentic voices.
Biocure	Pharma	Large	B2B	Operating from a sprawling research campus with state-of-the-art laboratories, BioCure Solutions is a science-driven B2B pharmaceutical company specializing in contract research and development services for drug discovery and clinical trials. This established industry leader employs scientists, researchers, and regulatory specialists who maintain rigorous FDA compliance standards and operate under strict Good Manufacturing Practice protocols that have secured partnerships with major pharmaceutical corporations worldwide. Their mission is to become the premier CRO partner for pharmaceutical companies developing tomorrow’s therapies, providing the scientific rigor and regulatory expertise that transforms promising compounds into approved medicines that improve human health globally.
Pound	Finance	X-Large	B2B	Pound Financial is a mid-market investment firm specializing in private equity and corporate restructuring for manufacturing and technology companies. The firm maintains a conservative, relationship-driven approach, focusing on long-term value creation rather than quick exits. Their clientele includes family-owned businesses seeking growth capital and Fortune 500 companies navigating complex financial transitions. Pound Financial’s reputation for discretion, thorough due diligence, and strategic guidance has made them a trusted partner for executives facing critical financial decisions.

Appendix

Table 5: Company/KB Information Overview.

Temporal Stage	Strength	Examples
Start	Explicit	I just joined {{object}} and wanted to introduce myself
		I’m excited to announce that I’m now part of {{object}}
		I recently became a member of {{object}}
		… 17 more
	Implicit	onboarding_to_team: Just going through the {{object}} onboarding materials today
		learning_team_processes: Still getting familiar with how {{object}} runs things

Appendix

Table 6: Example evidence strings for the memberOf relationship showing temporal stages and evidence strength levels. Explicit strings directly state team membership, while implicit strings demonstrate engagement through workplace actions. The placeholder {{object}} represents the team name.

Progress Level	Strength	Examples
0-25%	Explicit	{{employee}} {{have_verb}} {{task}} about 20% done
		{{employee}} {{be_verb}} roughly a quarter of the way through {{task}}
		{{task}} is still in early stages with {{employee}}, around 15-25% complete
		… 17 more
	Implicit	initial_research: {{employee}} {{be_verb}} still gathering requirements for {{task}}
		project_kickoff: {{employee}} sent out the project charter for {{task}} today

Appendix

Table 7: Example evidence strings for the taskProgress relationship showing progress levels and evidence strength. Explicit strings directly state completion percentages, while implicit strings demonstrate typical activities at each stage. The placeholders {{employee}} and {{task}} represent the employee name and task name respectively.

Size	Documents	Non-Business	Business
S	$\approx 4\cdot 10^{2}$	$3\cdot 10^{0}$	$3\cdot 10^{0}$
M	$\approx 4\cdot 10^{3}$	$3\cdot 10^{1}$	$3\cdot 10^{1}$
L	$\approx 3\cdot 10^{4}$	$3\cdot 10^{2}$	$3\cdot 10^{2}$
XL	$\approx 2\cdot 10^{5}$	$3\cdot 10^{3}$	$3\cdot 10^{3}$

Appendix

Table 8: Breakdown of the number of topics used to condition the generation of documents in CorporateBench based on different sizes of the company. We provide more detail in Appendix C .

Level 0	Level 1	Level 2	Level 3 (Examples)
Non-Business	Food	Cooking	Grilling techniques shared ( … 9 more )
		Baking	Bread making adventures ( … 9 more )
		Dining Out	Restaurant service reviews ( … 9 more )
		Food Culture	Cultural food traditions ( … 9 more )
		Nutrition	Vitamin supplement discussions ( … 9 more )
		… 5 more subcategories

Appendix

Table 9: Example non-meeting topic hierarchy for Biocure showing the four-level structure from top-level categories down to specific discussion topics.

Meeting Type	Topics
Direct Report	Employee Performance Review, Goal Setting and Progress Check-In, Career Development Discussion, Feedback and Coaching Session, Workload and Prioritization Review, Upcoming Deadlines and Deliverables, Team Dynamics and Collaboration Feedback, Skill Growth and Training Opportunities, Manager-Report Relationship Check-In, Quarterly Performance Summary
Team	Weekly Team Standup, Team Goals and KPIs Review, Process Improvement Discussion, Cross-Functional Collaboration Update, Retrospective and Lessons Learned, Upcoming Projects and Assignments, Team Communication and Workflow Alignment, Resource Planning and Allocation, Team Morale and Culture Check, Problem-Solving and Roadblock Discussion
Task	Task Progress and Status Update, Dependency and Blocker Resolution, Timeline and Deliverable Planning, Quality Review and Standards Alignment, Task Ownership and Accountability, Cross-Task Integration Discussion, Testing and Validation Checkpoint, Task Documentation and Handover, Risk and Issue Review, Next Steps and Action Items
Executive	Company Strategy and Vision Alignment, Financial Performance and Budget Review, Key Metrics and KPIs Discussion, Organizational Priorities and Focus Areas, Market Trends and Competitive Analysis, Resource Allocation and Headcount Planning, Operational Efficiency Review, Major Risks and Mitigation Planning, Executive Decisions and Approvals, Quarterly Business Review
Project	Project Kickoff and Scope Definition, Milestone Progress Review, Timeline and Deliverable Alignment, Resource and Budget Status, Stakeholder Communication Plan, Risk Management and Mitigation Discussion, Project Dependencies and Coordination, Quality Assurance and Testing Review, Project Retrospective and Lessons Learned, Next Phase Planning and Execution
Department	Department Strategy and Goals Review, Cross-Team Coordination Update, Department Resource Planning, Organizational Changes and Updates, Department Performance Metrics Review, Interdepartmental Collaboration Discussion, Budget and Resource Allocation Review, Department Culture and Team Building, Major Initiative Planning, Department Communication and Updates

Appendix

Table 10: Meeting topics.

Figure 5: Summary of theoretical Dirichlet distributions used for topic sampling when generating non-meeting documents.

Figure 6: Summary of empirical topic distributions of generated documents across all four KBs. Includes meeting and non-meeting topics. See Appendix C for more details on topics.

Meeting Type	Description	Minutes	Agenda	Empty Cal.	Frequency	Organizer	Substantiates
direct_report	employee and their manager	30%	30%	40%	weekly	manager	task progress, reports to, works on
team	employees that belong and lead a team	25%	25%	50%	monthly	team lead	task progress, works on, member of
task	employees collaborating on a task	30%	30%	40%	biweekly	any	task progress, works on
executive	employees that are executives	50%	50%	0%	monthly	any	project progress, title
project	employees that work on tasks part of a project	50%	50%	0%	monthly	team lead of the project	project progress, works on, has task
department	employees that lead the department and the teams part of it	50%	50%	0%	monthly	department lead	has task, team belongs to department, department belongs to company

Appendix

Table 11: Supported Meeting Types.

Question	Who participated in the meeting titled cross-team protocol alignment - work session?
Answer	James Moreira, James Vera
Company/KB	Streamvibe
Answer Type	List[str]
Constraints	1

Appendix

Table 17

Question	Who organized the target company due diligence discussion meeting?
Answer	Birgitta Lagarde
Company/KB	Pound
Answer Type	str
Constraints	1

Appendix

Table 18

Question	When was drug approval first mentioned?
Answer	2024-01-01
Company/KB	Biocure
Answer Type	date
Constraints	1

Appendix

Table 19

Question	Did anyone who works at Zenith Labs send emails about Daily commute issues during month 1 of 2024?
Answer	true
Company/KB	Zenith
Answer Type	bool
Constraints	4

Appendix

Table 20

Question	How many of jared chartier’s direct reports sent emails about technical debt management after 29 Mar 2024?
Answer	0
Company/KB	Zenith
Answer Type	int
Constraints	3

Appendix

Table 21

Figure 7: Visualization of results presented in Table 4 .

Figure 8: Visualization of results presented in Table 4 aggregated by method.

Non-Temporal Relationships
Error Type	Relationship	Gemini 2.5 FL	Haiku 4.5	GPT-5 Nano
Over-predicts (FP)	worksOn	69%	64%	43%
	worksAt	7%	$-$	3%
	reportsTo	8%	$-$	4%
	memberOf	16%	12%	50%
	attend	$-$	16%	$-$

Appendix

Table 12: Error Distribution by Relationship Type and Model

Model	Training Size	Biocure (L)			Pound (XL)
Model	Training Size	Precision	Recall	F 1	Precision	Recall	F 1
Haiku 4.5	0	.917 $\pm$ .00	.887 $\pm$ .01	.885 $\pm$ .01	.550 $\pm$ .01	.410 $\pm$ .02	.357 $\pm$ .01
	10	.950 $\pm$ .01	.959 $\pm$ .01	.951 $\pm$ .01	.649 $\pm$ .01	.650 $\pm$ .01	.594 $\pm$ .01
	50	.963 $\pm$ .01	.967 $\pm$ .00	.963 $\pm$ .01	.697 $\pm$ .01	.733 $\pm$ .01	.685 $\pm$ .01
Sonnet 4.5	0	.875 $\pm$ .04	.873 $\pm$ .04	.872 $\pm$ .04	.583 $\pm$ .01	.544 $\pm$ .02	.517 $\pm$ .02
	10	.963 $\pm$ .00	.966 $\pm$ .01	.962 $\pm$ .01	.681 $\pm$ .03	.692 $\pm$ .03	.661 $\pm$ .03

Appendix

Table 13: Topic Classification Results.

Model	Error Type	S	M	L	XL	S	M	L	XL	Total
		RAG				KB
Haiku 4.5	Prompt too long	0	0	0	2	0	0	8	22	32
	Tool calls limit exceeded	0	0	0	0	19	15	26	24	84
	Tool max retries exceeded	0	0	0	0	0	0	3	0	3
	Total Errors	0	0	0	2	19	15	37	46	119
Sonnet 4.5	Output validation failed	0	6	9	1	0	11	10	1	38

Appendix

Table 14: Error analysis across all models.

Biocure (L)	Pound (XL)
Non-business	Non-business
Approval Timeline Management	Exit Planning
Biomarker Identification	Financial Analysis
Clinical Data Analysis	Legal Compliance
Clinical Trial Planning	Market Intelligence
Competitive Intelligence	Market Research

Appendix

Table 15: Topic Categories for Biocure (L) and Pound (XL).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Sep 9, 2026

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We present the Era by Eon Benchmark for evaluating LLM agents that use enterprise tools. The benchmark is built around a complete fictional company. It includes product simulators, company-specific internal databases, benchmark questions, and computed answer keys. Industry, company size, business model, application portfolio, and a seed define each company. One seeded entity graph supplies shared company data to simulators of Salesforce, Zendesk, Slack, Gong, and other products. A questionconditioned generator creates the schemas and records for internal databases. It takes shared entities, keys, and values from the same graph before generating database-specific facts. Both mechanisms therefore describe one consistent enterprise estate. Every expected answer is computed from the final records, so grading is exact. Design and answer-key checks validate the internal databases. A realism scorecard and adversarial detector validate the entity graph. Across 23 generated companies, the mean realism score rose from 61.8 to 97.0, with zero records flagged as synthetic. In the reported simulator-track comparison, nine models answered the same 33 questions three times each. Accuracy estimates ranged from 42.4% to 76.8%, and three of 36 pairwise differences remained supported after correction.

Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3

Eon

Tool-Use Evaluation LLM Agent Evaluation Synthetic Benchmark Generation

Ask PDF

Aug 11, 2026

ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering

Enterprise question answering is framed as retrieving internal documents and generating grounded answers. Routine enterprise records, however, are work by-products in which required organizational relations remain implicit across heterogeneous sources. Existing benchmarks provide realistic multi-source evidence, but often materialize a predefined answer path and therefore test the composition of stated facts rather than recovery of a target relation absent from the corpus. We call the latter capability latent organizational reasoning. We introduce ENTLORE, a graph-grounded benchmark construction framework that reconstructs an audited enterprise world from routine documents, authoritative organizational tables, and operational records. Versioned organizational conventions certify derived relations in a truth graph, enabling complete golden answers and proof certificates. The aligned anonymized release exposes only the document corpus while withholding private structure and target relations. ENTLORE contains 2,341 documents from three source types and 907 questions spanning explicit lookup, cross-source composition, and latent organizational reasoning, evaluated across 56 model and access configurations. Structuring the released world as an induced entity graph or navigable knowledge base gives the strongest deployable results. Yet supplying gold documents still leaves 30.4% of latent questions unanswered, versus 12.6% and 6.2% for explicit and compositional questions. Enterprise QA therefore depends not only on document recall, but also on whether implicit organizational relations become usable. The benchmark, data, and code are publicly available at https://github.com/scitix/entlore .

Akrin Zheng, Alexander Wu, Alaia Liu

Graph Retrieval-Augmented Generation Knowledge Graph Question Answering Latent Reasoning

Ask PDF

May 29, 2026

Office Comprehension Benchmark

We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.docx, .xlsx, .pptx) and their variants. OCB consists of two tracks. File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges. Domain Q&A tests expert-level reasoning grounded in real-world industry documents across 12 professional domains, with queries requiring multi-step analysis and synthesis across documents. Each reference answer is decomposed into atomic, binary-gradable claims, and an ensemble of LLM judges scores responses against each claim independently. Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A; increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains. We release the dataset, evaluation tooling, judge prompt, and a public leaderboard.

Firoz Shaik, Mateus Picanço Lima Gomes, Tanvir Aumi +17

Microsoft · † Work done while at Microsoft.

Document Understanding LLM Evaluation Document QA Knowledge-Intensive QA

Ask PDF

Abstract

LLM Evaluation

Explore similar work

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering

Office Comprehension Benchmark