FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities

Tian Du; Tiantong Wu; Yafei Wang; Mengyu Liu; Xingyan Chen; Mu Wang

cs.AISep 15, 2026

FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities

Authors: Tian Du, Tiantong Wu, Yafei Wang, Mengyu Liu, Xingyan Chen, Mu Wang

Organizations: Southwest University of Finance and Economics · Nanyang Technological University · Beijing University of Post and Telecommunication

Abstract

Comprehensive structured data on inter-firm relationships is often scarce or inaccessible because many relationships are privately negotiated, selectively disclosed, and fragmented across proprietary databases. This scarcity hinders the discovery of collaboration opportunities, particularly for startups and small and medium-sized enterprises. Firm profiles are readily available, but collaboration potential cannot be inferred from business similarity alone, since similar firms may be competitors, whereas dissimilar firms may offer complementary products, technologies, channels, capabilities, or capital. We present FirmCORe (Inter-Firm Collaboration Opportunity Reasoning), a human-annotated benchmark for pairwise reasoning over weakly structured firm profiles, comprising 2,805 labeled firm pairs. Given two firm profiles, a model must determine whether the available evidence supports a collaboration opportunity and, for positive pairs, jointly predict its strength, primary collaboration type, and role direction. FirmCORe also provides parallel Chinese- and English-language evaluation sets containing identical instances and gold labels, enabling controlled analysis of input-language sensitivity. Experiments with representative locally deployed and hosted large language models (LLMs) show that the strongest model achieves a macro-F1 score of 74.51 for opportunity detection but only 61.57% exact match across all four output fields. Language effects vary across models, and high cross-language agreement can mask errors shared across languages. These results indicate that current LLMs are substantially more reliable at detecting broad collaboration opportunities than at identifying their specific types and role directions.

Figures & tables

Figure 1: Overview of FirmCORe . The benchmark is constructed from unified firm profiles through stratified candidate mining, quality screening, blind human annotation, and tuple-level adjudication. Models then perform zero-shot structured prediction on parallel Chinese and English profile pairs under a unified evaluation protocol.

Firm profiles
Statistic	Value
Firms / industries	3,230 / 1,114
Avg. profile length (characters)	179.09
Annotation outcomes
Gold / unresolved pairs	2,805 / 195
Positive / negative pairs	968 / 1,837

Table 1: Statistics of the FirmCORe benchmark.

Model	Macro-F1	Positive			MCC
Model	Macro-F1	P	R	F1	MCC
Qwen3-2B	51.50	38.10	26.96	31.58	4.30
Qwen3-8B	51.42	42.49	93.80	58.49	28.55
Gemma4-26B	50.82	41.69	97.93	58.48	31.25
DeepSeek-7B	37.64	36.69	94.73	52.90	12.96
Llama3.1-8B	36.50	36.87	98.97	53.73	17.62

Table 2: Opportunity detection performance on Chinese.

Figure 2: Opportunity-detection Macro-F1 by candidate-generation stratum on the Chinese benchmark. Stratum prevalence differs substantially, so cross-stratum values are descriptive rather than controlled estimates of intrinsic difficulty.

Model	Strength		Type		Direction
Model	Acc.	F1	Acc.	F1	Acc.	F1
Qwen3-2B	20.66	19.59	15.08	13.05	7.64	13.34
Qwen3-8B	69.94	46.42	67.56	47.42	69.11	52.10
Gemma4-26B	66.53	46.00	70.66	53.34	73.45	66.52
DeepSeek-7B	78.51	46.41	32.54	31.75	38.95	36.31
Llama3.1-8B	77.48	45.93	60.02	42.77	67.56	52.25

Table 3: End-to-end structured relation prediction on the Chinese benchmark. F1 denotes Macro-F1.

Figure 3: End-to-end and oracle-detection Macro-F1 for opportunity strength, collaboration type, and role direction on the Chinese benchmark. End-to-end evaluation retains all gold-positive instances and counts a missed opportunity as an attribute error; oracle evaluation conditions on correctly detected gold-positive instances.

Figure 4: Chinese versus English four-field exact match across evaluated models. The dashed diagonal indicates equal performance in both languages.

Figure 5: Paired Chinese–English prediction agreement and correctness outcomes. Paired agreement measures tuple identity and does not imply correctness.

Figure 6: English-minus-Chinese class-level F1 differences for collaboration types and role directions. Positive values indicate higher performance under English input.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Stratum	Criterion
Strong-signal	Upstream score $\geq 90$
Borderline	Upstream score in $[60,75)$
Same-industry Hard	Same industry, $\operatorname{Sim}(i,j)\geq 0.03$ , and $\operatorname{Comp}(i,j)\leq 0.12$
Random Non-edge	No edge in the upstream candidate graph

Appendix

Table 4: Candidate-mining strata. Same-industry hard pairs are not presumed to be negative.

Baseline	Macro-F1	Positive P	Positive R	Positive F1	MCC
Always-No	39.57	0.00	0.00	0.00	0.00
Majority-Prior	39.57	0.00	0.00	0.00	0.00
Always-Yes	25.66	34.51	100.00	51.31	0.00
LSA-Cosine	43.67	33.55	64.46	44.13	-2.84
Com.-Threshold	46.43	67.29	7.44	13.40	13.73
Industry-Rule	33.29	9.11	7.33	8.13	-33.12

Appendix

Table 5: Comparison of opportunity-detection baselines on the Chinese benchmark.

Figure 7: Opportunity-detection Macro-F1 with 95% instance-bootstrap confidence intervals on the Chinese benchmark.

Model	Overall	Pos. EM	95% CI	Cov.	Cond. Attr. EM	MCF
Qwen3-8B	35.47	47.62	[44.42, 50.72]	93.80	50.77	3.00
Gemma4-26B	34.47	47.11	[44.01, 50.31]	97.93	48.10	3.09
Llama3.1-8B	22.25	44.11	[41.01, 47.21]	98.97	44.57	3.04
Qwen3-4B	20.71	43.49	[40.39, 46.59]	97.93	44.41	3.13
Gemma3-12B	14.69	30.89	[28.10, 33.78]	99.90	30.92	2.83
Llama3.2-3B	4.92	14.15	[11.98, 16.43]	99.69	14.20	2.34

Appendix

Table 6: Joint structured prediction performance. Positive four-field exact match requires opportunity existence, strength, collaboration type, and role direction to be simultaneously correct on every gold-positive instance. Detection-conditioned three-attribute exact match is computed only for gold-positive instances predicted as opportunities and is reported together with positive coverage.

Model	Coverage	Strength Macro-F1	Type Macro-F1	Direction Macro-F1	Oracle–E2E Gap
Qwen3-2B	26.96	46.40	29.37	27.85	19.21
Qwen3-8B	93.80	47.99	49.89	54.87	2.27
Gemma4-26B	97.93	46.50	54.46	67.23	0.78
DeepSeek-7B	94.73	47.72	32.32	37.24	0.94
Llama3.1-8B	98.97	46.17	43.70	52.51	0.48
Qwen3-4B	97.93	51.27	57.70	55.54	0.58

Appendix

Table 7: Oracle-detection conditional performance.

Model	ZH EM	EN EM	$\Delta$ EM	Paired Agreement	Both Correct	ZH Only	EN Only
Qwen3-2B	51.98	59.22	+7.24	75.08	49.73	2.25	9.48
Gemma4-26B	44.15	51.11	+6.95	68.60	39.83	4.32	11.28
Gemma3-12B	14.69	14.47	-0.21	50.09	11.27	3.42	3.21
Qwen3-8B	35.47	30.52	-4.96	39.47	21.78	13.69	8.73
Qwen3-4B	20.71	25.92	+5.20	34.08	12.66	8.06	13.26
Llama3.1-8B	22.25	27.95	+5.70	31.27	14.40	7.84	13.55

Appendix

Table 8: Paired Chinese–English evaluation. Paired Agreement measures tuple identity rather than correctness.

Model	Exact Eq. $\uparrow$	Direction Eq. $\uparrow$	Overturned $\downarrow$	Severe $\downarrow$	Swapped Pos. Overturned $\downarrow$
Qwen3.6-Max	91.37	94.44	8.63	6.74	26.00
Qwen3-VL-8B	88.63	93.05	11.37	8.09	39.81
Qwen3.6-35B	86.35	89.77	13.65	8.48	34.19
Qwen3.7-Plus	81.21	91.27	18.79	16.93	28.10
Qwen3.6-27B	81.89	89.55	18.11	16.33	36.07
GLM-5.1	51.98	73.23	48.02	38.86	41.92

Appendix

Table 9: Prediction stability under A/B firm-order permutation. All rates are percentages. The balanced set contains the same 2,805 firm pairs as the original Chinese test set, with 427 gold-positive unidirectional pairs shown in reversed order.

Figure 8: Changes in four-field exact-match correctness after A/B firm-order permutation. Leftward and rightward bars denote Correct

\rightarrow

Wrong and Wrong

\rightarrow

Correct transitions, respectively; annotations report net correction rates.

	Output validity					Confidence reliability
Model	Strict JSON $\uparrow$	Parse $\uparrow$	Missing $\downarrow$	Invalid $\downarrow$	Violation $\downarrow$	High Cov.	EM@High	EM@Low	Gap	High Err. $\downarrow$
Qwen3-2B	$100.00$	$100.00$	$0.46$	$0.00$	$0.46$	$42.14$	$36.55$	$63.22$	$-26.67$	$63.45$
Qwen3-8B	$0.00$	$96.54$	$24.21$	$3.46$	$20.75$	$69.73$	$33.13$	$46.14$	$-13.01$	$66.87$
Gemma4-26B	$99.43$	$100.00$	$2.64$	$0.00$	$2.64$	$90.34$	$38.12$	$0.37$	$37.75$	$61.88$
DeepSeek-7B	$0.00$	$99.07$	$10.20$	$0.93$	$9.27$	$77.86$	$5.82$	$31.03$	$-25.22$	$94.18$
Llama3.1-8B	$100.00$	$100.00$	$5.13$	$0.00$	$5.13$	$92.62$	$16.44$	$95.12$	$-78.69$	$83.56$

Appendix

Table 10: Output validity and self-reported confidence reliability on the Chinese benchmark.

Figure 9: Prediction outcomes on the 968 gold-positive Chinese instances. Each prediction belongs to one of three mutually exclusive categories: (i) complete tuple correct; (ii) opportunity false negative, where the gold label is Yes but the model predicts No; or (iii) attribute error after a positive opportunity prediction, where the model predicts Yes but at least one of strength, collaboration type, or role direction is incorrect, missing, or invalid.

Failure mode	Operational criterion	Frequency	Main pattern
Unsupported opportunity prediction	Gold opportunity is No, but the model predicts Yes.	23,621/34,903 (67.7%)	Broad industrial or lexical relatedness is frequently treated as sufficient evidence for a concrete collaboration interface.
Opportunity false negative	Gold opportunity is Yes, but the model predicts No.	1,903/18,392 (10.3%)	Cross-field evidence for complementarity is sometimes insufficiently integrated, leading the model to assign the negative opportunity label.
Valid but incorrect type	The gold label is Yes, the model predicts Yes, and the returned valid type differs from the gold type.	4,495/18,392 (24.4%)	Predictions collapse toward frequent or semantically adjacent mechanisms, especially Supply and Production.
Valid but incorrect direction	The gold label is Yes, the model predicts Yes, and the returned valid direction differs from the gold direction.	5,092/18,392 (27.7%)	Errors reflect both majority-class attraction and model-specific overprediction of A2B or Bidirectional relations.

Appendix

Table 11: Aggregate failure modes on the Chinese benchmark. Counts are model–instance decisions aggregated over 17 models. Denominators include all eligible decisions for the corresponding task.

Figure 10: Collaboration-type confusion aggregated over 17 models on gold-positive Chinese instances for which the model predicts Yes and returns a valid collaboration type. Cell annotations report aggregated prediction counts, while cell colors encode row-normalized percentages. Gold-positive instances predicted as No , together with predictions containing invalid or missing type outputs, are excluded from the matrix.

Figure 11: Distribution of valid role-direction predictions on gold-positive Chinese instances for which the model predicts Yes. Models are ordered by their predicted B2A share, and the gold distribution is included as a reference. Gold-positive instances predicted as No and positive predictions with invalid or missing directions are excluded from the normalization.

Model	$N$	Both correct	Identical wrong	Different wrong	ZH only	EN only
Qwen3.6-27B	2,805	55.8	21.5	12.6	5.7	4.3
Qwen3.6-Max	2,805	54.6	25.0	9.8	3.0	7.6
Qwen3.7-Plus	2,805	49.7	26.2	13.7	6.2	4.2
Kimi-K2.7	2,805	35.6	26.1	23.9	5.9	8.5
Gemma3-12B	2,805	11.3	38.8	43.3	3.4	3.2
Llama3.2-3B	2,805	1.2	6.3	85.2	3.7	3.5

Appendix

Table 12: Representative paired Chinese–English outcomes. Values are percentages of usable paired instances. “Identical wrong” denotes the same incorrect four-field tuple in both languages.

Pair	Pattern	Key profile evidence	Gold / representative prediction	Diagnostic interpretation
HSAI_00165	Unsupported opportunity	A: telecommunications services, broadband, IoT, cloud, and data centers. B: ICT infrastructure, network equipment, enterprise solutions, cloud, and smart devices.	No / Yes–Strong–Supply–B2A	Broad ICT overlap is treated as a specific supply interface, although neither profile states a concrete demand relationship.
MLAI_01557	Opportunity false negative	A: retail of furniture, home-improvement materials, appliances, bathroom products, and lighting. B: broad import–export trade in agricultural products, machinery, textiles, and other goods.	Yes–Strong–Distribution–B2A / No	The model must connect a broad trading capability to a downstream retail channel despite limited direct product overlap.
HSAI_00848	Type and direction confusion	A: media distribution, advertising, brand promotion, and digital marketing. B: digital-marketing platforms, ad-delivery systems, analytics, and related technical services.	Yes–Strong–Technology–B2A / Yes–Strong–Distribution–Bidirectional	Commercial-channel cues obscure both the technical mechanism and the annotated resource-flow direction.
RNEG_01776	High-confidence type error	A: property sales, leasing, and consulting. B: real-estate development, investment, asset management, and property management.	Yes–Strong–Supply–A2B / Yes–Strong–Distribution–A2B	The direction is preserved, but service provision is conflated with channel distribution under High confidence.

Appendix

Table 13: Representative diagnostic cases. Evidence summaries are derived only from the supplied firm profiles.

Structured Reasoning in Language Models

See all 36 papers

Aug 27, 2026

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.

Sil Hamilton, Albert Yu Sun, Oscar J. Romero +4

Epiq AI Labs · Cornell University

LLM Evaluation LLM Information Extraction Knowledge Graph Question Answering Synthetic Benchmark Generation

Ask PDF

Jan 29, 2026

MoCo: A One-Stop Shop for Model Collaboration Research

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.

Shangbin Feng, Yuyang Bai, Ziyuan Yang +17

University of Washington · Texas A&M University · University of Notre Dame +7

Multi-LLM Collaboration

Ask PDF

May 9, 2026

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows

Large language model (LLM) agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles, permission-controlled systems, and cross-departmental procedures. However, existing enterprise benchmarks largely evaluate single agents with broad tool access, while existing multi-agent benchmarks rarely capture realistic enterprise constraints such as role specialization, access control, stateful business systems, and policy-based approvals. We introduce EntCollabBench, a benchmark for evaluating enterprise multi-agent collaboration. EntCollabBench simulates a permission-isolated organization with 11 role-specialized agents across six departments and contains two evaluation subsets: a Workflow subset, where agents collaboratively modify enterprise system states, and an Approval subset, where agents make policy-grounded decisions. Evaluation is based on execution traces, database state verification, and deterministic policy adjudication rather than natural-language response judging. Experiments with representative LLM agents show that current models still struggle with end-to-end enterprise collaboration, especially in delegation, context transfer, parameter grounding, workflow closure, and decision commitment. EntCollabBench provides a reproducible testbed for measuring and improving agent systems intended for realistic organizational environments.

Tao Yu, Hao Wang, Changyu Li +15

BAAI · CASIA · UCAS +2

LLM Agent Evaluation Business Process Automation Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent Systems

Ask PDF

Abstract

Structured Reasoning in Language Models

Explore similar work

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

MoCo: A One-Stop Shop for Model Collaboration Research

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows