ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-making

Ziyang Guo; Yifan Wu; Jason Hartline; Kenneth Holstein; Jessica Hullman

cs.AIFeb 23, 2026

ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-making

Authors: Ziyang Guo, Yifan Wu, Jason Hartline, Kenneth Holstein, Jessica Hullman

Organizations: Northwestern University · Microsoft Research · Carnegie Mellon University

Abstract

Multi-agent decision pipelines can outperform single agent workflows when complementarity holds, i.e., different agents bring unique information to the table to inform a final decision. We propose ComplLLM, a post-training framework based on decision theory that fine-tunes a decision-assistant LLM using complementary information as reward to output signals that complement existing agent decisions. We validate ComplLLM on synthetic and real-world tasks involving domain experts, demonstrating how the approach recovers known complementary information and produces plausible explanations of complementary signals to support downstream decision-makers.

Figures & tables

Figure 1: A graphical representation of how our work goes beyond simply identifying concepts that appear in text or signals that are predictive of a target state, by also ensuring that signals complement the existing recommendation. The diagonal stripes (twill pattern) represent the target of the corresponding methods.

Figure 2: The “two agents” setting in our framework. The recommending agent makes a recommendation $Z$ based on their own features $X$ , and the supervisor agent aggregates their own information $T$ with $Z$ to make a final decision $D$ .

Notation	Radiology Diagnosis instantiation
$Y$	Payoff state $\in\{0,1\}$ for the presence of cardiac dysfunction.
$Z$	Prediction from the vision model on the probability of the presence of cardiac dysfunction $\in[0,1]$ .
$X$	Features of the vision model (X-ray image).
$T$	Features of the clinician (radiology report and patient’s medical history).
$D$	Final decision by the clinician $\in\{0,1\}$ (e.g., whether to order a biopsy).
Signals in $T$

Table 1: Notation summary (instantiated with the example of radiology-diagnosis).

Method	Surf.	F1
CompLLM	0.98	0.67
Zero-shot	0.91	0.42
Few-shot	0.95	0.38
BERTopic	0.84	0.17
HypotheSAE	0.90	0.38

Table 2: Results comparing average surface similarity and F1 score of the extracted signals by each method on the synthetic dataset.

Source	Total Acc.	# Sig	Significant Signals	$\Delta$ Acc.
Agent Decision	0.819	-	-	-
ComplLLM	0.839	12	negative pneumothorax	+0.0048
			positive pleural effusion	+0.0040
			negative pleural effusion	+0.0032
			positive pulmonary congestion	+0.0024
			positive cardiac silhouette enlargement	+0.0016

Table 3: Signals on MIMIC-CXR dataset that are significant with

p<0.05

after Bonferroni correction. ‘

\Delta

Acc.’ represents the marginal accuracy improvement each signal

\mathbf{s}_{j}

marginally provides over the other signals and the agent recommendation. ‘Total Acc.’ shows the combined accuracy when all significant signals are included, i.e., sum of the signals’ marginal gain. ‘# Sig’ indicates the number of statistically significant signals discovered by each method.

Figure 3: Expected accuracy given the extracted signals and the agent’s recommendation by each method. Dashed lines represent agent decision accuracy and the accuracy of the benchmark method. Error bars depict bootstrapped 95% confidence intervals (N=5000).

Figure 4: Accuracy of LLM paper review decisions on the Review5K dataset. Error bars depict bootstrapped 95% confidence intervals (N=5000).

Method	Count	Signals
Ground Truth	2	Edema Pleural Effusion
ComplLLM	16	positive_pleural_effusion positive_edema positive_cardiomegaly positive_pleural_effusion_left positive_pleural_effusion_right positive_pneumonia positive_edema_resolution positive_troponins positive_atelectasis no_pulmonary_edema …
HypotheSAE	8	mentions presence of pulmonary edema mentions presence of interstitial edema mentions presence of interstitial lung disease or interstitial findings mentions no acute cardiopulmonary process mentions presence of pleural effusion and atelectasis mentions presence of pleural effusion mentions presence of bilateral pleural effusion mentions presence of interstitial pulmonary edema
BERTopic	15	Outlier/Noise Chest X-ray Findings Cardiac Enlargement Pancreatic Cancer and Chest Imaging Chest X-ray for TIA evaluation Chest X-ray for Upper GI Bleed Pulmonary Edema Monitoring Pulmonary Edema Evaluation Chest X-ray findings in edema Chest X-ray findings …
Few-shot	26	enlarged_cardiac_silhouette pulmonary_vascular_congestion pleural_effusion vascular_congestion moderate_cardiomegaly bilateral_pleural_effusions interstitial_edema cardiomegaly mild_pulmonary_edema pulmonary_congestion …
Zero-shot	55	pulmonary_vascular_congestion cardiac_enlargement pleural_effusions cardiac_silhouette_enlarged bilateral_pleural_effusions small_left_pleural_effusion no_cardiomegaly no_pleural_effusion moderate_cardiomegaly pulmonary_edema …

Table 4: Summary of Extracted Signals by Method on Synthetic Dataset.

Source	Total Acc.	# Sig	Significant Signals	$\Delta$ Acc.
Baseline Agent Decision Accuracy: 0.847
ComplLLM	0.878	4	promotes or condones violence	+0.0222
			misinformation conspiracy or false theory	+0.0061
			offers safe alternative	+0.0020
			oblique or profane language	+0.0010
BERTopic	0.856	2	Outlier/Noise	+0.0051

Table 5: Signals on DICES dataset. Significant signals with

p<0.05

after Bonferroni correction. ‘

\Delta

Acc.’ represents the incremental accuracy improvement each signal provides over the baseline agent decision (0.847). ‘Total Acc.’ shows the combined accuracy when all significant signals are included. ‘# Sig’ indicates the number of statistically significant signals discovered by each method.

Source	Total Acc.	# Sig	Significant Signals	$\Delta$ Acc.
Agent Decision	0.627	-	-	-
ComplLLM	0.692	8	clarity	+0.0310
			novelty	+0.0179
			experimental validation	+0.0110
			theoretical contribution	+0.0024
			clarity of presentation	+0.0021

Table 6: Signals on Review5k dataset. Significant signals with

p<0.05

after Bonferroni correction. ‘

\Delta

Acc.’ represents the incremental accuracy improvement each signal provides over the baseline agent decision (0.627). ‘Total Acc.’ shows the combined accuracy when all significant signals are included. ‘# Sig’ indicates the number of statistically significant signals discovered by each method.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Biomarker	Age	Gender	Threshold
NT-proBNP	$\leq$ 49	–	$>$ 449
	50–75	–	$>$ 899
	$\geq$ 76	–	$>$ 1799
Troponin	$<$ 64	Female	$\geq$ 0.014
	$\geq$ 65	Female	$\geq$ 0.018
	$<$ 50	Male	$\geq$ 0.019

Appendix

Table 7: Biomarker Thresholds by Age and Gender

Condition	Positive	Negative	Uncertain
Atelectasis	0.3384	0.3633	$-$ 0.1758
Cardiomegaly	0.7462	$-$ 0.3945	0.3062
Consolidation	0.9686	$-$ 0.0945	0.6488
Enlarged Cardiomediastinum	0.2368	$-$ 0.9585	0.1678
Fracture	0.3986	$-$ 0.7871	0.2674
Lung Lesion	0.3194	0.1314	0.0970

Appendix

Table 8: Coefficients of the logistic regression model on the synthetic ground-truth labels. * indicates the labels are withheld from the model to generate the recommending agent’s decision.

Panel of Figure 5 — Figure 5: Training curves of SFT and GRPO on the MIMIC-CXR dataset.

Panel of Figure 6 — Figure 6: Training curves of SFT and GRPO on the DICES dataset.

Panel of Figure 7 — Figure 7: Training curves of SFT and GRPO on the Review5K dataset.

Panel of Figure 8 — Figure 8: Training curves of SFT and GRPO on the synthetic dataset.

Figure 9: Experimental results for ComplLLM with only SFT. Dashed lines represent agent decision accuracy and the accuracy of the benchmark method. Error bars depict bootstrapped 95% confidence intervals (N=5000).

Figure 10: An example of the radiology report and X-ray image shown to the physicians in the qualitative study.

Figure 11: An example of ComplLLM ’s output shown to the physicians in the qualitative study.

Figure 12: An example of ComplLLM ’s output and the updated probablistic prediction shown to the physicians in the qualitative study.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

Multi-agent decision making: A Blackwell's informativeness approach
May 7, 2026Zheng Zhang, Cuong C. Nguyen, Kevin Wells +1Multi-Agent Debate Question Answering
Centre for Vision, Speech and Signal Processing, University of Surrey
Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
Oct 13, 2026Keru Chen, Sen Lin, Yingbin Liang +2LLM Agent Training Multi-Agent Collaboration
School of ECEE, Arizona State University · Computer Science Department, University of Houston · Department of ECE, The Ohio State University +1
Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs
Oct 23, 2026Seunghan Lee, Jun Seo, Junhyeok Kang +9LLM Decision-Making Value of Information
LG AI Research

May 7, 2026J/K move · Enter open · S save

Multi-agent decision making: A Blackwell's informativeness approach

Zheng Zhang, Cuong C. Nguyen, Kevin Wells, Gustavo Carneiro

Centre for Vision, Speech and Signal Processing, University of Surrey

Multi-Agent Debate Question Answering LLM Decision-Making

Open paper AskPDF arXiv

The rapid development of large language models (LLMs) has motivated research on decision-making in multi-agent systems, where multiple agents collaborate to achieve shared objectives. Existing aggregation approaches, such as voting and debate, are largely ad-hoc and lack formal guarantees regarding the informativeness of the resulting decisions. In this paper, we provide a principled approach to analyse decisions made in the multi-LLM setting using Blackwell's informativeness framework. Within the Blackwell information-structure abstraction, we show that voting and debate induce information structures that are no more informative than the pooled private information of all agents. This result identifies Bayesian pooled posterior maximisation as an information-theoretic upper-bound decision rule under the Blackwell ordering. Motivated by this theoretical analysis, we introduce a practical method for LLM-based question-answering (QA) tasks that estimates each agent's posterior and approximates the pooled posterior using a product-of-posteriors estimator. Extensive experiments on six QA benchmarks demonstrate that our approach outperforms state-of-the-art multi-LLM debate and voting methods.

Abstract

LLM Decision-Making

Explore similar work

Multi-agent decision making: A Blackwell's informativeness approach

Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs