Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning

Wei Yang; Defu Cao; Jiacheng Pang; Muyan Weng; Yan Liu

cs.AIMar 9, 2026

Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning

Authors: Wei Yang, Defu Cao, Jiacheng Pang, Muyan Weng, Yan Liu

Organizations: University of Southern California

Abstract

While scaling individual Large Language Models (LLMs) has delivered remarkable progress, the next frontier lies in scaling collaboration through multi-agent systems (MAS). However, purely autonomous MAS remain ''closed-world'' systems, constrained by the static knowledge horizon of pre-trained models. This limitation makes them brittle on tasks requiring knowledge beyond training data, often leading to collective failure under novel challenges. To address this, we propose the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework, a principled paradigm for human--agent collaboration. HILA trains agents to learn a metacognitive policy that governs when to solve problems autonomously and when to defer to a human expert. To operationalize this policy, we introduce Dual-Loop Policy Optimization, which disentangles immediate decision-making from long-term capability growth. The inner loop applies Group Relative Policy Optimization (GRPO) with a cost-aware reward to optimize deferral decisions, while the outer loop implements continual learning, transforming expert feedback into high-quality supervised signals that strengthen the agent's reasoning ability. Experiments on challenging mathematical and problem-solving benchmarks show that HILA, equipped with Dual-Loop Policy Optimization, consistently outperforms advanced MAS, establishing a principled foundation for collaborative and continually improving agentic systems.

Figures & tables

Figure 1: Overview of the proposed HILA framework and its Dual-Loop Policy Optimization (DLPO) training paradigm. Left: HILA coordinates multi-agent collaboration with both proactive human guidance and reactive expert feedback via metacognitive states and strategic actions ( Eval , Create , Defer ). Right: DLPO optimizes the meta-policy in an inner RL loop with cost-aware rewards, and expands the model’s knowledge boundary in an outer continual-learning loop by storing DEFER -triggered human feedback as offline supervision.

Model	Qwen2.5-7B	Qwen2.5-3B	LLaMA3-8B	LLaMA3-3B
Vanilla	90.71 (+0.00)	83.25 (+0.00)	72.76 (+0.00)	45.26 (+0.00)
CoT	91.13 (+0.42)	84.36 (+1.11)	74.22 (+1.46)	52.49 (+7.23)
SC	91.79 (+1.08)	87.54 (+4.29)	80.79 (+8.03)	58.68 (+13.42)
PHP	91.35 (+0.64)	86.83 (+3.58)	80.01 (+7.25)	66.21 (+20.95)
LLM-Debate	92.47 (+1.76)	87.41 (+4.16)	83.52 (+10.76)	76.32 (+31.06)
DyLAN	92.64 (+1.93)	87.85 (+4.60)	82.03 (+9.27)	76.17 (+30.91)

Table 2: Performance of baselines across four LLM backbones on GSM8K. All values are percentages (percent sign omitted). Parentheses show absolute differences (percentage points) relative to the Vanilla row for each backbone. HILA refers to the proposed method with DLPO.

Table 3

Figure 2: Effect of human proxy capability on HILA. Using stronger language models as the external expert consistently improves performance on GSM8K, AMC, and MMLU.

Panel of Figure 3 — Figure 3: Effect of scaling collaborative configurations. (a) shows how increasing the number of agents impacts accuracy on AMC and GSM8K, while (b) analyzes how varying the number of interaction rounds influences performance. Together, the results highlight the trade-offs between broader exploration through more agents and deeper refinement through additional rounds.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Accuracy vs. Defer rate on GSM8K across training stages (Init $\rightarrow$ GRPO $\rightarrow$ DLPO). DLPO achieves a joint improvement, increasing accuracy while reducing reliance on external intervention.

Figure 5: Accuracy vs. Defer rate on AMC across training stages. GRPO reduces deferral with a small accuracy fluctuation, while DLPO simultaneously improves accuracy and further lowers deferral, suggesting that continual learning mitigates over-conservative intervention.

Figure 6: Accuracy vs. Defer rate on MMLU across training stages. Accuracy increases monotonically as deferral decreases, indicating that the learned policy becomes both more selective and more capable.

Panel of Figure 7 — Figure 7: Deferral cost as a control knob for collaboration. (a) shows how changing the deferral cost $C$ affects task accuracy on AMC and MMLU. (b) shows how increasing cost shifts the learned policy on AMC away from Defer and toward autonomous actions, especially Eval . Together, the results illustrate the trade-off between reducing external intervention and preserving final performance.

# Agents	AMC	Avg. Input Tokens	Avg. Output Tokens	Avg. Total Tokens
1	26.67	3146	1299	4445
2	28.33	10184	2564	12748
3	32.50	23285	4463	27748
4	31.67	42540	6349	48889
5	33.33	64097	7585	71682
6	32.50	92135	9501	101636

Appendix

Table 7: Effect of the number of agents on performance and token cost on AMC. Increasing the collective size generally improves accuracy, but also leads to a steep rise in input, output, and total token usage, revealing a clear performance–efficiency trade-off.

# Agents	MMLU	Avg. Input Tokens	Avg. Output Tokens	Avg. Total Tokens
1	66.32	2054	508	2562
2	67.54	5575	972	6547
3	68.49	11049	1525	12574
4	68.95	18302	2077	20379
5	68.60	27133	2609	29742
6	69.47	38317	3223	41540

Appendix

Table 8: Effect of the number of agents on performance and token cost on MMLU. A larger collective typically yields better accuracy, especially in the low-agent regime, while substantially increasing token consumption as the system scales.

# Rounds	AMC	Avg. Input Tokens	Avg. Output Tokens	Avg. Total Tokens
1	18.33	455	2195	2650
2	22.50	13406	3480	16886
3	31.67	24129	4580	28709
4	40.83	33287	5203	38490
5	35.00	43984	6627	50611

Appendix

Table 9: Effect of the number of collaborative rounds on performance and token cost on AMC. Accuracy improves substantially as the interaction depth increases from one to four rounds, but declines when an additional round is added, indicating diminishing returns from over-iteration.

# Rounds	MMLU	Avg. Input Tokens	Avg. Output Tokens	Avg. Total Tokens
1	0.6404	5352	933	6285
2	0.6491	6385	1251	7636
3	0.6849	11049	1525	12574
4	0.6930	15318	1798	17116
5	0.6895	19689	2144	21833

Appendix

Table 10: Effect of the number of collaborative rounds on performance and token cost on MMLU. A moderate number of rounds improves accuracy, while further increasing the interaction depth yields only limited benefit and slightly higher cost.

Figure 8: Real human-in-the-loop collaboration in HILA with bidirectional interaction. Real human experts can act proactively by injecting prior knowledge at initialization (e.g., a high-level human_idea or a full human_reasoning ), shaping the agents’ starting state before autonomous collaboration begins. They can also act reactively by providing corrective feedback when the meta-policy selects DEFER . This unified view highlights that human expertise in HILA is not limited to post-hoc supervision, but can be integrated both as upfront guidance and as on-demand intervention during multi-round collaboration.

Figure 9: Real human feedback pipeline in HILA. Upon a help request ( DEFER ), the system routes the query to a pool of human experts, who provide either high-level human ideas or full human reasoning . The collected feedback is then stored as structured supervision for subsequent training and continual improvement.

Expert	GSM8K	AMC	MMLU
LLaMA3-8B	58.67	16.67	57.33
GPT-3.5-Turbo	66.00	17.50	58.00
GPT-4o-mini	85.67	39.17	86.67
GPT-4o	88.33	46.67	87.33
Human Experts	95.00	97.50	88.00

Appendix

Table 11: Standalone performance of candidate external experts on the human-evaluation benchmarks. Human Experts denote the aggregated real-expert pool collected from 20 PhD participants. All values are reported as percentages.

Reactive Expert in HILA	GSM8K	AMC	MMLU
GPT-3.5-Turbo	61.67	20.83	59.67
GPT-4o-mini	74.33	29.17	64.33
GPT-4o	77.33	33.33	67.33
Human Experts	78.67	61.67	75.33

Appendix

Table 12: Performance of HILA with different reactive experts, i.e., experts consulted only when the system selects DEFER . The autonomous backbone is fixed to LLaMA3-8B. All values are percentages.

Human Participation Setting	GSM8K	AMC	MMLU
GPT-4o as reactive expert only	77.33	33.33	67.33
Human Experts as reactive expert only	78.67	61.67	75.33
Human proactive idea + GPT-4o reactive expert	78.33	34.17	79.33
Human proactive full reasoning + GPT-4o reactive expert	90.67	65.83	86.67

Appendix

Table 13: Comparison of different real human participation modes under HILA. In the proactive setting, human information is injected before the initial agent reasoning, while GPT-4o remains available as the reactive expert for later DEFER decisions. All values are percentages.

Figure 10: Action distribution under HILA when real humans act as reactive experts , meaning that feedback is provided only after the system chooses DEFER . Across datasets, CREATE and DEFER dominate on GSM8K and AMC, while MMLU shows a noticeably larger proportion of EVAL .

Figure 11: Action distribution under HILA when humans provide a proactive idea at initialization, while GPT-4o remains available as the reactive expert for later DEFER decisions. Results are shown for GSM8K, AMC, and MMLU.

Figure 12: Action distribution under HILA when humans provide full proactive reasoning at initialization, while GPT-4o remains available as the reactive expert for later DEFER decisions. Results are shown for GSM8K, AMC, and MMLU.

Human-in-the-Loop AI

Human-in-the-loop AI combines human feedback, oversight, and decision-making with machine learning and autonomous systems.

See all 288 papers

Interactive Learning for LLM Reasoning
Sep 30, 2025Hehai Lin, Shilei Cao, Sudong Wang +5RL for Language Model Reasoning LLM Agent Training
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University · 4National University of Singapore +2
AgentCollab: A Self-Evaluation-Driven Collaboration Paradigm for Efficient LLM Agents
Mar 27, 2026Wenbo Gao, Renxi Liu, Xian Wang +8LLM Agents LLM Agent Workflow Optimization
Huawei · Hong Kong Polytechnic University
MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation
May 17, 2026Chenyu Wang, Yang ShuLLM Agent Orchestration LLM Agent Evaluation
School of Computer and Artificial Intelligence, Zhengzhou University, Zhengzhou, China · Zhejiang University, Hangzhou, China

Sep 30, 2025J/K move · Enter open · S save

Interactive Learning for LLM Reasoning

Hehai Lin, Shilei Cao, Sudong Wang, Haotian Wu+4

1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University · 4National University of Singapore+2

RL for Language Model Reasoning LLM Agent Training Multi-Agent Reasoning Multi-LLM Collaboration

Open paper AskPDF arXiv

Existing multi-agent learning approaches have developed interactive training environments to explicitly promote collaboration among multiple Large Language Models (LLMs), thereby constructing stronger multi-agent systems (MAS). However, during inference, they require re-executing the MAS to obtain final solutions, which diverges from human cognition that individuals can enhance their reasoning capabilities through interactions with others and resolve questions independently in the future. To investigate whether multi-agent interaction can enhance LLMs' independent problem-solving ability, we introduce ILR, a novel co-learning framework for MAS that integrates two key components: Dynamic Interaction and Perception Calibration. Specifically, Dynamic Interaction first adaptively selects either cooperative or competitive strategies depending on question difficulty and model ability. LLMs then exchange information through Idea3, an innovative interaction paradigm designed to mimic human discussion, before deriving their respective final answers. In Perception Calibration, ILR employs Group Relative Policy Optimization (GRPO) to train LLMs while integrating one LLM's reward distribution characteristics into another's reward function, thereby enhancing the cohesion of multi-agent interactions. We evaluate the effectiveness of ILR across three LLMs from two model families of varying scales on five mathematical, one coding, one general question answering, and one scientific reasoning benchmarks. Experimental results show that ILR consistently outperforms single-agent learning, yielding an improvement of up to 5% over the strongest baseline. We further discover that Idea3 can enhance the robustness of stronger LLMs during multi-agent inference, and dynamic interaction types can boost multi-agent learning compared to pure cooperative or competitive strategies.

Abstract

Human-in-the-Loop AI

Explore similar work

Interactive Learning for LLM Reasoning

AgentCollab: A Self-Evaluation-Driven Collaboration Paradigm for Efficient LLM Agents

MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation