TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks

Xiangyu Wang; Jin Wu; Haoran Shi; Wei Xia; Jiarui Yu; Chanjin Zheng

cs.CLApr 8, 2026

TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks

Authors: Xiangyu Wang, Jin Wu, Haoran Shi, Wei Xia, Jiarui Yu, Chanjin Zheng

Organizations: Department of Educational Psychology, East China Normal University · Lab of Artificial Intelligence for Education, East China Normal University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · School of Computer Science and Technology, East China Normal University

Abstract

Recently, multi-Large Language Model (LLM) frameworks have been proposed to solve contextualized tasks. However, these frameworks do not explicitly emulate human team role division, which may lead to a single perspective, thereby weakening performance on multi-step contextualized tasks. To address this issue, we propose TeamLLM, a human-like Team-Oriented Multi-LLM Collaboration Framework. TeamLLM adopts four team roles with distinct division and employs a three-phase multi-LLM collaboration for multi-step contextualized tasks. To evaluate the effectiveness of TeamLLM on multi-step contextualized tasks, we propose Contextually-Grounded and Procedurally-Structured tasks (CGPST) and construct the CGPST benchmark. This benchmark has four core features: contextual grounding, procedural structure, process-oriented evaluation and multi-dimensional assessment. We evaluate ten popular LLMs on CGPST at overall-level, step-level, and dimension-level. Results show that TeamLLM substantially improves performance on CGPST. We release the benchmark with scenarios, full-process responses and human scores from ten LLMs. The code and data are available at https://anonymous.4open.science/r/TeamLLM-anonymous-C50E/.

Figures & tables

Figure 2: TeamLLM: A Human-Like Team-Oriented Collaboration Framework.

	Multi-step	Evaluation
CGPST	✓	process-oriented
TTCW ( Chakrabarty et al., 2024 )	✗	result-oriented
NOVELTYBENCH ( Zhang et al., 2025b )	✗	result-oriented
CREATIVEMATH ( Ye et al., 2025 )	✗	result-oriented
AidanBench ( McLaughlin et al., 2024 )	✗	result-oriented
LiveIdeaBench ( Ruan et al., 2025 )	✗	result-oriented

Table 1: Comparison between the CGPST benchmark and several contextualized benchmarks.

Step	Requirement	Evaluation Metric	Score
Step-1: Identify Challenges	Identify up to 8 reasonable challenges based on the future scenario.	Fluency, Flexibility, Elaboration, Originality	48
Step-2: Select an Underlying Problem	Select the most promising and meaningful challenge from Step-1 as the underlying problem.	Integrity of Underlying Problem, Focus of Underlying Problem, Importance / Adequacy of Underlying Problem	30
Step-3: Produce Solutions	Generate up to 8 solutions for the underlying problem from Step-2.	Fluency, Flexibility, Elaboration, Originality	48
Step-4: Select Criteria	Generate 5 evaluation criteria for the solutions from Step-3.	Correctly Written, Relevance to the Underlying Problem	20
Step-5: Apply Criteria to Top Solution	Rank the solutions from Step-3 using the criteria from Step-4 and select the highest-scoring solution.	Correctly Used	5
Step-6: Develop an Action Plan	Develop the top solution from Step-5 into a comprehensive action plan to address the underlying problem from Step-2.	Relevance, Effectiveness, Criteria in Development of Action Plan, Impact, Humaneness, Development of Action Plan	35

Table 2: Detailed Step-wise Information and Evaluation Metrics on CGPST

Category	Model (Abbreviation)
Open-Source	qwen3-235b-a22b-instruct-2507 ( qwen3-instruct ), qwen3-235b-a22b-thinking-2507 ( qwen3-thinking ), DeepSeek-V3-0324 ( deepseek-v3 ), Deepseek-r1 ( deepseek-r1 ), kimi-k2-0711-preview ( kimi-k2 ), llama-4-scout-17b-16e-instruct ( llama-4-scout )
Closed-Source	gpt-4o ( gpt-4o ), gpt-5 ( gpt-5 ), claude-opus-4-1-20250805 ( claude-4-opus ), gemini-2.5-pro ( gemini-2.5-pro )

Table 3: LLMs Selected for Evaluation

Model	FS1	FS2	FS3	FS4	FS5	FS6	FS7	FS8	FS9	FS10	Avg
qwen3-instruct - TeamLLM	135.5	142	127	123.5	120.5	142	156	135.5	137	134.5	135.35 ∗∗
qwen3-instruct - baseline	116	113	110	122.5	107.5	127.5	125.5	113	117	110	116.2
qwen3-thinking - TeamLLM	115	133.5	135	124	117.5	140.5	146.5	111	116	112	125.1 ∗∗
qwen3-thinking - baseline	100.5	108.5	121	120.5	115	126.5	130.5	111.5	103.5	108	114.55
deepseek-v3 - TeamLLM	129	137	130.5	116.5	124.5	114	121.5	102.5	124.5	130	123 ∗∗
deepseek-v3 - baseline	114.5	104.5	91	104.5	110	110	116	105.5	104	109.5	106.95

Table 4: Overall performance across 10 CGPST scenarios with Wilcoxon signed-rank test results. Bold indicates the higher score. “Avg” is the model’s mean score, and ∗∗ marks a significant improvement under TeamLLM (p=0.01).

Model	TeamLLM	Baseline
qwen3-thinking	1.7	0
kimi-k2	2.4	0
gemini-2.5-pro	2.9	0.4

Table 5: Average number of blank responses in Step-3.

Metric	Before Calibration	After Calibration
PCC	0.71	0.87
ICC	0.58	0.84

Table 6: Consistency Before and After Calibration.

Figure 3: Step-level performance of TeamLLM (right bars) and the baseline (left bars), with different colors representing distinct steps. Horizontal lines with ∗∗ indicate statistically significant differences at p=0.01.

Figure 4: Comparison of model performance in Step-3 across Flexibility Efficiency, Originality Efficiency, and Elaboration Efficiency. Solid bars correspond to the baseline, while hatched bars correspond to TeamLLM.

Figure 5: Ablation study results.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Comparison of Diversity and Flexibility in Step-1.

Figure 7: Comparison of Diversity and Flexibility in Step-3.

Hyper-parameter	Condition / Role	Value
temperature	TeamLLM – Co-Ordinator	0.5
	TeamLLM – Plant	0.8
	TeamLLM – Monitor Evaluator	0.4
	TeamLLM – Implementer	0.4
	Baseline	0.6
top-p	–	0.9

Appendix

Table 7: Hyper-parameter Settings

Type	Description
Perhaps	Ambiguous
Why	Irrelevant to Scenario
Solution	A Solution to a Challenge
Duplicate	Similar to Another YES Challenge
Blank	No Response Provided

Appendix

Table 8: Types of invalid challenges in Step-1.

Type	Description
Perhaps	The relationship between the solution, the KVP, and the purpose is unclear.
Why	The solution is irrelevant to the underlying problem.
Duplicate	The solution is too similar to another Yes solution.
Blank	No response provided.

Appendix

Table 9: Types of invalid solutions in Step-3.

Prompts Variables	Content
$Role\_Play\ prompt$	Your role in the team is {Team_Role}, with the specialty of {Role_Speciality}. The other team members are {Other_Members}, each performing different team roles and responsibilities. {Role_Prompt}
$warm\_up\ prompt$	Before starting the task, please guide the team members through a warm-up interaction to help them get to know each other and foster a collaborative atmosphere. Encourage each member to briefly introduce their role and responsibilities and greet everyone. Use a natural and friendly tone to open the session, inviting members to speak one by one to complete this warm-up activity. Please note that this stage should focus only on team warm-up and not involve the task itself.
$Task\_Initiation\ prompt$	The information for Step {Step_Number} is as follows: {Step_Name}, {Step_Description}, {Step_Output} Please clearly and concisely communicate the information to the team, encouraging members to leverage their strengths to complete it together. Note that you do not need to assign specific roles; just convey the task information.
$Consensus\_Building\ prompt$	The discussion for Step {Step_Number} has concluded. Based on the contributions of the team, synthesize their opinions and viewpoints to produce the final answer for this step. Note that the final answer should meet the requirements in [Step_Output]. Please provide the answer directly without any explanation.

Appendix

Table 10: Prompts variables in TeamLLM.

Category	Roles	Description	Strengths
Thinking	Plant	Tends to be highly creative and good at solving problems in unconventional ways.	Creative, imaginative, free-thinking, generates ideas and solves difficult problems.
	Monitor Evaluator	Provides a logical eye, making impartial judgements where required and weighs up the team’s options in a dispassionate way.	Sober, strategic and discerning. Sees all options and judges accurately.
	Specialist	Brings in-depth knowledge of a key area to the team.	Single-minded, self-starting and dedicated. They provide specialist knowledge and skills.
Action	Shaper	Provides the necessary drive to ensure that the team keeps moving and does not lose focus or momentum.	Challenging, dynamic, thrives on pressure. Has the drive and courage to overcome obstacles.
	Implementer	Needed to plan a workable strategy and carry it out as efficiently as possible.	Practical, reliable, efficient. Turns ideas into actions and organises work that needs to be done.
	Completer Finisher	Most effectively used at the end of tasks to polish and scrutinise the work for errors, subjecting it to the highest standards of quality control.	Painstaking, conscientious, anxious. Searches out errors. Polishes and perfects.

Appendix

Table 11: Belbin Team Roles.

TeamLLM

Baseline

You are about to participate in a contextualized task. In this task, you will receive a scenario about future societal issues. Please work with your team to complete the problem solving task according to your assigned role and responsibilities. You will participate as a team member in all six steps of the problem solving process. Teamwork is key to success! You are expected to actively listen to others’ ideas, clearly express your own viewpoints, and collaborate based on your role, engaging in a systematic discussion throughout the task. Please maintain an active dialogue, continuously communicate and coordinate with your teammates, and work together to complete the task. The future scenario for this task is as follows: { future scenario } Please carefully read the above scenario and use it as the basis for participating in the subsequent problem solving task. Please note that during the task, you must adhere to the following requirements: 1. Actively collaborate with your team members, carefully consider their contributions, and work together to advance the task. 2. Clearly understand and adhere to your assigned team role and responsibilities, ensuring consistency in role-playing. 3. Follow the six-step problem solving process strictly. 4. All responses should be in Chinese, with clear language and accurate logic. 5. For each response, only output what you intend to say—do not include any role descriptions, labels, or guiding text.

You are about to participate in a contextualized task. In this task, you will receive a scenario about future societal issues. Please follow the task steps and independently complete the task. The future scenario for this task is as follows: { future scenario } Please carefully read the above scenario and use it as the basis for participating in the subsequent problem solving task. Please note that during the task, you must adhere to the following requirements: 1. For each response, only output what you intend to say—do not include any explanations, labels, or guiding text. 2. All responses should be in Chinese, with clear language and accurate logic.

Appendix

Table 12: Meta Prompts for TeamLLM and Baseline Conditions.

Team_Role	Role_Speciality	Role_Prompt
Co-Ordinator	Team guidance, task organization, consensus integration	As the Co-Ordinator of the team, your primary responsibility is to organize and guide team collaboration, drive task progress, and integrate perspectives. Your contributions should demonstrate leadership and structure, clearly guiding the team to focus on the current step, engaging each member to fulfill their role, and synthesizing everyone’s response to form team consensus.
Plant	Innovative thinking, generating novel solutions, proposing unusual yet potentially valuable ideas	As the Plant of the team, your main responsibility is to propose creative, even unconventional ideas to inspire the team. Your contributions should be imaginative and open-minded, daring to challenge conventions. Focus on the current task and generate as many creative ideas as possible to provide a draft for team discussion.
Monitor Evaluator	Rational analysis, logical judgment, reasoning	As the Monitor Evaluator of the team, your main responsibility is to evaluate the Plant’s contributions from a rational and logical perspective. Focus on the current task step, applying your reasoning and analytical skills to assess the feasibility and logical consistency of the Plant’s ideas, providing guidance for the team’s final integrated answer.
Implementer	Task execution, feasibility analysis, pragmatic implementation	As the Implementer of the team, your main responsibility is to assess the Plant’s ideas from a feasibility and practical implementation perspective. Focus on the current task step, leveraging your expertise in pragmatism and execution to provide recommendations that ensure the team’s final integrated answer is actionable and realistic.

Appendix

Table 13: Role play prompts. Each role is defined as a triplet {Team_Role, Role_Speciality, Role_Prompt} guiding the LLM’s behavior.

Step_Name	Step_Description	Step_Output
Step-1: Identify Challenges	Please carefully read the future scenario and identify up to eight challenges or problems specifically related to the scenario context.	Please describe each challenge in a complete sentence, following these requirements: 1. Use declarative sentences. 2. Use modal verbs like “might,” “could,” or “should.” 3. Explain what the challenge is, why it is a challenge, and how it connects to the scenario. 4. Number each challenge, e.g., “1. xxx.”
Step-2: Select an Underlying Problem	From the challenges in Step-1, select the most impactful one and reformulate it into a focused core problem statement.	Provide a complete description including: 1. Challenge number: the identifier of the specific challenge from Step 1 that is being developed into the underlying problem. 2. Conditional phrase (CP): a fact or condition drawn from the future scenario, which provides the theoretical or situational basis for the problem. 3. Stem + Key Verb Phrase (KVP): the core phrasing of the underlying problem, usually starting with “How might we…” or “In what ways can we…”. The KVP should contain only one active verb specifying the main action or intervention to be taken, and should avoid absolute or overly broad verbs to ensure focus and feasibility. 4. Purpose: typically expressed with “in order to” or “so that”, clarifying the intended goal of the KVP. 5. Future scenario parameters: the three parameters of time, location, and theme that situate the underlying problem within the scenario.
Step-3: Produce Solutions	Generate up to eight possible solutions based on the underlying problem.	Each solution should: 1. Each solution must be written as a complete sentence. 2. Use “will” rather than “might” to indicate certainty. 3. Each solution should address at least three of the following aspects: Who, What, How, Why, When, and Where. 4. Ensure alignment with the key verb phrase (KVP) and the intended purpose of the underlying problem. 5. Begin each solution with a number, e.g., “1. …”.
Step-4: Select Criteria	Create five criteria to evaluate the solutions.	Each criterion should: 1. Be properly phrased: single dimension, superlatives as needed, indicate evaluation direction, phrased as a question 2. Be relevant to the underlying problem 3. Numbered, e.g., “1. xxx”
Step-5: Apply Criteria to Top Solution	Evaluate the eight solutions from Step-3 using the criteria from Step-4 in a matrix format.	Please provide the answers for this step in the following matrix (grid) format: Solution ID \| Criterion 1 \| Criterion 2 \| Criterion 3 \| Criterion 4 \| Criterion 5 \| Total Score 1 \| 5 \| 7 \| 6 \| 4 \| 8 \| 30 2 \| 8 \| 6 \| 7 \| 5 \| 7 \| 33 ... \| ... \| ... \| ... \| ... \| ... \| ... Highest-scoring solution ID: <ID>, Solution: <content of the highest-scoring solution> Requirements: 1. For each criterion, all solutions must be scored. 2. Scores for each criterion should range from 1 to $x$ , where 1 represents the worst-performing solution and $x$ represents the best. 3. No two solutions may receive the same score under the same criterion; i.e., each column must be a unique permutation of 1 to $x$ . 4. Provide both the full scoring matrix and the ID and content of the highest-scoring solution.
Step-6: Develop an Action Plan	Develop the top solution from Step-5 into an actionable plan.	Develop the highest-scoring solution selected in Step-5 into a comprehensive action plan. The plan should systematically and thoroughly explain how the underlying problem is addressed, how the solution will be implemented, and how the future scenario will be impacted. Please provide the final action plan in the following format: Best Solution ID: # (fill in the solution ID) Best Solution Content: (fill in the content of the best solution) Action Plan: (full, detailed action plan)

Appendix

Table 14: Step-wise Task Requirements and Outputs on CGPST.

Step	Dimension	Rubric	Max Score
Step-1: Identify Challenges	Fluency	Each reasonable challenge is counted as Yes ; one Yes challenge = 1 point.	8
	Flexibility	If a Yes challenge belongs to a new category, 1 point is awarded.	8
	Elaboration	If a Yes challenge has a clear logical-causal relationship and is described in detail, 2 points are awarded.	16
	Originality	If a Yes challenge demonstrates unique creativity, 2 points are awarded.	16
Step-2: Select an Underlying Problem	Integrity: CP	Whether it is connected to the future scene and KVP.	2
	Integrity: Stem & KVP	Contains only one clear active verb phrase.	3

Appendix

Table 15: Evaluation Rubrics for CGPST Benchmark.

Ranking errors (A06_FS1): The ranking scores assigned by each criteria are not a permutation of 1 to the total number of solutions (i.e., 8).

Solution ID | Criterion 1 | Criterion 2 | Criterion 3 | Criterion 4 | Criterion 5 | Total Score | 1 | 8 | 6 | 7 | 9 | 5 | 35 | | 2 | 5 | 8 | 9 | 7 | 8 | 37 | | 3 | 9 | 5 | 6 | 8 | 9 | 37 | | 4 | 6 | 9 | 8 | 6 | 7 | 36 | | 5 | 7 | 7 | 5 | 9 | 6 | 34 | | 6 | 4 | 4 | 9 | 8 | 8 | 33 | | 7 | 3 | 9 | 7 | 5 | 9 | 33 | | 8 | 2 | 6 | 8 | 7 | 5 | 28 | The solution with the highest total score is: 2. The solution is: The government will formulate and enforce strict cybersecurity regulations, requiring companies developing autonomous driving systems to comply with specific safety standards. Regular security audits and evaluations will be conducted on these systems, with full implementation scheduled to be completed in Istanbul by 2052.

Ranking errors (A06_FS5): The ranking matrix is incomplete.

Solution ID | Criterion 1 | Criterion 2 | Criterion 3 | Criterion 4 | Criterion 5 | Total Score | 9 | 9 | 9 | 9 | 9 | 9 | 45 | The solution with the highest total score is: 9. The solution is: By leveraging technologies such as artificial intelligence, blockchain, and the Internet of Things (IoT), a decentralized platform for cherry tree breeding and disease resistance monitoring will be established. This platform can connect cherry farmers, research institutions, and government departments in Tasmania, enabling data sharing, resource integration, and collaborative work. It will provide functions including real-time monitoring, predictive analysis, and smart contract-based incentives, so as to accelerate the cultivation of disease-resistant cherry tree varieties and improve the risk resilience of Tasmania’s cherry industry.

Non-compliant responses (A06_FS4): The model fails to apply five criteria generated in Step-4 to rank the solutions with the scoring matrix.

Appendix

Table 16: Step 5 Answer Instances of llama-4-scout .

Figure 8: Dimension-level performance of TeamLLM (red) and baseline (blue) across all steps.

Panel of Figure 9 — Figure 9: Two representative pages of the Excel scoring sheet designed for human evaluation, with some annotated usage instructions.

FS10: Ocean Soup Future Scenario

As Jobie Sakai leans on the railing of the Ola Kai, she admires the beauty of her ocean paradise. Jobie, a fifth generation Hawaiian devoted to the future of her homeland, is dedicated to her job as an environmental chemist aboard the Ola Kai #6, one of Hawai’i’s floating science laboratories. The Ola Kai (meaning "healthy ocean") Project is a combined effort of the Hawaiian Environmental Council and the University of Hawai’i. Now, in 2035 after 15 active years, the project is struggling to live up to its nickname: the OK Project. Originally, the OK Project focused on the waters affected by Hawai’i’s island-generated pollutants. Public interest in the project led to a resurgence in eco-education; recycling and the reduced use of plastics became an accepted part of island life for Hawai’i residents. The "adopt a beach" clean-up program became a popular draw for eco-tourists. However, researchers like Jobie became increasingly aware that their efforts were not enough. The world’s largest manufacturers of plastic products border both sides of the Pacific. A ten million square mile system of rotating currents called the North Pacific Gyre has its axis near the 137 islands of the Hawaiian chain. Pacific environmental regulations have historically been weak or disregarded by heavy industrial nations who continue to use these waters as a dumping ground. Consequently, the 1500 mile-long archipelago paradise has been attacked by ocean soup for many years. The soup surrounds Hawai’i, placing the islands and their resources at risk of permanent damage. Especially vulnerable are the sparsely-inhabited northwest islands, the world’s largest protected marine sanctuary that is home to many endangered fish, birds, seals, and Hawai’i’s beleaguered fishing industry. Eco-tourism routes have been altered to reduce impact on indigenous species and circumnavigated due to the location of floating laboratories. With the increasing damaging effects from ocean soup on the island chain, Jobie and her coworkers realized that the Ola Kai Project’s numerous floating labs had to broaden their territory while narrowing their focus. Due to the scope of the damage, the project directors reached out to other groups working in the Pacific and consulted with the National Oceanic and Atmospheric Administration (NOAA). It was determined that the best approach would be to divide up the responsibilities among agencies. Now the OK Project labs focus solely on the battle against microplastics, leaving the collection of larger trash to other organizations. Ola Kai laboratory crews record various data for analysis, keeping track of multiple fish species and beneficial organisms like plankton. Specified lab crews weigh the microplastic debris collected on a weekly basis and compile that data while Jobie and other chemists continue their in-depth examination of the plastic degradation and its effect on the waters surrounding the islands. Collection of debris that is smaller than a pencil eraser has often done more harm than good to sea life. After experimenting with several collection methods, the OK Project currently uses below-surface robotic collectors that randomly collect plastic particles being carried by the currents. Project teams are also experimenting with alternative collection methods, including new nanofiber sieves and use of pollution-dissolving lasers. Ola Kai’s floating labs have plastic-to-fuel conversion systems capable of harvesting tons of plastic pollution and converting it into diesel fuel for the labs, thus eliminating the need to return to shore for disposal of the waste in a landfill. In spite of their progress, water samples still show an alarming amount of plastic particles.

Appendix

Table 17: A complete future scenario FS10 used on CGPST.

Challenges	Score Item	H01	H02
1. The concentration of microplastics may have exceeded the density of plankton by tenfold, disrupting the energy input of the base food chain. This challenge arises from the warning in the scenario that "water samples still show an alarming amount of plastic particles."	Category Elaboration Originality	Environment 1 0	Environment 1 0
2. Subsurface robotic collectors may miss low-velocity eddy zones, creating data gaps and masking local ecological collapse points. This is directly related to the scenario’s mention that "after experimenting with several collection methods," weekly weighing is still required, implying sampling limitations.	Category Elaboration Originality	Technology 1 1	Technology 1 1
3. The plastic-to-fuel conversion system may emit nanoscale black carbon particles, which could exacerbate imbalances in ocean surface heat absorption. This challenge stems from the scenario emphasizing "eliminating the need to return to shore for disposal" without addressing potential secondary emissions.	Category Elaboration Originality	Technology 1 0	Technology 1 1
4. Endangered species in the northwest islands may face unknown toxicological effects from ingesting micro-fragments of plastics broken down by lasers. The scenario mentions that laser technology "can dissolve pollutants" but does not evaluate the byproducts of fragmentation.	Category Elaboration Originality	Environment 1 0	Environment 1 1
5. Adjustments to eco-tourism routes around floating laboratories may transfer visitor pressure to other more fragile reefs. This challenge is directly related to the scenario’s note that routes were "altered to reduce impact" but without ensuring overall pressure reduction.	Category Elaboration Originality	Recreation 1 0	Recreation 1 1
6. Legal exemptions for manufacturers on both sides of the Pacific may cause Hawaiian regional governance to operate in isolation. This challenge is closely related to the scenario’s statement that "environmental regulations have historically been weak or disregarded."	Category Elaboration Originality	Law & Justice 1 0	Law & Justice 1 1

Appendix

Table 18: Step-1 (Identify Challenges) of A05_FS10.

Responses
Challenge ID: 1. In 2035, in the waters surrounding the Hawaiian Islands, when microplastic concentrations may have exceeded the density of plankton tenfold, threatening the energy input at the base of the food chain, how can we reduce the overwhelming proportion of microplastics at the bottom of the food chain to restore energy flow in the Pacific ecosystem and protect the future of endangered species? Time: 2035 Location: Waters surrounding the Hawaiian Islands Theme: Microplastic Management and Ecological Restoration

Rater Input (This field requires human raters to manually input it in the scoring sheet.)
Input Item	H01	H02
Condition Phrase	"microplastic concentrations may have exceeded the density of plankton tenfold, threatening the energy input at the base of the food chain"	"In 2035, in the waters surrounding the Hawaiian Islands, when microplastic concentrations may have exceeded the density of plankton tenfold"
Stem	"how can we"	"how can we"

Appendix

Table 19: Step-2 (Select an Underlying Problem) of A05_FS10.

Solutions	Score Item	H01	H02
1. The Ola Kai project chemistry team will deploy glycosylated nanosponges, dispersing 2 tons within a 20 km radius of the Ola Kai mooring point by August 2035. Subsurface robots will recover the flocs and recycle them through the onboard plastic-to-diesel system, directly reducing microplastic ingestion by plankton, lowering the proportion of plastics at the base of the food chain, restoring energy flow, and protecting endangered species.	Category Elaboration Originality	Technology 2 0	Technology 2 1
2. Google X Lab and Hawaiian community divers will run a crowdsourced "photoacoustic unmanned vessel + AR snorkeling goggles" collection program across the northwest islands by December 2035. Unmanned vessels will map microplastic clouds in real-time using laser sonar, while AR glasses guide divers to precise retrieval points, clearing high-density fragments weekly to reduce microplastic accumulation at the base of the food chain and preserve Pacific ecosystem energy balance.	Category Elaboration Originality	Technology 2 0	Technology 2 2
3. NOAA and Hawaiian Electric will pilot a 2-nautical-mile-diameter "bubble curtain + photocatalytic net" system north of Kaua’i by October 2035. Wave-driven pumps will concentrate microplastics, which are then broken down by photocatalytic nets into short-chain acids absorbable by phytoplankton, reducing microplastic dominance on plankton and restoring baseline energy input.	Category Elaboration Originality	Technology 2 0	Technology 2 1
4. SpaceX and a local high school team will launch the CubeSat constellation "KiloEye" by July 2035. Weekly scans of the 137 Hawaiian Islands will use hyperspectral data to direct Ola Kai drones for targeted microplastic removal, lowering the risk of plankton mis-ingestion at the source and ensuring Pacific ecosystem energy flow is rebalanced.	Category Elaboration Originality	✗Duplicate ✗ ✗	✗Duplicate ✗ ✗
5. Japan’s SpiraNova and the University of Hawai’i will plant 300 "biopolymer-coated kelp ropes" off the west coast of the Big Island in Q3 2035. Kelp leaves will adsorb microplastics, and harvested ropes will be processed into high-value composites, directly removing plastics at the base of the food chain and generating revenue while protecting endangered species’ habitats.	Category Elaboration Originality	Technology 2 2	Technology 2 2
6. The Ola Kai project biology team will release living blue-green algae “Plastic Sentinel” strains in a 500-hectare demonstration area off the northwest reef of the main Hawaiian Island by November 2035. These algae continuously secrete degrading enzymes to break down 0.1–1 mm microplastics, reducing plankton ingestion, restoring baseline energy input, and protecting the Pacific ecosystem.	Category Elaboration Originality	Technology 2 2	Technology 2 1

Appendix

Table 20: Step-3 (Produce Solutions) of A05_FS10.

Criteria	Score Item	H01	H02
1. Which approach can most rapidly reduce the net microplastic content inside plankton?	Correctly Written Relevance	1 2	1 2
2. Which pathway requires the lowest one-time capital investment?	Correctly Written Relevance	1 0	1 1
3. Which model has the highest potential for rapid global replication and scaling?	Correctly Written Relevance	1 1	1 1
4. Which technology poses the least risk of secondary ecological disturbance to endangered species?	Correctly Written Relevance	1 1	1 2
5. Which solution maintains the lowest full-life-cycle carbon footprint over 15 years?	Correctly Written Relevance	1 2	1 2
Step-4 Score: H01: { Correctly Written : 5, Relevance : 6, Total Score : 11} H02: { Correctly Written : 5, Relevance : 8, Total Score : 13} NOTE: 1. All criteria satisfy the requirements of single dimension , superlatives as needed , clear indication of evaluation direction , and phrasing as questions , and thus are all "correctly written". 2. The rubrics for Relevance are defined as follows: 0 points for irrelevance or repetition ; 1 point for being vague and unspecific ; 2 points for being relatively specific yet improvable ; 3 points for being clear, specific and highly relevant . 3. The rubrics and detailed explanations for the two dimensions are available in the scoring sheet .

Appendix

Table 21: Step-4 (Select Criteria) of A05_FS10.

Responses	Score Item	H01	H02
Solution ID \| Criterion 1 \| Criterion 2 \| Criterion 3 \| Criterion 4 \| Criterion 5 \| Total Score 1 \| 8 \| 6 \| 7 \| 7 \| 6 \| 34 2 \| 6 \| 8 \| 5 \| 6 \| 7 \| 32 3 \| 7 \| 3 \| 6 \| 3 \| 5 \| 24 4 \| 5 \| 5 \| 8 \| 5 \| 4 \| 27 5 \| 4 \| 4 \| 4 \| 4 \| 3 \| 19 6 \| 3 \| 7 \| 3 \| 5 \| 8 \| 26 7 \| 2 \| 2 \| 2 \| 2 \| 2 \| 10 8 \| 1 \| 1 \| 1 \| 1 \| 1 \| 5 The solution with the highest total score is: 1. The solution is: The Ola Kai project chemistry team will deploy glycosylated nanosponges, dispersing 2 tons within a 20 km radius of the Ola Kai mooring point by August 2035. Subsurface robots will recover the flocs and recycle them through the onboard plastic-to-diesel system, directly reducing microplastic ingestion by plankton, lowering the proportion of plastics at the base of the food chain, restoring energy flow, and protecting endangered species.	Correctly Used	4	4
Step-5 Score: H01: { Correctly Used : 4, Total Score : 4} H02: { Correctly Used : 4, Total Score : 4} NOTE: In the score matrix, the ranking for Criterion 4 does not follow the required 1–8 sequence (there are two 5s and no 8), resulting in an error in the matrix. Therefore, a score of 4 is assigned. Detailed scoring rubrics can be found in the human evaluation sheet.

Appendix

Table 22: Step-5 (Apply Criteria to Top Solution) of A05_FS10.

Solutions	Score Item	H01	H02
The solution with the highest total score is: 1. The solution is: The Ola Kai project chemistry team will deploy glycosylated nanosponges, dispersing 2 tons within a 20 km radius of the Ola Kai mooring point by August 2035. Subsurface robots will recover the flocs and recycle them through the onboard plastic-to-diesel system, directly reducing microplastic ingestion by plankton, lowering the proportion of plastics at the base of the food chain, restoring energy flow, and protecting endangered species. Action Plan: 1. Problem Closure Nanosponges use a "molecular magnet" mechanism to selectively adsorb 0.1–1 mm microplastics, aggregating them into millimeter-scale flocs that cannot be ingested by plankton. Recovered flocs are immediately converted into diesel, achieving a “collect-convert-use” zero-waste cycle, directly reducing the overwhelming proportion of microplastics at the base of the food chain. 2. Implementation Steps and Timeline •Phase A – R&D and Validation (Now–Oct 2025): Ola Kai Chemistry Team × MIT Materials Department iterate the third-generation biodegradable nanosponges and complete biotoxicity-degradation tests. •Phase B – Pilot Demonstration (Nov 2025–Apr 2026): Deploy 100 kg in South Bay, Oahu; 30-day monitoring shows $\geq 70\%$ reduction of microplastics in plankton. •Phase C – Scale-up Production (May 2026–Feb 2027): Hawaii biorefinery produces 2 tons per month and establishes freeze-dry packaging chain. •Phase D – Full Deployment (Mar 2027–Jul 2035): Deploy 2 tons in March, June, September each year, covering a 20 km radius grid; six “Kokua- $\gamma$ ” subsurface robots operate in shifts, producing 400 L/day diesel for self-use. •Phase E – Monitoring and Iteration (Parallel): Weekly underwater imaging + Raman verification to maintain microplastic concentration <50% of plankton; formula updated every six months. 3. Resources and Responsibilities • Funding: Ola Kai Research $300k + NOAA Innovation Fund$ 400k + State Green Bonds $300k + Carbon Credit Pre-sale; total$ \leq1M. •Team: Chemistry team handles materials, MIT provides R&D, NOAA provides monitoring platform, State Environmental Department supervises approvals. 4. Risks and Contingency •Nanomaterial leakage: Three passive samplers monitor in real-time; $>10\,\mu\text{g}/\text{L}$ triggers magnetic recovery nets. •Robot malfunction: 1:1 spare parts + 48-hour offshore repair; if failure rate >15%, NOAA backup ROVs are deployed. •Regulatory delays: Suspension during typhoon season; stock maintained at 1.5× safety level. 5. Impacts and Scaling •Local: By 2028, microplastic content in plankton decreases by 80%, coral spawning rates increase by 30%. •Regional: By 2030, open “Nanosponges Sharing Depot” allows replication in Guam, Palau, Tuvalu. •Global: By 2032, included in IMO Green Shipping Guidelines; long-haul fleets can treat plastics in-transit, establishing a Pacific-wide “food chain firewall.”	Relevance Effectiveness Criteria in Development of Action Plan Impact Humaneness Development	3 3 4 4 3 8	3 3 3 3 2 4
	Total Score	25	18

Appendix

Table 23: Step-6 (Develop an Action Plan) of A05_FS10.

Step	H01	H02
Step-1: Identify Challenges	24	28
Step-2: Select an Underlying Problem	22	22
Step-3: Produce Solutions	30	32
Step-4: Select Criteria	11	13
Step-5: Apply Criteria to Top Solutions	4	4
Step-6: Develop an Acion Plan	25	18

Appendix

Table 24: Score Summary of A05_FS10. All total scores in the table are automatically calculated in the scoring sheet.

Category	Description
Arts & Aesthetics	Arts include expression via music, painting, sculpture, theatre, film, or other creative media. Aesthetics focuses upon beauty and whether something is pleasant to view or experience.
Basic Needs	Basic Needs include food, clothing and shelter—elements important for survival.
Business & Commerce	Business includes retail stores, restaurants, offices, corporations, etc. Commerce focuses upon the trade or the buying and selling of goods and services.
Communication	Communication deals with the exchange of information.
Culture & Religion	Culture includes the customs, arts, social institutions, and achievements of a particular group. Religion includes a system of faith or worship.
Defense	Defense involves protection, safety, security, and/or privacy. This includes physical efforts by individuals, or groups as well as protection provided by firewalls and passwords.

Appendix

Table 25: Category List for Flexibility in Step-1 and Step-3.

Multi-LLM Collaboration

Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.

See all 85 papers

May 1, 2026

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds. Experiments show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement. Code is available at https://github.com/Yydc/TeamTR.

Yi Xie, Siao Liu, Falong Fan +3

Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1

Multi-Agent System Optimization Multi-Agent Coordination Trust-Region Optimization Multi-Agent LLMs RL Fine-Tuning Multi-Agent Reinforcement Learning

Ask PDF

May 17, 2026

MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation

Multi-agent large language model (LLM) systems have shown promise for solving complex tasks through agent collaboration. However, existing frameworks assign tasks based on predefined roles without considering whether an agent can accurately assess its own competence boundaries, leading to overconfident execution on tasks beyond its expertise. Inspired by metacognition theory from cognitive science, we propose MetaCogAgent, a multi-agent LLM framework where each agent is equipped with a Metacognitive Self-Assessment Unit that evaluates task-capability alignment before execution. The framework introduces three contributions: (1) a self-assessment mechanism that estimates per-task confidence by combining verbalized uncertainty with historical capability profiles; (2) an adaptive delegation protocol that routes low-confidence tasks to better-suited agents through cross-agent evaluation; and (3) a capability boundary learning module that iteratively refines each agent's competence model via cybernetic feedback. Experiments on our constructed MetaCog-Eval benchmark (700 tasks across 5 cognitive dimensions) demonstrate that MetaCogAgent achieves 82.4% task accuracy -- 8.7% above the best routing baseline -- while using 5% fewer API calls than AutoGen and 34% fewer than ensemble voting. Ablation studies confirm that each metacognitive component contributes to overall system performance.

Chenyu Wang, Yang Shu

School of Computer and Artificial Intelligence, Zhengzhou University, Zhengzhou, China · Zhejiang University, Hangzhou, China

LLM Agent Orchestration LLM Agent Evaluation LLM Agents Multi-Agent LLMs

Ask PDF

Apr 20, 2026

STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs

Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight into compositional skill gaps of LLMs and how to improve them. To make these weaknesses visible, we propose Scaffolded Task Design (STaD) framework. STaD generates controlled variations of benchmark tasks based on the concept of scaffolding, which introduces structured, incremental support in a step-by-step manner. Rather than inspecting failures individually, this approach enables systematic and scalable probing of model behavior by identifying the specific reasoning skill compositions they lack. Treating the LLM as a black box, our experiments on six models of varying sizes reveal multiple failure points in three reasoning benchmarks and highlight each model's unique and distinct skill gaps.

Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2

IBM Research

LLM Evaluation Reasoning Evaluation Language Model Probing

Ask PDF

Abstract

Multi-LLM Collaboration

Explore similar work

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation

STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs