Multi-expert agent marketplaces -- where creators publish domain-specific AI experts for end users to consult -- are an emerging deployment pattern for language technology. A published expert can receive many requests yet little evidence about its creator, who remains active elsewhere: the cold-expert problem. We frame continuous cognitive distillation as converting attributed creator observations into revisable decision rules, communication patterns, and domain procedures. CogniLife proposes a cross-expert attribution design for this problem, with Creator Profiles for transferable creator patterns and Expert Journals for listing-specific knowledge; this design is not yet implemented or evaluated. A deployed platform partially realizes the surrounding lifecycle through creator-authorized authoring, capability-owned proposal filtering, and reversible consolidation of eligible owner-scoped memory. On ProfyBench-v1, an 85-task offline benchmark built from replay seeds and controlled internal tests, the Profy Full configuration as a whole achieves a mean overall score of 4.65/5 from the primary LLM judge, compared with 4.01 for Raw LLM and 4.28 for Expert Prompt, with the largest gains in completeness, specificity, and missing-information tasks. These configuration-level scores do not test cold-expert mitigation, cross-expert transfer, or longitudinal distillation.
Figures & tables
Figure 1: Profy marketplace workflow. Solid panels show implemented skill proposals, scoped memory, and creator-authorized publication; dashed purple paths show proposed routing to Creator Profiles and Expert Journals. The routing is neither implemented nor evaluated. Skill activation and rollback have separate eligibility checks (Section 3.4 ; Appendix F ).
Challenge ( status )
CogniLife design response
Tacit expertise ( partial )
Elicit commitments, then derive revisable guidance from attributed evidence.
Cold experts ( partial )
Seed sparse listings; propose reuse of relevant creator evidence.
Evidence attribution ( proposed )
Resolve source, applicability, and permitted reuse before routing claims.
Persistent refinement ( partial )
Consolidate scoped memory; evaluate and version skill updates.
Creator control ( implemented )
Keep authored commitments separate from learned candidates and releases.
Table 1: Deployment challenges motivating CogniLife and implementation status (partial: some steps implemented, others proposed or incomplete). ProfyBench-v1 tests the Full configuration, not individual components.
Family
Domain
N
Cmpl.
Miss.
Adv.
ADV
Evidence advising
20
10
6
4
EDU
Education
20
10
6
4
MKT
Marketing
20
10
6
4
MMW
Multimodal wf.
15
8
4
3
PER
Personalization
10
5
2
3
Table 2: Composition of ProfyBench-v1 : 85 instances across five production families and three variants (Complete / Missing-information / Adversarial). Evaluation focus by family—ADV: evidence grounding, uncertainty calibration, safe boundaries; EDU: pedagogical structure and artifact completeness; MKT: platform fit and audience-specific rewriting; MMW: workflow decomposition and text–image consistency; PER: style fidelity and long-turn coherence.
System
Ovr.
Comp.
Spec.
Act.
Coh.
Safe
Raw LLM
4.01
3.61
3.38
3.95
4.65
4.48
Expert Prompt
4.28
4.01
3.62
4.15
4.84
4.78
Profy Full
4.65
4.62
4.26
4.47
4.99
4.92
Table 3: ProfyBench-v1 automatic judge results ( N=85 , 1–5 scale, higher is better). Overall (Ovr.) is the average of completeness, specificity, actionability, coherence, and safety-boundary adherence.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Robustness and error analysis on ProfyBench-v1 . The source tally also has two entries labeled evidence_grounding (Raw 1; Prompt 1), a label absent from the fixed error-tag list in Listing 1 ; these entries are excluded here pending raw-output verification. The remaining counts show no missing-input failures and one weak-personalization failure for Full.
Base
Metric
Gain
95% CI
Win
Raw
Overall
0.64
[0.48, 0.82]
76%
Raw
Completeness
1.01
[0.78, 1.26]
62%
Raw
Specificity
0.88
[0.65, 1.13]
56%
Prompt
Overall
0.37
[0.28, 0.48]
64%
Prompt
Completeness
0.61
[0.44, 0.79]
47%
Prompt
Specificity
0.64
[0.44, 0.85]
47%
Appendix
Table 4: Paired Profy gains over two baselines on the same 85 tasks. Win denotes the fraction of task-level paired comparisons where Profy scores strictly higher on the specified metric.
Layer
Owner / writer
Purpose
Soul
Creator-authorized authoring, including AI assistance
Behavioral constraints; background evolution does not edit this field.
Published configuration
Creator publication
Stable serving snapshot of expert configuration.
Caller profile
Invoking user
Personal information and communication preferences used for that caller.
Engagement memory
Memory tools and background review
User–expert scope; temporal memory also isolates external end users.
Expert skills
Scoped candidate acceptance and evaluated promotion
Personal candidates, shared skill versions, and rollback history.
Conversation context
Runtime only
Temporary history and task state for the current invocation.
Appendix
Table 5: Ownership and write boundaries in CogniLife . The key invariant is that automated evolution never writes to the Soul.
A/B test
Treatment
Primary metric
Input robustness
Structured parser for pasted task content
First-turn valid input rate
Replay reuse
Prefilled “run similar case” template
Successful replay-to-run conversion
Expert onboarding
Example-rich opening prompt
Continuation rate
Creator wizard
Guided expert creation checklist
Publish rate and revision count
Appendix
Table 6: Proposed, not yet conducted, low-risk A/B tests that connect benchmark observations to product iteration.
Figure 3: Anonymized production screenshots illustrating three CogniLife marketplace task geometries. The examples illustrate deployed usage related to ProfyBench-v1 task families: advising with source/tool traces, multimodal artifact production through multi-step workflow execution, and marketing copy optimization with concrete revision rationales. Account-side identifiers and exact user-specific admissions details have been removed or masked.
Figure 4 (continued), panel (b).
Figure 5 (continued), panel (c).
Figure 6: A deployed CogniLife expert (“Video Novel Expert”) executing a multimodal, multi-step workflow that turns one user goal—simulate Chinese college life and a major-choice decision—into an interactive HTML artifact ( college_life_simulator.html ). The expert (a) opens with a generated anime background and an in-character narrator, (b) presents a branching decision point where the chosen family-resource background (working-class / middle-class / well-connected) conditions later advice, (c) delivers major-choice advice grounded in school brand, city network, and fallback risk, and (d) closes with an ending screen that summarizes the run as stat bars (Versatility, Financial Status, and a tongue-in-cheek “Hair Remaining”) plus a structured Preference Review.
Figure 7 (continued). This is a concrete instance of the multimodal-workflow (MMW) family in ProfyBench-v1 : artifact-oriented execution, staged image generation (left pane: live action: generate log, status Done ), text–image consistency, and advice-like content delivered inside an interactive simulation rather than a single chat turn. The narrative is a simulation for illustration; it is not evidence of the quality or fairness of real admissions advice, and its family-background branch has not been tested for stereotyped recommendations. User-specific inputs are masked; the simulator content is procedurally assembled from the user’s in-session choices.
Figure 8: Opening message of a deployed CogniLife expert (Yu Jun PM Mentor). The message introduces a product-management methodology and asks three diagnostic questions before advising the user. This is an example of visible service-opening text; Figure 9 separately illustrates an authoring workflow. No record-level or release-version link between the two examples is established. Journal-based maintenance remains proposed.
Figure 9: Creator-directed initial grounding in the Expert Distillation Studio. (a) Product-decision scenarios invite the creator to compare alternative reasoning styles. (b) Reported authoring stages include a research review skipped at the creator’s request, calibration awaiting confirmation, and pending verification. The later “Distillation Complete” status indicates that content was written; it does not indicate creator confirmation, quality verification, or publication. (c) Seven content fields are stored for expert record distilled-9c3a9274 , including separate Soul (2,287 chars), persona (4,879 chars), and agent-instructions (2,584 chars) fields. (d) The reusable skill yujun-product-value-exchange , scoped to the same record, encodes the User Value formula, principles, and decision framework. These panels document inspectable authoring outputs and opportunities for review. Creator-authorized AI authoring can write Soul; background memory and skill maintenance follow separate update paths.
Figure 10 (continued).
Figure 11: Task formulation and scoring scope of the inspected customs-extraction example. Source documents are converted into canonical JSON. Canonical-structure, schema, and item-count checks gate comparison over 389 reviewed objective fields; 545 registered fields are excluded. Counts describe one task. A high subset score does not establish business completion.
Table 18
Figure 12: Recorded scores for 52 customs-extraction tasks, sorted by the mean of three runs. Dark dots show task means, light dots individual runs, and vertical bars observed minimum–maximum ranges, not confidence intervals. The dashed line marks the macro-average of 0.9576. The aggregate records and the single-task scoring audit have distinct evidential scopes.
Do expert personas improve language model performance? The Wharton Generative AI Lab reports that they do not, broadcasting to millions via social media the recommendation that practitioners abandon a technique recommended by Anthropic, Google, and OpenAI. We demonstrate that this null finding was structurally predictable. Five core mechanisms precluded detection before data collection began: baseline contamination elevating the starting point to near-ceiling, system prompt hierarchy subordinating experimental manipulation, impossible expert specifications collapsing to generic competence, format constraints suppressing reasoning processes, and provider exclusion limiting generalizability. Controlled trials correcting these limitations reveal what the original design obscured. To test this, we selected the GPQA Diamond hardest questions to prevent baseline pattern matching, forcing reliance on genuine expert reasoning. On items with valid key answers, expert personas achieve ceiling accuracy. They eliminated all baseline errors through confidence amplification. Furthermore, forensic examination of model divergence identified that half of the hardest GPQA items contain chemically or logically indefensible answers. The model's CoT revealed reasoning away from impossible answers, yielding penalization for accurate chemistry. These findings recontextualize the original null results. Methodologically sound persona research faces measurement constraints imposed by benchmark validity limitations. Answering the persona question requires evaluation infrastructure the field does not yet possess.
Drake Mullens, Stella Shen
Tarleton State University · University of Texas at Tyler
As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce $OneMillion-Bench ($OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, $OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.
Mixture-of-Experts (MoE) architectures scale language models efficiently by activating only a few experts per token, but the saving is confined to computation: every expert must stay resident on the accelerator, so memory bounds where these models can be deployed. Expert pruning reduces this footprint, yet existing methods are static --- a single mask, calibrated offline, is applied to the model for every subsequent request. This assumption can fail when the workload is heterogeneous, most prominently in multi-agent systems, where one backbone serves many tasks and roles at once: our analysis shows that different tasks and roles recruit different experts, while static methods assign one fixed subset to all of them. We therefore propose Dynamic Expert Pruning (DEP), which rests on a finding we establish here: an agent's system and task prompts are by themselves sufficient to identify the experts that agent and its task require, since that text already describes what the agent will do. A lightweight predictor, trained once on workflow transcripts, turns those prompts into a specialized per-request mask in a single forward pass, with no per-configuration calibration. Across diverse tasks and roles, model scales, and MoE architectures, DEP achieves better overall accuracy than static pruning and merging baselines, and generalizes to workflows unseen in training without retraining. Its margin over those baselines is largest when few experts are retained, suggesting that the role specialization inherent to multi-agent systems permits sparser serving than static pruning allows.
Jabin Koo, Soheil Abbasloo, Sungjae Lee +1
Pohang University of Science and Technology (POSTECH) · Microsoft Research Asia