As real-world datasets become more complex and heterogeneous, supervised learning is often bottlenecked by input representation design. Modeling multimodal data, such as time-series, free text, and structured records, often requires non-trivial domain expertise. We propose an agentic pipeline to streamline this process. First, an LLM analyzes a small but diverse subset of text-serialized input examples in-context to synthesize a global rubric, which acts as a programmatic specification for extracting and organizing evidence. This rubric is then used to transform naive text-serializations of inputs into a more standardized format for downstream models. We also describe local rubrics, which are task-conditioned interpretive summaries generated by an LLM. Across 15 clinical tasks from the EHRSHOT benchmark, our rubric approaches significantly outperform count-feature models, naive LLM baselines, and a clinical foundation model pretrained on orders of magnitude more data. Beyond performance, rubrics offer operational advantages such as being easy to audit, cost-effectiveness at scale, and facilitating tabular representations.
Figures & tables
Figure 1: Performance (macro-averaged) over 15 clinical prediction tasks in the EHRSHOT benchmark [ 61 ] , swept over training-set size n (per task). Our rubric-style representations constructed by LLMs outperform the naive text-serialization baseline in Hegselmann et al. [15] , as well as a clinical foundation model pretrained on 2.57M patient timelines (CLMBR-T, [ 61 ] ), and a count feature-based gradient boosting machine (Count-GBM, [ 21 , 61 ] ).
Figure 2: Synthetic electronic health record representation examples. Left. Naive text-serialization adopted from Hegselmann et al. [15] . Middle. Local rubric representation (task-conditioned summary of the naive text-serialization). Right. Global rubric transformed version of the naive-text serialization.
Figure 3: Agentic global-rubric pipeline for EHRSHOT tasks. Full prompts are in Appendix F .
Figure 4: Prompts for generating local rubric representations. Left. Task-conditioned interpretive summaries (Local-Rubric). Middle. Edited summaries with interpretations stripped away (Local-Rubric-NoInterp, full prompt in Appendix F.6 ). Right. Fact-listing summaries (Local-Rubric-Basic).
Figure 5: Average performance over all tasks. For confidence bands, see Appendix E.3 .
Figure 6: Three rubric outputs for the same synthetic hyponatremia case. Local-Rubric renders the evidence as a task-aligned risk assessment, Local-Rubric-NoInterp keeps the same evidence with interpretive language stripped, and Global-Rubric extracts it into standardized fields.
Figure 7: Average performance by task group. For confidence bands, see Appendix E.3 .
Figure 8: Key excerpts from learned hypertension global rubric instructions.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Task
Train
Val
Test
Operational Outcomes (3)
ICU transfer
2402 (113)
100 (50)
2037 (85)
Length of stay > 7 days
2569 (681)
100 (50)
2195 (552)
30-day readmission
2608 (370)
100 (50)
2189 (260)
Assignment of New Diagnosis (6)
Hypertension
1259 (182)
100 (50)
1258 (159)
Hyperlipidemia
1684 (205)
100 (50)
1317 (172)
Pancreatic cancer
2576 (155)
100 (50)
2220 (56)
Appendix
Table 1: Number of samples (positive cases) per task and split. All splits for Lab results and Chest X-ray tasks are subsampled from the original dataset. Validation splits are subsampled for all tasks.
Figure 9: Average overall performance. Bands denote 95% CI, bootstrapped over the test set.
Figure 10: Average performance per task group. Bands denote 95% CI, bootstrapped over the test set.
Task
AUROC
AUPRC
Operational Outcomes
ICU transfer
.800.752−.846
.195.130−.277
Long length of stay
.737.714−.758
.464.421−.509
30-day readmit
.757.725−.787
.357.300−.419
Group Avg.
.765.745−.786
.340.308−.374
Assignment of New Diagnoses
Appendix
Table 4: Full-dataset EHRSHOT results for Global-Rubric-Tabular .
Representation
Mean
Median
Min
Max
Std
NaiveText
5,980
7,769
131
8,267
2,648
Local-Rubric
853
844
492
1,430
116
Global-Rubric
1,283
1,193
318
3,136
452
Appendix
Table 5: Token count statistics for the textual representations that are passed into the text embedding model prior to downstream learning across different methods. Statistics for each representation are computed after merging data from all 15 tasks and 3 splits in Table 1 . Qwen3-8B tokenizer is used.
Figure 11: AUROC per-task, n=All
Figure 12: AUPRC per-task, n=All
Figure 13: AUROC per-task, n=10
Figure 14: AUPRC per-task, n=10
AUROC
AUPRC
Qwen3-8B-Zeroshot
.707 .650-.764
.090 .064-.120
GPT5-Mini-Zeroshot
.688 .631-.744
.101 .068-.139
n=10
n=All
n=10
n=All
Count-GBM
.564 .508-.622
.726 .665-.779
.066 .040-.103
.112 .076-.157
CLMBR-T
.678 .617-.736
.845 .797-.892
.115 .069-.174
.314 .226-.416
NaiveText
.759 .711-.807
.801 .751-.843
.149 .097-.214
.177 .119-.243
Appendix
Table 6: Guo ICU. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.650 .629-.672
.333 .308-.359
GPT5-Mini-Zeroshot
.742 .724-.763
.416 .387-.449
n=10
n=All
n=10
n=All
Count-GBM
.628 .604-.654
.709 .685-.733
.325 .295-.355
.390 .355-.426
CLMBR-T
.611 .587-.637
.818 .799-.837
.332 .298-.368
.589 .546-.632
NaiveText
.616 .588-.641
.743 .723-.765
.313 .286-.344
.455 .413-.502
Appendix
Table 7: Guo length of stay. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.561 .534-.586
.133 .116-.150
GPT5-Mini-Zeroshot
.609 .588-.627
.147 .130-.166
n=10
n=All
n=10
n=All
Count-GBM
.712 .676-.748
.785 .754-.816
.249 .208-.295
.321 .272-.373
CLMBR-T
.771 .738-.800
.791 .760-.819
.347 .289-.405
.373 .313-.434
NaiveText
.678 .645-.711
.780 .751-.808
.212 .176-.253
.312 .259-.368
Appendix
Table 8: Guo readmission. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.664 .629-.698
.100 .082-.119
GPT5-Mini-Zeroshot
.739 .697-.782
.172 .135-.214
n=10
n=All
n=10
n=All
Count-GBM
.572 .520-.622
.704 .654-.748
.086 .068-.106
.169 .125-.219
CLMBR-T
.577 .528-.621
.737 .697-.775
.099 .073-.131
.191 .142-.247
NaiveText
.520 .470-.568
.746 .706-.785
.081 .062-.105
.179 .134-.229
Appendix
Table 9: Acute MI. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.497 .495-.498
.009 .005-.014
GPT5-Mini-Zeroshot
.497 .496-.499
.010 .005-.014
n=10
n=All
n=10
n=All
Count-GBM
.603 .457-.745
.671 .510-.828
.038 .012-.080
.077 .024-.160
CLMBR-T
.532 .409-.642
.543 .415-.671
.013 .007-.025
.017 .007-.036
NaiveText
.712 .599-.810
.614 .482-.736
.026 .012-.044
.030 .009-.087
Appendix
Table 10: Celiac disease. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.500 .452-.546
.132 .109-.157
GPT5-Mini-Zeroshot
.640 .601-.678
.176 .148-.205
n=10
n=All
n=10
n=All
Count-GBM
.546 .496-.597
.702 .662-.745
.172 .138-.211
.287 .225-.349
CLMBR-T
.512 .466-.560
.689 .647-.733
.153 .122-.189
.251 .202-.307
NaiveText
.559 .510-.613
.722 .684-.762
.180 .142-.223
.263 .213-.320
Appendix
Table 11: Hyperlipidemia. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.552 .502-.600
.142 .116-.168
GPT5-Mini-Zeroshot
.646 .609-.684
.172 .145-.205
n=10
n=All
n=10
n=All
Count-GBM
.662 .622-.704
.693 .653-.732
.206 .164-.251
.258 .203-.317
CLMBR-T
.653 .611-.696
.721 .682-.759
.201 .160-.251
.263 .213-.324
NaiveText
.566 .525-.609
.620 .574-.669
.150 .121-.182
.234 .183-.291
Appendix
Table 12: Hypertension. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.500 .499-.500
.009 .005-.013
GPT5-Mini-Zeroshot
.550 .499-.625
.084 .007-.235
n=10
n=All
n=10
n=All
Count-GBM
.754 .637-.849
.826 .734-.901
.051 .012-.144
.109 .026-.236
CLMBR-T
.614 .486-.747
.681 .588-.768
.016 .008-.028
.020 .009-.037
NaiveText
.670 .575-.767
.695 .556-.822
.025 .009-.063
.047 .013-.112
Appendix
Table 13: Lupus. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.630 .574-.694
.086 .043-.155
GPT5-Mini-Zeroshot
.604 .554-.662
.205 .099-.314
n=10
n=All
n=10
n=All
Count-GBM
.574 .503-.644
.933 .902-.959
.034 .023-.049
.406 .258-.546
CLMBR-T
.669 .589-.740
.812 .741-.876
.114 .046-.205
.276 .165-.407
NaiveText
.697 .630-.766
.860 .803-.908
.120 .051-.201
.321 .202-.449
Appendix
Table 14: Pancreatic cancer. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.500 .478-.523
.495 .468-.520
GPT5-Mini-Zeroshot
.530 .510-.549
.514 .490-.538
n=10
n=All
n=10
n=All
Count-GBM
.563 .538-.587
.562 .537-.587
.540 .511-.571
.533 .502-.563
CLMBR-T
.646 .624-.669
.821 .803-.837
.634 .602-.668
.788 .759-.816
NaiveText
.581 .554-.605
.649 .625-.673
.573 .541-.607
.634 .602-.667
Appendix
Table 15: Anemia. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.758 .738-.779
.735 .705-.763
GPT5-Mini-Zeroshot
.786 .766-.805
.744 .717-.771
n=10
n=All
n=10
n=All
Count-GBM
.568 .543-.593
.665 .640-.687
.561 .530-.595
.644 .612-.677
CLMBR-T
.562 .536-.588
.752 .730-.774
.568 .535-.601
.763 .735-.791
NaiveText
.592 .566-.617
.748 .726-.769
.581 .548-.616
.750 .719-.779
Appendix
Table 16: Hyperkalemia. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.687 .665-.710
.662 .629-.693
GPT5-Mini-Zeroshot
.662 .636-.687
.650 .617-.683
n=10
n=All
n=10
n=All
Count-GBM
.571 .543-.598
.591 .563-.621
.561 .526-.597
.596 .557-.634
CLMBR-T
.611 .584-.639
.777 .755-.799
.595 .560-.631
.764 .730-.797
NaiveText
.587 .559-.616
.680 .654-.706
.584 .546-.621
.675 .638-.709
Appendix
Table 17: Hypoglycemia. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.686 .666-.707
.655 .626-.683
GPT5-Mini-Zeroshot
.706 .685-.727
.657 .629-.687
n=10
n=All
n=10
n=All
Count-GBM
.518 .491-.545
.539 .514-.563
.529 .497-.561
.526 .495-.558
CLMBR-T
.548 .522-.572
.658 .633-.681
.557 .523-.589
.641 .608-.674
NaiveText
.514 .489-.538
.595 .571-.619
.522 .490-.554
.595 .563-.626
Appendix
Table 18: Hyponatremia. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.705 .683-.725
.637 .608-.666
GPT5-Mini-Zeroshot
.739 .719-.759
.670 .645-.696
n=10
n=All
n=10
n=All
Count-GBM
.498 .473-.525
.556 .531-.580
.507 .477-.540
.551 .519-.582
CLMBR-T
.522 .496-.548
.627 .604-.654
.523 .492-.554
.606 .575-.640
NaiveText
.518 .491-.545
.612 .586-.635
.524 .491-.555
.588 .556-.622
Appendix
Table 19: Thrombocytopenia. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
AUROC
AUPRC
Qwen3-8B-Zeroshot
.546 .519-.573
.526 .491-.560
GPT5-Mini-Zeroshot
.520 .498-.541
.507 .476-.538
n=10
n=All
n=10
n=All
Count-GBM
.571 .541-.602
.609 .576-.642
.543 .502-.583
.582 .540-.624
CLMBR-T
.453 .421-.484
.609 .578-.640
.465 .427-.500
.600 .559-.643
NaiveText
.558 .533-.584
.616 .592-.640
.551 .519-.583
.609 .577-.641
Appendix
Table 20: Chest X-ray. AUROC and AUPRC with 95% bootstrap CI. Best per column for each setting is highlighted. Embeddings are from Qwen3-8B; sample sizes refer to per-class training examples.
Figure 15: Prompt for converting textual inputs to embeddings. An example task query: “Will the patient develop lupus within next year?”
Figure 16: Prompt used with GPT-5-mini to guide global rubric creation from NaiveText serializations ( xtext ) of EHR examples.
Figure 17: Prompt used with GPT-5-mini for transforming a naive text serialized input ( xtext ) into its rubric text serialization version ( xrubric ).
Figure 18: Prompt used with GPT-5.2 to create a parser script for transforming a naive text serialized input ( xtext ) into its rubric text serialization version ( xrubric ).
Figure 20: Prompt used with GPT-5-mini for stripping away interpretive language from Local-Rubric representation ( i.e., modifying xrubric ).
Large language models (LLMs) offer substantial promise for automating clinical text summarization, yet maintaining factual consistency remains challenging due to the length, noise, and heterogeneity of clinical documentation. We present AgenticSum, an inference-time, agentic framework that separates context selection, generation, verification, and targeted correction to reduce hallucinated content. The framework decomposes summarization into coordinated stages that compress task-relevant context, generate an initial draft, identify weakly supported spans using internal attention grounding signals, and selectively revise flagged content under supervisory control. We evaluate AgenticSum on two public datasets, using reference-based metrics, LLM-as-a-judge assessment, and human evaluation. Across various measures, AgenticSum demonstrates consistent improvements compared to vanilla LLMs and other strong baselines. Our results indicate that structured, agentic design with targeted correction offers an effective inference time solution to improve clinical note summarization using LLMs.
Large language models (LLMs) and agentic systems have shown promise for clinical decision support, but existing works largely assume that evidence has already been curated and handed to the model. Real-world clinical workflows instead require agents to actively seek, iteratively plan, and synthesize multimodal evidence from heterogeneous sources. In this paper, we introduce ClinSeekAgent, an automated agentic framework for dynamic multimodal evidence seeking that shifts the paradigm from passive evidence consumption to active evidence acquisition. Given only a clinical query and access to raw data sources, ClinSeekAgent gathers evidence by querying medical knowledge bases, navigating raw EHRs, and invoking medical imaging tools; refines its hypotheses as new information emerges; and integrates the collected evidence into grounded clinical decisions. ClinSeekAgent serves both as an inference-time agent for frontier LLMs and as a training-time pipeline for distilling high-quality agent trajectories into compact open-source models. To validate its inference-time effectiveness, we construct ClinSeek-Bench, which pairs Curated Input reasoning from fixed pre-selected evidence with Automated Evidence-Seeking over raw clinical data. On text-only EHR tasks, ClinSeekAgent improves Claude Opus 4.6 from 60.0 to 63.2 overall F1 and MiniMax M2.5 from 43.1 to 47.3, with positive risk-prediction gains in 7 out of 9 evaluated host models. On multimodal tasks, ClinSeekAgent improves Claude Opus 4.6 from 47.5 to 62.6 (+15.1); all evaluated models improve across the three CXR-related task groups. We further validate ClinSeekAgent as a training pipeline by distilling agentic evidence-seeking trajectories into ClinSeek-35B-A3B, which achieves 34.0 average F1 on existing AgentEHR-Bench, improving over its Qwen3.5-35B-A3B baseline by +11.9 points and approaching Claude Opus 4.6.
While interpretable prototype networks offer compelling case-based reasoning for clinical diagnostics, their raw continuous outputs lack the semantic structure required for medical documentation. Bridging this gap via standard Retrieval-Augmented Generation (RAG) routinely triggers ``retrieval sycophancy,'' where Large Language Models (LLMs) hallucinate post-hoc rationalizations to align with visual predictions. We introduce ProtoMedAgent, a framework that formalizes multimodal clinical reporting as an iterative, zero-gradient test-time optimization problem over a strict neuro-symbolic bottleneck. Operating on a frozen prototype backbone, we distill latent visual and tabular features into a discrete semantic memory. Online generation is strictly constrained by exact set-theoretic differentials and a reflective Scribe-Critic loop, mathematically precluding unsupported narrative claims. To safely bound data disclosure, we introduce a semantic privacy gate governed by k-anonymity and ℓ-diversity. Evaluated on a 4,160-patient clinical cohort, ProtoMedAgent achieves 91.2% Comparison Set Faithfulness where it fundamentally outperforms standard RAG (46.2%). ProtoMedAgent additionally leverages a binding ℓ-diversity phase transition to systematically reduce artifact-level membership inference risks by an absolute 9.8%.