Multimodal agentic pipelines are transforming human-computer interaction by enabling efficient and accessible automation of complex, real-world tasks. However, recent efforts have focused on short-horizon or general-purpose applications (e.g., mobile or desktop interfaces), leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored. To address this, we introduce CareFlow, a high-quality human-annotated benchmark comprising complex, long-horizon software workflows across medical annotation tools, DICOM viewers, EHR systems, and laboratory information systems. On this benchmark, existing vision-language models (VLMs) perform poorly, struggling with long-horizon reasoning and multi-step interactions in medical contexts. To overcome this, we propose CarePilot, a multi-agent framework based on the actor-critic paradigm. The Actor integrates tool grounding with dual-memory mechanisms (long-term and short-term experience) to predict the next semantic action from the visual interface and system state. The Critic evaluates each action, updates memory based on observed effects, and either executes or provides corrective feedback to refine the workflow. Through iterative agentic simulation, the Actor learns to perform more robust and reasoning-aware predictions during inference. Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38%, respectively, on our benchmark and out-of-distribution dataset.
Figures & tables
Figure 1: CareFlow is a large-scale benchmark for evaluating multimodal agents on a range of real healthcare software. It enables execution-based evaluation and interactive reasoning across DICOM viewers, image annotation tools, EMR/EHR, and LIS platforms. Each task pairs a natural-language goal with GUI screenshots representing authentic clinical workflows.
Figure 2: Overview of the CarePilot framework. An Actor–Critic multi-agent architecture governs hierarchical decision-making for long-horizon healthcare workflows. At each step, the Actor observes the current interface and instruction, integrates tool-grounding signals, and its past experience that is stored in short- and long-term memories, and predicts the next semantic action. The Critic evaluates outcomes, provides corrective feedback, and updates both short-term and long-term memory buffers to guide subsequent decisions.
Figure 3: Distribution of task lengths (number of steps) in the test split of CareFlow .
Figure 4: Category distribution of tasks in CareFlow across four major healthcare software domains.
Function
Description
CLICK
Move the cursor (if needed) and click at the specified item in the screen
SCROLL
Scroll the active view vertically or horizontally by n units.
ZOOM
Adjust the magnification level of the displayed image or view.
TEXT
Type string s into the focused input field (e.g., patient ID, study date, filter query).
SEGMENT
Create or edit a segmentation or region of interest (ROI) on the displayed medical image.
COMPLETE
Mark the workflow or task as finished.
Table 1: Core mouse and keyboard actions in CarePilot . Arguments in parentheses denote examples; (x,y) are pixel coordinates, n is a signed step count, and s is a text string.
Split
# Tasks
Avg. Steps
Min
Max
OOD
Actions
Train
735
12.7
7
22
–
6
Test
315
12.9
9
24
50
6
Total
1050
–
–
–
50
6
Table 2: Dataset statistics of CareFlow .
Model
Weasis
3D Slicer
Orthanc
OpenEMR
Average
SWA
TA
SWA
TA
SWA
TA
SWA
TA
SWA
TA
Qwen 2.5 VL 7B
58.64
1.33
61.4
1.66
65.5
2.5
63.20
1.66
57.18
1.78
Qwen 3 VL 8B
65.20
1.33
68.38
3.33
67.71
5.0
66.31
5.0
66.90
3.66
Qwen2.5 VL 32B
79.95
5.13
68.00
1.90
48.62
2.42
41.60
0.32
60.72
2.43
Llama 3.2 11B
86.03
10.26
76.47
4.60
69.58
13.58
62.56
11.47
75.65
9.50
Llama 4 Scout
84.34
10.26
78.47
2.70
88.56
23.90
85.55
21.79
85.65
13.50
Table 3: Results on CareFlow . Columns group Step-Wise Accuracy (SWA) and Task Accuracy (TA) for each software; the last group reports the overall Average . Best results are bold , best among baselines are underlined . Green-highlighted rows denote our proposed method.
Figure 5: Qualitative visualization of Llama-4 Maverick-17B performing CarePilot’s radiology workflow tasks. The traces highlight typical action–mode confusions such as issuing ZOOM instead of CLICK for tool selection and CLICK in place of SEGMENT or TEXT operations. These incomplete branches illustrate the model’s inconsistent tool arming and gesture execution.
Model
SWA
TA
Qwen2.5 VL 32B
71.74
12.72
Llama 3.2 11B
70.76
16.36
Llama 4 Scout
72.20
20.75
Llama 4 Maverick
73.71
27.27
Qwen3 VL 235B
75.18
25.46
Mistral 3.2 VL 24B
69.63
1.82
Table 4: Results on OOD OpenHospital . Best is bold , best among baselines are underlined . Green rows denote our method ( CarePilot ), gray rows indicate closed-source models, and white rows represent open-source baselines.
Model
SWA
TA
CarePilot(WC)
65.37
3.75
CarePilot(WC + TG)
72.98
12.5
CarePilot (Qwen-VL 2.5-7B)
88.05
48.90
Table 5: Results to show the impact of Critic agent in CarePilot . WC represents without critic. (WC +TG) represents without critic agent but using tool grounding.
Figure 6: Variation of performance of CareFlow across different steps range.
Contextual Information
Overall
TG
LTM
STM
SWA
TA
✗
✓
✓
73.20
9.37
✓
✗
✓
82.10
23.67
✓
✓
✗
80.40
30.42
✓
✓
✓
88.05
48.90
Table 6: Ablation on contextual components using CarePilot (Qwen 2.5 VL 7B) : Tool Grounding (TG), Long-Term Memory (LTM), and Short-Term Memory (STM). Reported are overall Step-Wise Accuracy (SWA) and Task Accuracy (TA). Best results are bold , and second-best are underlined .
Contextual Information
Overall
TG
LTM
STM
SWA
TA
✗
✓
✓
75.38
11.25
✓
✗
✓
84.22
26.25
✓
✓
✗
82.53
32.50
✓
✓
✓
90.18
51.45
Table 7: Ablation on contextual components using CarePilot (Qwen 3 VL 8B) : Tool Grounding (TG), Long-Term Memory (LTM), and Short-Term Memory (STM). Reported are overall Step-Wise Accuracy (SWA) and Task Accuracy (TA). Best results are bold , and second-best are underlined .
Tool Components
Overall
OD
ZOOM
OCR
TM
SWA
TA
✗
✓
✓
✓
81.16
38.59
✓
✗
✓
✓
85.67
46.31
✓
✓
✗
✓
76.65
30.87
✓
✓
✓
✗
72.14
25.73
✓
✓
✓
✓
88.05
48.90
Table 8: Ablation on tool components using CarePilot (Qwen 2.5 VL 7B) : Object Detection (OD), Zoom (ZOOM), Optical Character Recognition (OCR), and Tool Memory (TM). Reported are overall Step-Wise Accuracy (SWA) and Task Accuracy (TA). Best results are bold , and second-best are underlined .
Figure 7: Qualitative visualization of GPT-5 predictions for the same CarePilot tasks. While GPT-5 improves on low-level action consistency, it still fails under domain-shifted UI states—issuing repeated SCROLL commands or premature free-text annotations before activating the correct tool. These errors contrast with CarePilot’s complete, state-aware execution across both CT abdomen and chest workflows.
Method
Avg. Time / Task
TWA
Qwen 2.5 VL (Zero Shot)
∼ 20 s
8.5
Actor-Critic, tools
∼ 150 s
42.5
CarePilot (distilled)
∼ 90 s
48.9
Table 9: Empirical average inference time per long-horizon task on CareFlow(315 tasks).
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · University of Chinese Academy of Sciences, Beijing, China