cs.LGOct 22, 2026
SavennInteractive Agent: General-Purpose AI Agents as 3D Image Annotators
Organizations: Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), Heidelberg, Germany · National Center for Tumor Diseases (NCT) Heidelberg, a partnership between the German Cancer Research Center (DKFZ) and Heidelberg University Hospital (UKHD), Heidelberg, Germany · Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany · Faculty of Mathematics and Computer Science, Heidelberg University, Germany ·
Abstract
Interactive 3D segmentation models such as nnInteractive turn a few spatial prompts (points, scribbles, boxes) into accurate volumetric masks, but a human still does the annotation work: finding each structure, placing prompts, checking the result and correcting it. General-purpose AI agents can now interpret images, call tools and write code. We show that such agents can take over this work without additional training: the user makes a request in plain language, and the agent works out what has to be segmented, locates the targets, prompts nnInteractive, inspects the 3D result, corrects it and responds to feedback. Asked for 'the muscles used for chewing', an agent identifies and segments the masseters and pterygoids; asked to count liver lesions, it writes its own code to find candidates and splits a merged lesion on request. Our open-source nninteractive-agent provides nnInteractive as tools over the Model Context Protocol (MCP), together with a skill that encodes how experienced users prompt it. On a benchmark spanning CT, MRI, PET and micro-CT of humans, a mouse and an ant (14 images; 51 targets from organs and tumours to muscles, foot bones, vessel trees and an insect brain), Claude Code and Codex reach a mean Dice of 0.81 and 0.84 from the request alone; when told where each target is, both reach 0.85 and prompt nearly as effectively as an oracle that places prompts using the reference segmentation. A text-prompted 3D model (VoxTell) is more accurate on most standard radiological targets but less robust: it returns an empty mask for one of them and fails on the leg muscles, the foot bones, the mouse and the ant, where the agents remain accurate. Code: https://github.com/MIC-DKFZ/nninteractive-agent
Figures & tables
Figure 1: From a request to 3D masks. Four requests to Claude Code (Opus 5.5 medium effort) operating nnInteractive in the web demo: the request, the result in 3D and in one slice (zoomed in for the suitcase), and what the agent did. The second request followed the first in the same conversation. Images: TotalSegmentator MRI [ 1 ] , RibFrac [ 2 , 3 ] , a CT scan of a packed suitcase (image provided by Smiths Detection).
Figure 2: Overview. The agent reads the skill and works through the tools of the MCP server. The server holds the image, the objects segmented so far and an nnInteractive session. Each prompt goes through the server to nnInteractive; the agent checks the resulting 3D mask in the returned text and in rendered views, and repeats this until it accepts the object. It reports to the user once per request. Example: “Segment and count all liver lesions” on MSD Liver, case 51 [ 30 , 31 ] . nnInteractive runs locally or on a shared server.
Figure 3: What the agent sees , here Codex segmenting the brainstem in a head-and-neck MRI ( headneck_mr , HaN-Seg [ 32 ] ; unguided track). (a) The preview of its first scribble, in a sagittal slice. (b) The 3D rendering after this prompt: the object (169 ml) extends far to both sides (Dice 0.28), and (c) the montage shows that it covers the cerebellum. (d) The second prompt, a negative scribble through the cerebellum in an axial slice, removed most of the leak (Dice 0.80). (e) The third prompt, a short negative scribble at the lower end, shown in a sagittal slice with the mask after the second prompt. (f) The 3D rendering before the commit (Dice 0.82). The agent committed the brainstem with the note “Brainstem separated from cerebellum and cervical cord; superior midbrain boundary and cervicomedullary transition are approximate on 3 mm slices.” Panels (a)–(c) and (f) are renders the agent looked at. It sent the two corrections without a preview, so (d) and (e) were drawn afterwards with the same renderer from the logged prompts and masks.
Figure 5: Liver lesions from a one-line request . (a) A figure the agent drew for itself with its Python tool to review lesion candidates. (b) The committed lesions in one slice. (c) Lesion 6 as first segmented and (d) after the user asked to split it.
Figure 6: The benchmark images , one slice each. Upper rows: the reference; lower rows: what Claude Code segmented from the request alone (unguided track, the middle of three repeats), with that run’s Dice averaged over the image’s targets. Fat-weighted MRI (calf, thigh) is shown on a log scale; for the whole-body CT ( foot_ct ), a sagittal slice through the left foot in a bone window. All tracks: Figs. S3 and S4 in the supplement.
| Track | Dice | Prompts | Scribble oracle, | Minutes | |||
|---|---|---|---|---|---|---|---|
| all (14) | radiology (9) | rad. without | legs, mouse, | per target | same prompts | per image | |
| adrenal_ct (8) | ant (5) | ||||||
| Codex, guided | 0.855 0.002 | 0.823 | 0.812 | 0.912 | 2.9 | 0.869 | 3.8 |
| Claude Code, guided | 0.847 0.006 | 0.828 | 0.818 | 0.880 | 2.2 | 0.844 | 3.6 |
| Codex, unguided | 0.842 0.004 | 0.815 | 0.804 | 0.892 | 2.8 | 0.886 | 4.2 |
| Claude Code, unguided | 0.814 0.006 | 0.814 | 0.803 | 0.813 | 2.3 | 0.854 | 4.3 |
Table 1: Benchmark results (mean Dice over images, each image’s targets averaged first; agents: mean standard deviation of the track mean over three repeats, i.e. run-to-run variation; for the uncertainty over images see Section 6.1 ). Numbers of images in brackets. Radiology: human CT, MRI and PET of the head and neck, chest, abdomen and pelvis (25 targets); legs, mouse, ant: the thigh and calf MRIs, the foot bones in a whole-body CT, and the micro-CTs of a mouse and an ant (26 targets). Rad. without adrenal_ct : the radiology images except the one on which VoxTell returns an empty mask (Section 6.3 ). “Scribble oracle, same prompts”: its Dice after as many prompts on each target as the agent sent on it (after five where the agent sent more). Best Dice per column in bold.
Figure 7: Agents compared with oracles (mean Dice over images). (a) Dice over the number of prompts per target: oracle trajectories, the agents at their mean number of prompts (filled: guided, hollow: unguided) and after their first prompt on each target (small markers at one prompt), nnInteractive with one point, and VoxTell. (b) Each agent next to the scribble oracle given the agent’s own number of prompts on each target.
| Prompts per target | Scribbles | Boxes | Points | From code | Views per target | |
|---|---|---|---|---|---|---|
| Claude Code, guided | 2.2 | 94% | 4% | 2% | 14% | 9.1 |
| Claude Code, unguided | 2.3 | 94% | 3% | 2% | 17% | 10.4 |
| Codex, guided | 2.9 | 100% | 0% | 0% | 0% | 13.9 |
| Codex, unguided | 2.8 | 99% | 0% | 1% | 0% | 14.3 |
Table 2: How the agents work (all runs: 14 images, three repeats each). Prompt types as a share of all prompts sent; “from code”: prompts the agent sent from its own Python code. Prompts and views per target: means over images.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Image | Targets | Source, licence |
|---|---|---|
| pediatric_ct | liver, spleen, right kidney, bladder, heart, pancreas, gallbladder, left adrenal | Pediatric-CT-SEG (TCIA) [ 34 ] , CC BY 4.0 |
| hcc_ct | liver, liver tumour | HCC-TACE-Seg (TCIA) [ 35 ] , CC BY 4.0 |
| adrenal_ct | adrenal tumour | Adrenal-ACC-Ki67-Seg (TCIA) [ 36 ] , CC BY 4.0 |
| lung_ct | lung tumour | RIDER Lung CT, version 3 (TCIA) [ 37 ] , CC BY 4.0; tumour mask from RIDER-LungCT-Seg (TCIA) [ 38 ] , CC BY 3.0 |
| lung_pet | FDG-avid tumour | ACRIN-NSCLC-FDG-PET (TCIA) [ 39 ] , CC BY 3.0; lesion mask from the BAMF/AIMI annotations [ 40 , 41 ] , CC BY 4.0 |
| headneck_mr | mandible, brainstem, left parotid, right submandibular gland, thyroid, spinal cord, left optic nerve | HaN-Seg [ 32 ] , CC BY-NC-ND 4.0 |
Table S1: Benchmark sources. The build script downloads these sources and derives the reference masks as defined in benchmark/cases.json .
Figure S2: Mean Dice per image and track (over the image’s targets and, for the agents, the three repeats).
| Image | Target | Codex, guided | Claude Code, guided | Codex, unguided | Claude Code, unguided | One point | VoxTell | Scribble oracle, 5 prompts | Point oracle, 5 prompts |
|---|---|---|---|---|---|---|---|---|---|
| adrenal_ct | adrenal tumour | 0.91 | 0.91 | 0.90 | 0.90 | 0.82 | 0.00 ∗ | 0.93 | 0.91 |
| ant_microct | brain | 0.93 | 0.79 | 0.93 | 0.79 | 0.44 | 0.33 | 0.94 | 0.74 |
| bladder_mr | bladder tumour | 0.76 | 0.77 | 0.71 | 0.76 | 0.81 | 0.86 | 0.86 | 0.85 |
| calf_mr | soleus | 0.75 | 0.79 | 0.74 | 0.58 | 0.57 | 0.00 | 0.88 | 0.84 |
| tibialis anterior | 0.83 | 0.81 | 0.82 | 0.63 | 0.72 | 0.06 | 0.91 | 0.86 | |
| medial gastrocnemius | 0.86 | 0.87 | 0.80 | 0.46 | 0.90 | 0.00 | 0.95 | 0.94 |
Table S2: Dice per target and track (agents and one point: mean over three repeats; oracles: after five prompts).
Figure S3: All tracks , images 1–7: reference, VoxTell, and the middle of three repeats of each agent track. Numbers: Dice of that run, averaged over the image’s targets.
Figure S4: All tracks , images 8–14 (as Fig. S3 ).
Figure S5: The web demo after “Segment and count all liver lesions.” (MSD Liver, case 51, Claude Opus 5.5): axial, coronal, sagittal and 3D views, the objects, and the agent’s report.
Figure S6: Previews and slices : Claude Code segmenting the heart in a paediatric CT ( pediatric_ct , Pediatric-CT-SEG [ 34 ] ; guided track). All panels are renders the agent requested. (a) The preview of its first scribble. (b) A montage after this prompt: only the left chambers are segmented (Dice 0.62). (c) The preview of a correction: the line lies in the lung, and the tool reports air intensities along it. The agent dropped this line without sending it. (d) A zoomed view of the slice: the mask covers the left chambers but not the right heart. The agent wrote “Heart prediction only covers the left chambers; adding the right heart with a positive scribble” and (e) drew the line again, inside the right heart, and sent it. (f) The montage after this second prompt. The agent committed the heart (Dice 0.92) with the note “Includes proximal pulmonary trunk root; great vessels otherwise excluded.”
Figure S7: A single prompt : Claude Code segmenting the pancreas, a low-contrast organ, in a paediatric CT ( pediatric_ct ; guided track). (a) The view in which the agent looked for the pancreas, with voxel coordinates, patient directions and the organs committed so far. (b) The preview of its first scribble. (c) The agent shortened the scribble and sent it. (d) The result of this single prompt, checked in an axial and a coronal slice. (e) The 3D rendering of the pancreas (yellow) with its committed neighbours. The agent then committed it (Dice 0.89) with the note “Low contrast against adjacent bowel; tail and head boundaries uncertain.”
Figure S8: A check in another plane : Claude Code segmenting the liver in a contrast-enhanced CT from the request alone ( hcc_ct , HCC-TACE-Seg [ 35 ] ; unguided track). (a) An axial view with voxel coordinates and patient directions. (b) The preview of its first scribble, along the liver. (c) The result after that prompt (Dice 0.66), checked in a coronal slice: the large tumour is missing (the agent segments the liver including its tumour and separates the tumour later). (d) The preview of a second scribble in that slice, which adds the tumour. (e) The 3D rendering of the liver before the agent commits it (Dice 0.97).
Figure S9: An insect brain : Claude Code segmenting the brain in a micro-CT of an ant’s head ( ant_microct [ 51 , 52 ] ; guided track). (a) A view with voxel coordinates and anatomical directions. (b) The preview of its first scribble, along the brain. (c) The result after that prompt, checked in another slice. (d) The 3D rendering of the object from three directions. The final object, after four prompts (one of them undone), reaches Dice 0.82.