cs.DCSep 5, 2026

Diamond Agent: Agentic Control of Federated HPC Resources as a Service

Authors: Haotian Xie, Junlin Chen, Mingkai Zheng, Yifan Zhu, Minu Mathew, Max Burnette, Yadu Babuji, Volodymyr Kindratenko, 

Organizations: Rutgers University · University of Rochester · National Center for Supercomputing Applications · University of Chicago · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Efficiently aggregating and orchestrating computing power across heterogeneous clusters for HPC workflows faces four practical challenges: preserving workflow context across independently administered clusters, moving large datasets between sites, reasoning about site-specific environments and scheduler policies, and exploiting live queue and resource states for efficient task scheduling. To this end, we design Diamond Agent, an agentic system that enables intelligent execution of HPC workflows across heterogeneous clusters with typed skills as the interface. Diamond Agent provides an agent-facing workspace and skills that unify cross-site resource discovery, resource specification, data movement, task execution, and result retrieval. A centralized Diamond Agent instance can operate multiple supercomputers without being deployed separately on each login node. Diamond Agent translates high-level agent actions into valid site-specific executions, moves data through Globus Transfer, and uses live system capability and queue information to select feasible placements. Its event-driven continuation mechanism decouples agent actions from long-running batch jobs: persistent services monitor remote execution and resume the agent only when a result or decision-relevant event is available. We experiment with 27 hours of telemetry and 19 matched multi-site submission rounds comprising 83 jobs across four production supercomputers. Compared with a fixed-site baseline, Diamond Agent reduces the median additional completion time relative to the fastest observed placement from 42 seconds to 4 seconds, a 10.5x reduction.

Figures & tables

Fig. 1: Pending jobs and allocatable accelerators in the four-system monitoring window. Delta, DeltaAI, and Anvil use Generic Resources(GRES) level free-device counts. Vista gh-dev uses idle nodes converted at one GH200 per node. Larger filled markers denote medians.
OperationAgent intentImmediate responseLater trigger condition
inspect_federationObserve systems, queues, capacity, health, and credit scopeTimestamped normalized snapshotNone
draft_envelopeConvert a task description into auditable resource bounds and constraintsTyped draft, rationale, confidence, validation resultNone
submit_taskExecute one scientific or engineering actionTask handle, selected placement, granted shape, decision traceCompletion, failure, unresolved policy event, or requested milestone
submit_groupFan out related tasks while preserving one objectiveGroup and child handles, placements, round stateRound terminal, stalled child, or group exception
move_dataStage or replicate large inputs and outputs across sitesTransfer handle, source and destination references, policy traceTransfer completion, failure, or requested milestone
get_resultPerceive outputs and provenanceResult manifest, attempt history, selected metadataNone
TABLE I: Agent-facing operations and asynchronous continuation semantics.
Fig. 2: Diamond Agent Architecture
SystemArch.AcceleratorsPartitions / access
Delta@NCSAx86_64A100 40G, A40 48G3, multi-user endpoint
DeltaAI@NCSAaarch64GH200 120G, 4/node1, login-node endpoint
Anvil@RCACx86_64A100 40G, H100 80G2, login-node endpoint
Vista@TACCaarch64GH200 96G, 1/node3, login-node endpoint
TABLE II: Federation used in the evaluation.
WorkloadAI envelopeSelected shapeActual completionObserved implication
CPU preparation4–64 CPUs 2–8 GB no GPU64 CPUs 2 GB42 s; minimum 4-CPU shape: 226 s; 128-CPU probe: 2029 s64 CPUs reduced runtime without a queue penalty. 128 CPUs still scaled in-kernel but waited 2012 s, so the AI upper bound was conservative for scaling and appropriate for completion.
Single-GPU inference1 GPU 2–16 CPUs 16–64 GB1 GPU 16 CPUs 16 GB146 s; minimum 2-CPU shape: 148 sGPU utilization and runtime were unchanged across the tested CPU range, so the upper CPU bound provided no completion benefit and should be refined by later profiling.
DDP training1–4 GPUs 4–32 CPUs 16–128 GB4 GPUs 32 CPUs 16 GB345 s; 1 GPU: 58 s; 2-GPU probe: 55 sThe 1 →\rightarrow 2 →\rightarrow 4 GPU speedups were 1.90 and 1.96 per doubling, so 4 GPUs passed the usefulness threshold, but its longer queue wait made 1–2 GPUs better for end-to-end completion.
TABLE III: AI-generated ResourceEnvelopes and actual sizing outcomes. Values are means over three repetitions unless noted. The 128-CPU and 2-GPU rows are two-repeat scaling probes.
PolicyFastest selectedMedian CCMean CCMean waitMedian additional completion timeP95 additional completion time
Fixed Delta0%227s1422s1198s42s5873s
Shortest feasible queue47%322s738s513s4s3000s
Best-feasible47%196s817s592s4s3000s
Fastest observed placement100%194s281s57s0s0s
TABLE IV: Actual matched-submission outcomes over 19 rounds and 83 jobs. Completion and additional completion time are measured from scheduler accounting, not model replay.
Fig. 3: Live task-group adaptation. Child c3 was queued on Anvil. After its siblings completed, the soft distinct-system constraint was released and c3 was re-placed on Vista.
AgentRemote jobAsync intervalStatus calls5-min polls
Claude CodeDelta, A4037 min08
CodexDeltaAI, GH20084 min017
TABLE V: Event-driven behavior in two end-to-end coding-agent traces. The polling column is a derived five-minute counterfactual, not a measured baseline.
More in this topic

High-Performance Computing

High-performance computing (HPC) uses parallel systems and optimized software to solve computationally demanding scientific and engineering problems.

See all 143 papers

Explore similar work

CardsList

  1. Hierarchical Server Architecture for Agentic Science

    Aug 5, 2026Vanessa Sochat, Daniel MilroyMulti-Agent CoordinationAI Agents for Scientific Discovery

    Lawrence Livermore National Laboratory Livermore, California, USA

  2. ASCEND: Personal AI Agents for Autonomous Scientific Computing Across HPC Clusters and GPU Workstations

    Sep 26, 2026J. Paul Liu, Uthpala Herath, Andrew PetersenAI for ScienceHigh-Performance Computing

    AI Hub for Science, College of Sciences, NC State University, Raleigh, NC · Research Computing and Support Services, Office of Information Technology, Duke University, Durham, NC · Office of Information Technology, NC State University, Raleigh, NC

  3. Multi-Agent Orchestration for High-Throughput Materials Screening on a Leadership-Class System

    Apr 9, 2026Thang Duc Pham, Harikrishna Tummalapalli, Fakhrul Hasan Bhuiyan +5Multi-Agent OrchestrationScientific Workflow Automation

    Computational Science (CPS) Division, Argonne National Laboratory, Lemont IL 60439, USA · Argonne Leadership Computing Facility (ALCF), Argonne National Laboratory, Lemont IL 60439, USA