SC26AI agents lab + dinner on the river · Nov 17
Parallel Works
Back to Blog
hpcaicoreweaveslurmkubernetesgpu

From Node Handover to Running Jobs: ACTIVATE on CoreWeave

14 min read
Share:

A dedicated GPU reservation becomes useful to a research program only after accounts, allocations, software, and reporting are in place. This post describes how Parallel Works put those layers on a CoreWeave reservation for the DARPA NODES program, which Parallel Works and CoreWeave announced in September: researchers were running work three days after node handover, and the Slurm pool passed 80 percent utilization within three weeks.

The problem

GPU capacity is bought as hardware or as a provider reservation, and the work of turning it into a shared research environment usually falls to the customer. For a program with several research teams, that work includes accounts that resolve the same way in every interface, a policy for dividing GPUs among teams, a maintained software stack, usage reporting for the program office, and user support. A program that builds this itself typically spends weeks to months on it after the hardware arrives, and a reservation is billed whether or not anyone is using it.

The approach

CoreWeave provides the reservation, and Parallel Works operates it as a managed environment through ACTIVATE. ACTIVATE already implements each of the layers above: it is the source of truth for users and groups, it expresses allocation policy as Slurm accounts and QOS and as Kubernetes namespace quotas, and it carries the software environment and usage reporting. Parallel Works staffs the environment with the engineers who build it. Three things shortened the timeline on this deployment.

Integration done before the hardware: Parallel Works and CoreWeave built and tested the OIDC path into CKS and the SCIM path into SUNK on a CPU-only cluster in the spring, and documented both. The work after handover was configuration and verification.

Configuration as code: the cluster configuration lives in Git and Argo CD applies it, so identity, scheduler policy, monitoring, and user tooling went in as reviewed changes and are adjusted the same way.

No per-user setup: Slurm accounts, associations, and Kubernetes namespaces are created from ACTIVATE group membership, so onboarding a team means adding its members to a group.

Results

Researchers were running work on day 3 after node handover. GPU use then rose as teams onboarded. Measured as GPU-hours used by jobs against GPU-hours available in the Slurm pool, utilization reached about 70 percent by the end of the second week and about 80 percent in the third week, held between 70 and 80 percent through early September as the remaining teams joined, and reached 96 percent in the week of September 21. The usage report in Figure 6 shows the weekly trend.

Division of responsibilities

The environment is a dedicated reservation of NVIDIA HGX H100 systems on NVIDIA Quantum InfiniBand. At handover, CoreWeave provides the nodes, the fabric, distributed file storage, and two orchestration layers: SUNK, which runs Slurm on Kubernetes, and CoreWeave Kubernetes Service (CKS) for container workloads. CoreWeave also monitors fleet health and replaces failed hardware under its infrastructure SLA.

Figure 1. CoreWeave provides the GPU nodes, fabric, storage, SUNK, and CKS under its infrastructure SLA. ACTIVATE provides identity, allocation policy, the software environment, usage reporting, and support across both orchestration layers.

Identity: OIDC for CKS, SCIM for SUNK

ACTIVATE is the source of truth for users, groups, POSIX IDs, and SSH keys. Kubernetes and Slurm consume that identity through different standard mechanisms, and CoreWeave supports both without custom components (Figure 2).

Kubernetes: ACTIVATE is registered as the cluster's OIDC identity provider. Users receive short-lived tokens scoped to the cluster, with their ACTIVATE groups carried as claims, and the CKS API server validates each token against ACTIVATE. Sharing the cluster with an ACTIVATE group creates a namespace and RoleBinding for that group, so access follows group membership and nobody is provisioned by hand on either side.

Slurm: ACTIVATE publishes users and groups through a read-only SCIM 2.0 endpoint that includes CoreWeave's POSIX extension: username, UID, GID, shell, home directory, and SSH keys. SUNK polls the endpoint and keeps identity caches on the login pods, so accounts keep resolving if ACTIVATE is briefly unreachable, and a live SSH key lookup applies key changes at the next login. A separate sync turns ACTIVATE groups into Slurm accounts and associations, so group membership decides which account a user charges and which QOS tiers they can use.

Two lessons came out of this part of the build. The POSIX extension is returned only when it is requested, and without it users appear on the cluster with names but no UIDs, so it is the first thing to check when accounts do not resolve. And only one system should write Slurm associations: SUNK's built-in user provisioning and the ACTIVATE group sync overwrote each other until SUNK's was turned off.

Figure 2. Kubernetes access uses short-lived OIDC tokens and a namespace per group. Slurm access uses SCIM-fed identity caches on the login pods, live SSH key lookup, and a group sync that maintains Slurm accounts.

GPU sharing with Slurm QOS

The Slurm pool runs one GPU partition, set as the default, and shares it through Slurm QOS in three tiers (Figure 3).

Project QOS: each team has one, applied by default, that caps the team's concurrent GPUs at its guaranteed share. Project jobs have no walltime limit and are never preempted.

General QOS: an opt-in tier that bursts into a shared pool, with a per-team limit inside the pool and a three-day walltime. General jobs are never preempted.

Spot QOS: an opt-in tier, uncapped, that runs on otherwise idle GPUs for up to a day. When a project or general job needs the GPUs, the spot job gets a ten-minute grace window to checkpoint and is then requeued.

Figure 3. Each team's project QOS caps it at its guaranteed share. The general QOS bursts into a shared pool, and the spot QOS runs on idle GPUs and yields when a guaranteed or burst job needs them.

Caps limit how many GPUs a team runs at once, never how many jobs it queues. A job over its cap pends, Slurm reports which limit it is waiting on, and the job starts when capacity frees (Figure 4).

Figure 4. A job's path through the tiers. Only spot jobs are preempted, and a preempted spot job is requeued.

A few configuration points matter more than the QOS definitions themselves. Slurm ignores QOS GPU limits unless GPUs are tracked as an accounting resource and limit enforcement is turned on, so a cluster can look configured while enforcing nothing. Per-GPU memory and CPU defaults scale each job's memory and cores with its GPU count, so teams sharing a node do not starve each other. And the default account that newly provisioned users land in is capped at zero GPUs, so nobody runs outside the caps before being assigned to a team.

Researchers see their own limits with one command, mylimits, which shows their account, the tiers they can use, caps and walltimes, current jobs, and storage use. The public activate-cluster-demos repository walks through the tiers with runnable examples.

Kubernetes alongside Slurm

Some teams run container workloads that fit Kubernetes better than a batch scheduler. Those teams get CKS namespaces with GPU ResourceQuotas, enforced at admission, on nodes held outside Slurm. Both pools mount the same shared file system, so home, project, and software directories are the same paths in a Slurm job and in a pod (Figure 5).

The split between pools is a configuration value: the number of Slurm compute nodes is one setting in the Git-managed SUNK configuration. Moving a node between pools is a one-line change and a merge, with no rebuild of either pool.

Figure 5. The reservation divides into a Slurm pool and a Kubernetes pool over one shared file system. The node count in each pool is set in Git.

Software environment and module builds

NVIDIA drivers come from the GPU operator on the nodes, and everything else users load comes from a shared software tree. SUNK runs login and compute nodes as pods, and anything installed into a pod's filesystem disappears when the pod restarts. All user software therefore lives on a shared volume mounted at the same path on every login and compute pod, and the only per-pod setup is a short shell hook recreated at each pod start from the Git-managed configuration.

The module system follows the same rule. Lmod is installed onto the shared volume with its own Lua. A first attempt with the operating system package worked on the login pod and failed inside jobs, because the module command carried into the job environment pointed at a binary that existed only on the login pod.

Each package gets its own versioned directory. Most come from conda-forge with one environment per tool, so dependencies do not collide. The machine learning stack uses pip on a conda-provided Python for broader CUDA wheel coverage, and licensed codes such as NAMD are installed from the vendor's build. Every module sets a home variable for build systems and includes notes on running the tool under Slurm.

Two problems shaped the process. Installers twice left files unreadable to ordinary users, which root's access hides: the module loaded, and the user then hit a permission or import error. Every install is now followed by a permissions pass and a check as an ordinary user. Apptainer was dropped after testing, because its image format needs kernel features the node containers do not expose. Containers run through enroot and Pyxis instead, which start an NGC or Docker image directly as a Slurm job. Job templates cover the common job shapes, from single-GPU to multi-node, container, spot, and burst.

The login service runs two replicas, so one login pod can restart without disconnecting every user. SSH to a compute node is limited to users with a running job on that node.

Storage and home directory quotas

The shared file system holds three kinds of volume. /home is a single volume for all users, with per-user quotas that the storage system enforces by UID: a soft limit of about 10 GiB with a grace period and a hard limit of about 15 GiB. Past the hard limit, writes that grow the directory fail, while reads and deletes still work, so a user can always clean up. Each team's project directory is a separate volume, sized for that team, for datasets and job output, and a third volume holds the shared software. Small home directories keep large data in project space, where the whole team can see it.

Quotas are accounted by UID, so the stable POSIX UIDs that ACTIVATE assigns and publishes through SCIM are also what keep a quota attached to the right person across pod restarts. mylimits reads the user's live quota and reports usage against the soft and hard limits. An hourly job records the same figures for the usage report.

Monitoring and usage reporting

CoreWeave's observability stack monitors fleet health. For the program and its teams, ACTIVATE runs a monitoring and reporting stack in the cluster: VictoriaMetrics scrapes CoreWeave's DCGM exporters, node exporters, and a Slurm exporter, Grafana presents the telemetry, and a usage report is built from Slurm accounting. Both are reached through ACTIVATE sign-in. The usage report shows GPU-hours used by week against Slurm capacity, GPUs in use over time, and GPU-hours by team, with CSV export on each view (Figure 6). Grafana carries GPU and job telemetry (Figure 7).

Figure 6. The usage report through the week of September 21, with team names withheld and absolute values removed for this post. Bars show GPU-hours used each week against Slurm capacity (dotted line); the week of September 21 is at 96 percent.
Figure 7. Grafana panels for GPU utilization across the cluster and Slurm job states, 23 to 29 September.

Mission engineering and 24x7 support

Parallel Works delivers ACTIVATE with a mission engineering team: the engineers who configure and operate an environment also take user tickets, build software on request, and work with research teams on their codes and job scripts. On this environment that team provides 24x7 support with direct access to engineers, application-level help with scientific software, the module builds described above, onboarding, and the usage reporting the program receives. A request for a missing package or help with a failing job reaches an engineer who can change the environment.

CoreWeave carries the infrastructure SLA: hardware, fabric, storage, power and cooling, and replacement of failed nodes. Parallel Works carries the user experience: the ACTIVATE control plane, Slurm and Kubernetes configuration, software, onboarding, reporting, and support. A researcher contacts Parallel Works, and Parallel Works takes hardware issues to CoreWeave.

Bring-up sequence

Figure 8 shows the sequence after the reservation was handed over in late July. On the day of handover, SUNK was running with ACTIVATE identity, every node was in Slurm, and group sync was creating Slurm accounts. On day 1 the Kubernetes pool and its namespaces went live, along with the second login pod and the monitoring stack. On day 2 the QOS tiers were enforced, and mylimits and the job templates were in place. On day 3 the first researchers were running work, and the remaining teams joined over the following weeks.

Figure 8. From node handover to researchers running work.

Documentation

Step-by-step configuration for each piece is in the Parallel Works documentation and public repositories:

Related reading

The GPU Productivity Gap: Why Buying GPUs Is Easy and Using Them Is Hard

How Parallel Works ACTIVATE closes the gap between GPU spend and production AI

Read article →

Spending on AI Is Easy. Governing It Is Hard.

Enterprises learned that buying GPUs is not the same as using them. The same lesson is arriving one layer up, in tokens, and the fix is governing every token where it is spent: in the request path.

Read article →

Matthew Shaxted of Parallel Works On Pushing the Boundaries of AI

Parallel Works CEO Matthew Shaxted discusses the company's journey, AI infrastructure, and what it takes to shape the future of AI in this Authority Magazine interview.

Read article →