Connecting a CoreWeave Slurm Cluster

This guide connects a CoreWeave Slurm cluster (SUNK - Slurm on Kubernetes) to ACTIVATE so your users can submit batch jobs, open desktop sessions, and run workflows on it.

There are two sides to it:

1. Identity - point the cluster's identity cache (nsscache) at ACTIVATE's SCIM API and add an SSH authorized-keys command, so ACTIVATE users, groups, and SSH keys resolve as real Linux accounts on the cluster.
2. Connection - register the cluster in ACTIVATE as an existing cluster.

Each identity step below can be applied two ways - pick one and stay consistent:

• Helm values (GitOps) - set everything in the values file your SUNK slurm chart deployment consumes. This is the right choice when the deployment is managed through ArgoCD (the usual setup for CoreWeave-managed clusters). Depending on the Application's sync policy, hand-edited chart-managed resources are reverted immediately (self-heal), reverted on the next sync (auto-sync), or left in place but flagged as out-of-sync drift. Changes made through the values file survive in every case.
• kubectl (manual) - edit the ConfigMaps directly. Fine for a quick proof of concept, or on clusters where automatic sync is disabled.

Prerequisites

• Organization admin permissions in ACTIVATE.
• SCIM provisioning enabled for your organization, plus a bearer token. Follow SCIM Provisioning first and keep the token and endpoint URL handy.
• kubectl access to the cluster's tenant-slurm namespace (via the kubeconfig from CoreWeave).
• For the Helm values method, write access to the GitOps repository whose values file the slurm ArgoCD Application consumes.
• POSIX UIDs/GIDs and SSH public keys configured on your ACTIVATE users and groups - these are what get synchronized onto the cluster.

Point nsscache at ACTIVATE's SCIM API

CoreWeave's SUNK clusters resolve Linux identity through nsscache, which periodically syncs passwd, group, shadow, and sshkey maps from a source. We configure that source to be ACTIVATE's SCIM API, reading POSIX identity from the CoreWeave extension attributes.

nsscache:
  enabled: true
  existingSecret: nsscache-scim-secret
  nsscacheConfig:
    default:
      source: scim
      scim_base_url: https://<platform-host>/api/organizations/<organization>/scim/v2
      scim_users_endpoint: Users
      scim_users_parameters: attributes=urn:coreweave:params:scim:schemas:extension:coreweave:2.0:CoreWeaveUser
      scim_groups_endpoint: Groups
      scim_groups_parameters: excludeInactiveUsers=true
    passwd:
      scim_override_home_directory: /mnt/home/%%u
    group:
      scim_path_gid: urn:coreweave:params:scim:schemas:extension:coreweave:2.0:CoreWeaveGroup/sunkPosixGroupId
      scim_path_groupname: urn:coreweave:params:scim:schemas:extension:coreweave:2.0:CoreWeaveGroup/sunkPosixGroupName

What the key settings do:

• scim_base_url - your organization's SCIM endpoint, shown on the SCIM Provisioning page (https://<platform-host>/api/organizations/<organization>/scim/v2).
• scim_users_parameters=attributes=...CoreWeaveUser - requests the CoreWeave user extension. ACTIVATE omits that block by default, so without this parameter the POSIX UID/GID, shell, home directory, and SSH keys would be missing.
• scim_groups_parameters=excludeInactiveUsers=true - drops disabled ACTIVATE accounts from group membership, so deactivated users stop resolving on the cluster.
• [group] paths with the ...CoreWeaveGroup URN prefix - ACTIVATE nests sunkPosixGroupId and sunkPosixGroupName under the CoreWeave group extension in each Group resource. Without the URN prefix, nsscache looks for them at the top level, finds no GID, and skips every group (Group <name> missing GID, skipping in the update job logs). members/sunkPosixUsername stays unprefixed - member entries carry that field at the top level.
• scim_override_home_directory=/mnt/home/%%u - forces home directories under /mnt/home. This overrides the sunkPreferredHomeDirectory value from SCIM; set it to wherever home directories are mounted on your cluster.

Provide the bearer token

The SCIM API requires a bearer token on every request. On a SUNK cluster, nsscache reads it from the nsscache-scim-secret Secret in the tenant-slurm namespace - not from the configuration above (in the Helm values, nsscache.existingSecret names it). This Secret is provisioned with the cluster; update it with the token you minted in SCIM Provisioning:

kubectl edit secret nsscache-scim-secret -n tenant-slurm

Secret values are base64-encoded, so encode the token before pasting it into the Secret's data field:

printf '%s' '<your-scim-token>' | base64

Configure the authorized keys command

So that sshd can authorize logins using each user's ACTIVATE SSH keys, install an AuthorizedKeysCommand that fetches them through the pw CLI.

login:
  sshdConfig: |
    PasswordAuthentication no
    AllowAgentForwarding yes
    AuthorizedKeysCommand /usr/local/share/pw-authorized-keys-command.sh
  s6:
    pw-authorized-keys-setup:
      type: oneshot
      timeoutUp: 120000
      script: |
        #!/usr/bin/env bash
        # Writes the AuthorizedKeysCommand script referenced by
        # login.sshdConfig and pre-installs the pw CLI as root at pod start,
        # so key lookups (which run unprivileged) never need to install it.
        set -e
 
        cat > /usr/local/share/pw-authorized-keys-command.sh <<'EOS'
        #!/bin/bash
        # AuthorizedKeysCommand: fetch the user's SSH public keys via the pw CLI.
        set -e
 
        PLATFORM_HOST="${PLATFORM_HOST:-https://<platform-host>}"
        PW_INSTALL_DIR="${PW_INSTALL_DIR:-/usr/local/bin}"
        PW_BIN="$PW_INSTALL_DIR/pw"
        # /tmp is always writable, even by `nobody`. The lock only needs to
        # exist during one install attempt, so ephemeral storage is fine.
        INSTALL_LOCK="${PW_INSTALL_LOCK:-/tmp/pw-install.lock}"
 
        locate_pw() {
            if [ -x "$PW_BIN" ]; then
                return
            fi
            local found
            found="$(command -v pw 2>/dev/null || true)"
            if [ -n "$found" ] && [ -x "$found" ]; then
                PW_BIN="$found"
            fi
        }
 
        locate_pw
        if [ ! -x "$PW_BIN" ]; then
            # flock prevents concurrent sshd invocations from racing the install.
            (
                flock -x 9
                if [ ! -x "/usr/local/bin/pw" ] && ! command -v pw >/dev/null 2>&1; then
                    # Send install output to stderr so it doesn't end up in the
                    # keys stream sshd reads from stdout.
                    curl -fsSL https://<platform-host>/cli/install.sh \
                        | bash -s -- --to "$PW_INSTALL_DIR" 1>&2
                fi
            ) 9>"$INSTALL_LOCK"
            locate_pw
        fi
 
        if [ ! -x "$PW_BIN" ]; then
            echo "pw CLI not found and install failed" >&2
            exit 1
        fi
 
        # Validate username contains only safe characters.
        if [[ ! "$1" =~ ^[a-zA-Z0-9._-]+$ ]]; then
            exit 1
        fi
 
        exec "$PW_BIN" ssh-public-keys --platform-host "$PLATFORM_HOST" "$1"
        EOS
        chmod 0755 /usr/local/share/pw-authorized-keys-command.sh
 
        # Best-effort: the authorized-keys script lazily installs pw as a
        # fallback, so a transient failure here must not block pod startup.
        if ! command -v pw >/dev/null 2>&1 && [ ! -x /usr/local/bin/pw ]; then
            curl -fsSL https://<platform-host>/cli/install.sh \
                | bash -s -- --to /usr/local/bin \
                || echo "WARN: pw CLI pre-install failed; will retry lazily" >&2
        fi

kubectl delete pod -n tenant-slurm <login-pod>

kubectl exec -it -n tenant-slurm <login-pod> -- \
  /usr/local/share/pw-authorized-keys-command.sh <username>

On the first SSH login after the pod comes back, the script installs the pw CLI if it isn't already present, then calls pw ssh-public-keys to return the user's keys for sshd to authorize. When everything is wired up correctly, running the command manually prints that user's authorized SSH public keys - for example, passing mcquade returns mcquade's keys.

Register the cluster in ACTIVATE

With identity resolving on the cluster, connect it like any other on-premises cluster:

1. Follow Configuring Existing Clusters to create the cluster definition.
2. Set the Scheduler Type to Slurm.
3. Enter the Cluster Login Node (the cluster's login/jump host) and your Username.

You can use the __USER__ token in any field and ACTIVATE substitutes the logged-in user's username automatically.

Verify

First, confirm the identity cache is populating. nsscache writes each synced map into a slurm-nsscache-<map> Secret in tenant-slurm. Decode the passwd cache to check that your ACTIVATE users are landing on the cluster with the expected UID, GID, shell, and home directory:

kubectl get secret slurm-nsscache-passwd -n tenant-slurm -o yaml \
  | yq '.data."passwd.cache"' | base64 -d

The group, shadow, and sshkey maps populate the parallel slurm-nsscache-group, slurm-nsscache-shadow, and slurm-nsscache-sshkey Secrets (keyed group.cache, shadow.cache, and sshkey.cache). An empty or stale cache usually means the SCIM URL, the bearer token, or the attributes parameter is wrong. If the group cache specifically stays empty, check the latest slurm-nsscache-update-cronjob job logs: Group <name> missing GID, skipping means the [group] paths are missing the ...CoreWeaveGroup URN prefix described above.

Then confirm the end-to-end connection:

1. From the Sessions tab, power on the cluster and confirm the connection succeeds.
2. Confirm your account resolves on the cluster (id <username> should show the POSIX UID/GID and group memberships synced from ACTIVATE).
3. Submit a test job and confirm it runs. See Submitting Jobs via Slurm.