Operate and Troubleshoot Managed Clusters

Use this guide for routine monitoring, agent updates, incident checks, and safe removal of a managed-cluster integration.

Monitor the Integration

The agent sends a heartbeat every 30 seconds, including node identity, operating-system and hardware information, resource metrics, filesystems, and supported scheduler data. A node is marked offline after about 90 seconds without a heartbeat.

Review these signals separately:

Use a real pilot terminal, session, and test job after changes that affect networking, identity, SSH, the scheduler, or the agent.

Update Agents

When an online, tunnel-connected node reports an older supported agent version, organization administrators can select Update agent from that node's actions. Use Update all agents on Nodes to dispatch the update to every eligible node.

During an update, the running agent replaces itself with the new binary and reconnects without starting a new systemd service process. It reports its new version on a later heartbeat. Offline, disconnected, already current, already updating, or unsupported agents are skipped and reported separately. Update a pilot first, verify heartbeats, tunnel connectivity, access, and scheduler data, and then update the remaining nodes in batches.

Troubleshooting

A Node Does Not Finish Registering

• Generate a new token; the previous token may have expired, been used, or been exposed.
• Confirm the node can reach the exact ACTIVATE platform host over TCP port 443 with trusted TLS.
• Confirm the CLI was installed at /usr/local/bin/pw and rerun the displayed registration command with root privileges.
• Confirm the system hostname is not already in use by a different registered machine.
• Read logs with sudo journalctl -xefu pw-agent --output=cat.

A Node Is Offline or Its Tunnel Is Disconnected

• Check sudo systemctl status pw-agent and, if appropriate, restart it with sudo systemctl restart pw-agent.
• Confirm outbound HTTPS and WebSocket traffic is not blocked or intercepted by a proxy.
• Remember that heartbeat and tunnel status are separate. A node can briefly remain online after its tunnel disconnects.
• If the node is the login target, determine whether ACTIVATE is attempting the documented direct fallback and whether that route is intended.

Metrics on TCP 9100 Are Exposed or Unavailable

• Restrict TCP 9100 at the host or network firewall to approved monitoring systems. The endpoint has no credential check.
• Confirm no other service is already using port 9100.
• Check the agent log for a Prometheus exporter startup error.
• A failure of this listener does not by itself prove that platform heartbeats or the managed tunnel have failed; check them independently.

Scheduler Partitions or Jobs Do Not Appear

• Confirm the agent is installed on the scheduler controller.
• For Slurm, confirm the node can run squeue --version, sinfo, and squeue, and that slurmctld is running.
• For PBS, confirm the node can run qstat --version and qstat, and that pbs_server is running.
• Expect partitions only for Slurm. The PBS integration reports jobs, not partitions.
• Review agent logs for scheduler command errors.

A User Cannot Sign In

• Confirm the user or one of their groups has Login, Sudo, or Admin access.
• Confirm the user's ACTIVATE username and POSIX UID do not conflict with local identities.
• For an ACTIVATE-managed identity, verify User Population is enabled on the login node and check getent passwd <username> and id <username>.
• If the user needs automatic home creation, verify Home Directories is enabled and inspect PAM logs.
• If using ACTIVATE SSH keys, verify SSH Keys is enabled, the user has a public key under Account > Authentication, and sshd accepted its managed configuration.
• Wait for a subsequent heartbeat, then inspect both agent and SSH service logs.

Sudo Membership Does Not Appear

• Enable both User Population and Sudo Access on the node. The sudoers rule alone does not synchronize pwsudo group membership.
• Confirm the user has Sudo or Admin cluster access.
• Check id <username> after the next heartbeat and review agent logs.
• Confirm that a node-level override has not disabled either required setting. See Node-Level Overrides.

Offboard a Node or Cluster

Deleting a node or cluster in ACTIVATE revokes platform access and removes its ACTIVATE record, but it does not uninstall the agent or restore every host change. Disable access management while the agent remains registered and online so it can perform its supported cleanup.

1. Prepare the Workload and Recovery Path

1. Notify users and account for running jobs, sessions, processes, files, scheduled tasks, and services.
2. Preserve an administrative access path that does not depend on ACTIVATE-managed identities or SSH keys.
3. Back up affected NSS, PAM, sshd, and sudo configuration under your site's change process.

2. Disable Managed Access Settings

In the cluster and any node overrides, disable Sudo Access, SSH Keys, Home Directories, and User Population. Keep the agent running and connected through at least a subsequent heartbeat. Reset or disable overrides that would otherwise keep a setting enabled.

Verify on every affected node that:

• /etc/sudoers.d/pwsudo is removed.
• The ACTIVATE-managed AuthorizedKeysCommand configuration is removed and SSH still validates and restarts successfully.
• ACTIVATE-managed PAM home-directory configuration and its first-login key-generation script are removed.
• /etc/passwd.cache, /etc/shadow.cache, and /etc/group.cache are removed.
• Users can still sign in through the independent administrative path.

Disabling User Population intentionally leaves the installed libnss_cache library and cache references in nsswitch.conf; the cache files are removed. Review those retained artifacts with the Linux administrator. Remove them only if no other service uses them and your normal host-change procedure calls for it. Existing home directories and user-owned data are not deleted.

3. Remove the ACTIVATE Records

To remove one node, open Nodes, select the node, open Settings, and click Delete Node. ACTIVATE removes its node record, revokes the node credential, and disconnects its tunnel.

To remove the integration, select Delete from the cluster actions menu. This removes only the ACTIVATE record; it does not delete machines, scheduler configuration, user data, or agent installations.

4. Remove Host Artifacts

On each retired node, stop and disable the service:

sudo systemctl disable --now pw-agent

After confirming that no other service uses them, remove /etc/systemd/system/pw-agent.service and reload systemd under your normal administration process. Review /root/.config/pw/credentials for the agent identity; do not delete the whole credentials file if it contains another CLI context. The pw context commands can list and remove individual contexts. Remove /usr/local/bin/pw only if no user, service, or remaining ACTIVATE integration on the host needs the CLI.

Finally, verify that the service cannot restart, the node no longer appears in ACTIVATE, TCP 9100 is closed, and no unintended ACTIVATE-managed access remains.