# Operate and Troubleshoot Managed Clusters > Source: https://parallelworks.com/docs/compute/managed-clusters/operations # Operate and Troubleshoot Managed Clusters Use this guide for routine monitoring, agent updates, incident checks, and safe removal of a managed-cluster integration. :::info Responsibilities Organization administrators perform ACTIVATE actions such as agent updates and record removal. Linux or HPC administrators run host commands, validate system configuration, and preserve an independent recovery path. Coordinate both roles during troubleshooting and offboarding. ::: ## Monitor the Integration The agent sends a heartbeat every 30 seconds, including node identity, operating-system and hardware information, resource metrics, filesystems, and supported scheduler data. A node is marked offline after about 90 seconds without a heartbeat. Review these signals separately: | Signal | What it proves | What it does not prove | | --- | --- | --- | | **Online** | ACTIVATE recently received a heartbeat | The persistent tunnel is connected or SSH works | | **Tunnel Connected** | ACTIVATE can currently open tunnel streams to the agent | Linux identity, SSH, or session software is configured correctly | | Scheduler data | The controller agent detected supported scheduler services and commands | Every scheduler command or job is healthy | | Access-management logs | The agent attempted to reconcile desired settings | The user can complete a real login and workload | Use a real pilot terminal, session, and test job after changes that affect networking, identity, SSH, the scheduler, or the agent. ## Update Agents When an online, tunnel-connected node reports an older supported agent version, organization administrators can select **Update agent** from that node's actions. Use **Update all agents** on **Nodes** to dispatch the update to every eligible node. During an update, the running agent replaces itself with the new binary and reconnects without starting a new systemd service process. It reports its new version on a later heartbeat. Offline, disconnected, already current, already updating, or unsupported agents are skipped and reported separately. Update a pilot first, verify heartbeats, tunnel connectivity, access, and scheduler data, and then update the remaining nodes in batches. ## Troubleshooting ### A Node Does Not Finish Registering - Generate a new token; the previous token may have expired, been used, or been exposed. - Confirm the node can reach the exact ACTIVATE platform host over TCP port 443 with trusted TLS. - Confirm the CLI was installed at `/usr/local/bin/pw` and rerun the displayed registration command with root privileges. - Confirm the system hostname is not already in use by a different registered machine. - Read logs with `sudo journalctl -xefu pw-agent --output=cat`. ### A Node Is Offline or Its Tunnel Is Disconnected - Check `sudo systemctl status pw-agent` and, if appropriate, restart it with `sudo systemctl restart pw-agent`. - Confirm outbound HTTPS and WebSocket traffic is not blocked or intercepted by a proxy. - Remember that heartbeat and tunnel status are separate. A node can briefly remain online after its tunnel disconnects. - If the node is the login target, determine whether ACTIVATE is attempting the [documented direct fallback](/docs/compute/managed-clusters/planning-and-setup#login-address) and whether that route is intended. ### Metrics on TCP 9100 Are Exposed or Unavailable - Restrict TCP 9100 at the host or network firewall to approved monitoring systems. The endpoint has no credential check. - Confirm no other service is already using port 9100. - Check the agent log for a Prometheus exporter startup error. - A failure of this listener does not by itself prove that platform heartbeats or the managed tunnel have failed; check them independently. ### Scheduler Partitions or Jobs Do Not Appear - Confirm the agent is installed on the scheduler controller. - For Slurm, confirm the node can run `squeue --version`, `sinfo`, and `squeue`, and that `slurmctld` is running. - For PBS, confirm the node can run `qstat --version` and `qstat`, and that `pbs_server` is running. - Expect partitions only for Slurm. The PBS integration reports jobs, not partitions. - Review agent logs for scheduler command errors. ### A User Cannot Sign In - Confirm the user or one of their groups has Login, Sudo, or Admin access. - Confirm the user's ACTIVATE username and POSIX UID do not conflict with local identities. - For an ACTIVATE-managed identity, verify **User Population** is enabled on the login node and check `getent passwd ` and `id `. - If the user needs automatic home creation, verify **Home Directories** is enabled and inspect PAM logs. - If using ACTIVATE SSH keys, verify **SSH Keys** is enabled, the user has a public key under **Account > Authentication**, and `sshd` accepted its managed configuration. - Wait for a subsequent heartbeat, then inspect both agent and SSH service logs. ### Sudo Membership Does Not Appear - Enable both **User Population** and **Sudo Access** on the node. The sudoers rule alone does not synchronize `pwsudo` group membership. - Confirm the user has Sudo or Admin cluster access. - Check `id ` after the next heartbeat and review agent logs. - Confirm that a node-level override has not disabled either required setting. See [Node-Level Overrides](/docs/compute/managed-clusters/identity-and-access#node-level-overrides). ## Offboard a Node or Cluster Deleting a node or cluster in ACTIVATE revokes platform access and removes its ACTIVATE record, but it does not uninstall the agent or restore every host change. Disable access management while the agent remains registered and online so it can perform its supported cleanup. ### 1. Prepare the Workload and Recovery Path 1. Notify users and account for running jobs, sessions, processes, files, scheduled tasks, and services. 2. Preserve an administrative access path that does not depend on ACTIVATE-managed identities or SSH keys. 3. Back up affected NSS, PAM, `sshd`, and sudo configuration under your site's change process. ### 2. Disable Managed Access Settings In the cluster and any node overrides, disable **Sudo Access**, **SSH Keys**, **Home Directories**, and **User Population**. Keep the agent running and connected through at least a subsequent heartbeat. Reset or disable overrides that would otherwise keep a setting enabled. Verify on every affected node that: - `/etc/sudoers.d/pwsudo` is removed. - The ACTIVATE-managed `AuthorizedKeysCommand` configuration is removed and SSH still validates and restarts successfully. - ACTIVATE-managed PAM home-directory configuration and its first-login key-generation script are removed. - `/etc/passwd.cache`, `/etc/shadow.cache`, and `/etc/group.cache` are removed. - Users can still sign in through the independent administrative path. Disabling **User Population** intentionally leaves the installed `libnss_cache` library and `cache` references in `nsswitch.conf`; the cache files are removed. Review those retained artifacts with the Linux administrator. Remove them only if no other service uses them and your normal host-change procedure calls for it. Existing home directories and user-owned data are not deleted. ### 3. Remove the ACTIVATE Records To remove one node, open **Nodes**, select the node, open **Settings**, and click **Delete Node**. ACTIVATE removes its node record, revokes the node credential, and disconnects its tunnel. To remove the integration, select **Delete** from the cluster actions menu. This removes only the ACTIVATE record; it does not delete machines, scheduler configuration, user data, or agent installations. ### 4. Remove Host Artifacts On each retired node, stop and disable the service: ```bash sudo systemctl disable --now pw-agent ``` After confirming that no other service uses them, remove `/etc/systemd/system/pw-agent.service` and reload systemd under your normal administration process. Review `/root/.config/pw/credentials` for the agent identity; do not delete the whole credentials file if it contains another CLI context. The [`pw context` commands](/docs/cli/pw/context) can list and remove individual contexts. Remove `/usr/local/bin/pw` only if no user, service, or remaining ACTIVATE integration on the host needs the CLI. Finally, verify that the service cannot restart, the node no longer appears in ACTIVATE, TCP 9100 is closed, and no unintended ACTIVATE-managed access remains.