Operate and Troubleshoot Managed Clusters

Monitor nodes, update agents, troubleshoot connections, and remove a managed-cluster integration. ACTIVATE configuration changes require organization administrator permission; commands on a node require Linux administrative access.

Monitor the Integration

The agent sends a heartbeat every 30 seconds, including node identity, operating-system and hardware information, resource metrics, filesystems, and supported scheduler data. A node is marked offline after about 90 seconds without a heartbeat.

After a configuration change, open a terminal or session and run a test job to verify the affected features.

Update Agents

When an online, tunnel-connected node reports an older supported agent version, organization administrators can select Update agent from that node's actions. Use Update all agents on Nodes to dispatch the update to every eligible node.

The running agent replaces its binary, reconnects, and reports its new version on a subsequent heartbeat. Nodes that are offline, disconnected, already current, already updating, or unsupported are skipped and reported separately. Update the pilot first, verify its connection and features, then update the remaining nodes.

Troubleshooting

A Node Does Not Finish Registering

• Check token mode and expiry. Generate a replacement if a single-use token has been consumed or a reusable token has expired or been revoked.
• Confirm the node can reach the exact ACTIVATE platform host over TCP port 443 with trusted TLS.
• Confirm the CLI was installed at /usr/local/bin/pw and rerun the displayed registration command with root privileges.
• Confirm the system hostname is not already in use by a different registered machine.
• Read logs with sudo journalctl -xefu pw-agent --output=cat.

A Node Is Offline or Its Tunnel Is Disconnected

• Check sudo systemctl status pw-agent and, if appropriate, restart it with sudo systemctl restart pw-agent.
• Confirm outbound HTTPS and WebSocket traffic is not blocked or intercepted by a proxy.
• Check both heartbeat and tunnel status. A node can briefly remain online after its tunnel disconnects.
• If the node is the login target, check the login address and connectivity for the selected route.

Prometheus Metrics Are Unavailable

• Confirm firewall rules allow the intended monitoring system to reach TCP 9100. See Check Network Paths.
• Confirm no other service is already using port 9100.
• Check the agent log for a Prometheus exporter startup error.
• Check Online and Tunnel Connected separately; the Prometheus listener is independent of these connections.

Scheduler Partitions or Jobs Do Not Appear

• Confirm the agent is installed on the scheduler controller.
• For Slurm, confirm the node can run squeue --version, sinfo, and squeue, and that slurmctld is running.
• For PBS, confirm the node can run qstat --version and qstat, and that pbs_server is running.
• Check the data supported by the scheduler: partitions and jobs for Slurm, jobs for PBS.
• Review agent logs for scheduler command errors.

A User Cannot Sign In

• Confirm the user or one of their groups has Login, Sudo, or Admin access.
• Confirm the user's ACTIVATE username and POSIX UID do not conflict with local identities.
• For an ACTIVATE-managed identity, verify User Population is enabled on the login node and check getent passwd <username> and id <username>.
• If the user needs automatic home creation, verify Home Directories is enabled and inspect PAM logs.
• If using ACTIVATE SSH keys, verify SSH Keys is enabled, the intended key source is available, and sshd accepted its managed configuration. Personal SSH connections require the matching public key under Account > Authentication; platform and managed workspace connections can use their managed keys.
• Wait for a subsequent heartbeat, then inspect both agent and SSH service logs.

Sudo Membership Does Not Appear

• Check the effective No Root Access policy first. When it is enabled, ACTIVATE does not assign pwsudo membership, disables the managed sudoers rule, and prevents enabling Sudo Access.
• If policy permits sudo, enable User Population for synchronized pwsudo membership and Sudo Access for the managed sudoers rule. They are separate controls.
• Confirm the user has Sudo or Admin cluster access.
• Check id <username> after the next heartbeat and review agent logs.
• Confirm that a node-level override has not disabled either required setting. See Node-Level Overrides.

Identity Records or Revocation Do Not Match Expectations

• Check agent logs to determine whether the node uses systemd user records or local Linux accounts. Use the checks for that storage method below.
• For the userdb backend, inspect the ACTIVATE-owned records in /etc/userdb and confirm systemd appears in the passwd, group, and shadow sources in /etc/nsswitch.conf.
• For local accounts, inspect getent passwd <username> and id <username>. The agent uses /var/lib/pw/localusers.db to identify the accounts and memberships it manages; retain this file through cleanup.
• Resolve reported name, UID, or GID collisions with local or directory-service identities. The agent does not take over unrelated existing accounts.
• Check all organization and group permissions that apply to the user. Another permission assignment can still give them access.
• Keep the agent online while it applies access changes and inspect any errors. Confirm that it has removed the user's managed account or updated group membership. See Remove or Reduce Access.

Offboard a Node or Cluster

To remove the integration, first let the agent clean up its managed configuration, then delete the ACTIVATE record and uninstall the service. The agent needs to remain registered and online during cleanup.

1. Prepare the Nodes

1. Finish or move the jobs and sessions that use the integration.
2. Sign in with a Linux administrator account that will remain available after ACTIVATE-managed accounts and SSH settings are removed.

2. Disable Managed Access Settings

For one node, set Sudo Access, SSH Keys, Home Directories, and User Population to Disabled in that node's settings overrides.

For the whole cluster, disable these features in the cluster settings and reset or disable any node overrides that keep them enabled.

Choose Disabled to request cleanup; Unset leaves the feature unmanaged. Keep the agents connected while they apply the changes on a subsequent heartbeat. User Population cleanup can terminate user processes.

Verify on every affected node that:

• /etc/sudoers.d/pwsudo is removed.
• The ACTIVATE-managed AuthorizedKeysCommand configuration is removed and SSH still validates and restarts successfully.
• ACTIVATE-managed PAM home-directory configuration and its first-login key-generation script are removed.
• ACTIVATE-owned records in /etc/userdb or managed local accounts and memberships are removed, according to the node's backend.
• The Linux administrator account can still sign in.

Cleanup retains home directories, generated user SSH keys, and user-owned files. The systemd source in /etc/nsswitch.conf, local groups, and SELinux settings can also remain. Remove retained configuration only after checking whether other services use it. Keep /var/lib/pw/localusers.db until local-account cleanup is verified.

3. Remove the ACTIVATE Records

To remove one node, open Nodes, select the node, open Settings, and click Delete Node. ACTIVATE removes its node record, revokes the node credential, and disconnects its tunnel.

To remove the cluster record, select Delete from the cluster actions menu. The machines, scheduler configuration, and user data remain in place.

4. Remove Host Artifacts

On each retired node, stop and disable the service:

sudo systemctl disable --now pw-agent

Remove /etc/systemd/system/pw-agent.service and reload systemd. Use the pw context commands to remove the agent's context from /root/.config/pw/credentials, preserving any other CLI contexts. Remove /usr/local/bin/pw if the node no longer uses the ACTIVATE CLI.

Confirm the service is stopped and disabled, the node record is removed from ACTIVATE, and TCP 9100 is closed.