An overview of what changed at PW in the third quarter of 2024

Introducing the new Parallel Works CLI

The PW CLI being used to see the status of buckets.

We're excited to announce major improvements to our cloud clusters with the release of our next-generation provider.

For the past two years, we've provisioned clusters using Infrastructure as Code templates (IaC), which has allowed us to share the responsibility of development of cloud infrastructure provisioning with software engineers, system administrators, and our support staff which have more of an HPC focus. As we've scaled out our offering, and provide users with the ability to provision more varied types of resources across more clouds, we've learned a lot of lessons.

We want to provide a very transparent "window" into the provisioning process as there are a lot of places where things can go wrong. Lifting those problem moments up and making it easier to troubleshoot was a challenge using IaC, because it was ultimately meant to be used interactively. The best you can get is a set of logs, which for most users is not very helpful.

Another major concern we wanted to address was resiliency, so we ultimately needed more control over the entire provisioning process. In our newest provider, all of the code for provisioning clusters has been centralized into our core platform, using CSP-provided Software Development Kits. This will allow us to start building out more interesting and useful features around provisioning as well as rapidly adapt to changes in the CSPs' offerings.

Coming soon, we'll have a new module on the sessions page which shows each component that is being provisioned and its current status, and if something fails, it will be very easy to determine where things went wrong. Since we were already majorly overhauling the provisioning process, we also took this chance to tackle our "wish list" of cluster enhancements.

An overview of what changed at PW in the second quarter of 2024

We're excited to tell you about a new set of disk features we've been adding over the past several months.

Starting today, you can add optional disks to your clusters on all three cloud service providers (CSPs). Disks are created either by creating creating a persistent disk or inlining configuration directly into your cluster definition.

Disks support:

• Specifying their size in GiB or GB, depending on the CSP.
• Specifying a type, as offered by the CSP. This controls the performance characteristics of the disk.
• Optionally, restoring from a snapshot.
• Optionally, exporting the disk to the cluster via NFS.

We were excited to attend ACFEA WEST in February. Continue reading for more info and an overview of what changed at PW since January.

From left to right: Al Williams of the DIU, Nick Miller of AWS, Phyllis Rhodes, and Matthew Shaxted

We were excited to present at the 104th Annual Meeting of the American Meteorological Society (AMS). Continue reading for an overview of what changed at PW over the last month.

Matt Long presenting at the AMS Annual Meeting

Follow this guide to test a PW workflow that seamlessly integrates JupyterLab and Dask, allowing you to run a sample Jupyter Notebook with ease.

We're excited to introduce our new Senior Computational Scientist, Dr. Luke Peffers. He brings a wealth of experience and expertise in the field of computational science.

In November 2023, Parallel Works presented at AWS, Azure, and Google booths at Supercomputing 2023 as well as attended the Naval Postrgraduate school's NATO Tech Day. Continue reading for an overview of what changed at PW in November 2023.

Michael McQuade (center) and Unni Kirandumkara (right) presenting at the Microsoft booth at SC23

A guide to converting Docker to Singularity for MPI jobs across multi-node SLURM clusters

An overview of what changed at PW in the third quarter of 2023

A detailed guide to developing workflows that automate task execution across resources

v4 is a foundational release of the Parallel Works platform, with a completely reimagined workflow framework, cluster filesystem flexibility, and several other improvements.

Announcing our new feature to manage HPC storage resources in May 2023