Skip to main content

Accessing Storage Configuration Settings

You’ll see the storage configuration page immediately after creating any storage option.

You can also access configuration settings from the Storage tab. Click the name of any existing storage to configure it.

Screenshot of the user selecting a storage option in the Storage tab.

About the Storage Configuration Page

When you navigate to the storage configuration page, there are five tabs for customization.

Sessions

By default, you’ll see the Sessions tab when you navigate to storage settings. This tab shows your previous sessions for using the storage as well as provision and deletion logs.

Screenshot of the Sessions tab in a storage option's configuration settings.

About Storage Provisioning Logs

For persistent storage options, you can see the creation and deletion logs in this tab. All logs for current and previous sessions will be displayed. If a persistent storage option is active, you’ll see all the clusters that the storage is attached to.

Ephemeral storage options do not have a Sessions page. Instead, you can see their creation logs by navigating to Resources > ResourceName > Sessions > Logs > Storages.

For more information about persistent and ephemeral storage, please see About Storage Types.

Definition

Here, you can adjust the configuration parameters of your storage. For more information, see General Settings below.

Screenshot of the Definition tab in a storage option's configuration settings.

JSON

This tab shows the code version of your storage configuration settings. Here, you can manually adjust the parameters seen in the Definition tab.

Screenshot of the JSON tab in a storage option's configuration settings

Properties

This tab shows the name that you entered when the storage was created. Here, you can change the name and upload a new thumbnail for the cluster.

Screenshot of the Properties tab in a storage option's configuration settings

Sharing

This tab lets you control storage sharing options for users in your organization who are assigned to the same group.

Screenshot of the Sharing tab in a storage option's configuration settings

For more information, please see Sharing Storage.

General Settings

Storage options have these settings in the Definition tab of the configuration page.

Screenshot of the general settings in a storage option's configuration settings.

Cloud Infrastructure

Use this dropdown menu to choose your infrastructure, which determines which region your storage will be created in.

Group

Use this dropdown menu to select the group name that your organization uses to allocate costs. This menu is especially important if your organization uses multiple groups.

If you’re not sure which group to select, you can contact us or your organization’s PW platform administrator.

Lustre Options: AWS FSx for Lustre

The configuration parameters below are exclusive to AWS FSx for Lustre.

Screenshot of configuration settings for AWS FSx for Lustre.

Availability Zone

Use this dropdown menu to select the availability zone that your infrastructure will be deployed in. An availability zone refers to an isolated location inside a region.

We recommend that your storage availability zone matches the availability zone of the cluster you’re pairing the storage with.

Storage Capacity (GB)

Use this field to enter the total capacity of your storage. This field is set to 1200 by default. 1200 is the minimum amount of storage and must be increased in increments of 2400.

Please note that altering the storage size will affect your estimated hourly cost.

Filesystem Deployment

Use this dropdown menu to select which type of filesystem your storage will be deployed as. Currently, the options are scratch and persistent.

Scratch filesystems are meant for shorter-term workloads. They provide a higher throughput per TiB of storage capacity. If a scratch filesystem fails, your data is not replicated.

Persistent filesystems are meant for long-term workloads. If this filesystem fails, your data will automatically replicate in the same Availability Zone.

Filesystem Compression

Use this dropdown menu to select whether your filesystem will be compressed. Currently, the options are LZ4 and NONE.

LZ4 compresses data when writing to Lustre. When reading from Lustre, LZ4 decompresses data while minimally impacting performance. Using LZ4 can improve read and write performance as well as reduce storage capacity.

S3 Import Path (Optional)

Use this field to enter an S3 bucket path to import data from. The format for this field is s3://BucketName.

S3 Export Path (Optional)

Use this field to enter an S3 bucket path to export data to. The format for this field is s3://BucketName.

Lustre Options: Lustre on Google

The configuration parameters below are exclusive to Lustre on Google.

Screenshot of configuration settings for Lustre on Google.

Region

Use this dropdown menu to select the region that your storage will be deployed into. A region represents a geographic area.

Zone

Use this dropdown menu to select the zone to use your storage. A zone refers to an isolated location inside a region.

We recommend that your storage zone matches the zone of the cluster you’re pairing the storage with.

MDS Nodes

Use this field to enter the number of nodes on your storage’s MDS. This field is set to 1 by default and will be sufficient for most workloads.

About MDS

MDS stands for Metadata Service. In Lustre filesystems, the MDS manages the namespace hierarchy, records the layout of files, and handles object allocation on object storage targets (OSTs).

MDS fields largely affect data transfer speeds.

MDS Machine Type

Use this dropdown menu to select the machine type for your MDS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.

Please note that altering the MDS machine type may affect your estimated hourly cost.

MDS Boot Disk Type

Use this dropdown menu to select the type of disk your MDS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.

Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.

Please note that altering the boot disk type will slightly affect your estimated hourly cost.

MDS Boot Disk Size (GB)

Use this field to enter the total capacity of your MDS boot disk storage. This field is set to 20 by default.

Please note that altering the boot disk size will slightly affect your estimated hourly cost.

MDT Disk Type

Use this dropdown menu to select the type of disk your MDT nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.

Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.

Please note that altering the boot disk type will slightly affect your estimated hourly cost.

About MDT

MDT stands for Metadata Target. Metadata content is stored on MDTs.

MDT fields largely affect data transfer speeds.

MDT Disk Size (GB) (Optional)

Use this field to enter the total capacity of your MDT boot disk storage.

Please note that altering the boot disk size will slightly affect your estimated hourly cost.

OSS Nodes

Use this field to enter the number of nodes on your storage’s OSS. This field is set to 2 by default.

Changing the number of OSS disk nodes changes your storage’s capacity and overall throughput; if you’re interested in tuning your Lustre filesystem, this parameter as well as OSS disk type will most strongly affect your storage’s performance.

About OSS

OSS stands for Object Storage Servers. In Lustre filesystems, OSS provide the bulk data storage for all file content.

OSS Machine Type

Use this dropdown menu to select the machine type for your OSS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.

Please note that altering the OSS machine type may affect your estimated hourly cost.

OSS Boot Disk Type

Use this dropdown menu to select the type of disk your OSS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.

Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.

Please note that altering the boot disk type will slightly affect your estimated hourly cost.

OSS Boot Disk Size (GB)

Use this field to enter the total capacity of your OSS boot disk storage. This field is set to 20 by default.

Please note that altering the boot disk size will slightly affect your estimated hourly cost.

OST Disk Type

Use this dropdown menu to select the type of disk for your OST nodes. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.

Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.

Please note that altering the OST disk type will affect your estimated hourly cost.

About OST

OST stands for Object Storage Target. An OST is a set of storage volumes that OSS provides access to.

OST Disk Size (GB)

Use this field to enter the total capacity of your OST boot disk storage. This field is set to 1500 by default.

Please note that altering the boot disk size will slightly affect your estimated hourly cost.

Lustre Version

Use this field to enter the version of Lustre software that your storage will use.

This field is set to latest-release by default, and we recommend using this setting for the best performance.

Lustre Image

Use this dropdown menu to select the Lustre image that your storage will use. The image has prepackaged software that our engineers have installed to maximize performance.

We recommend selecting Latest for the best performance.

GVNIC

Use this toggle button to enable Google Virtual Network Interface Card (gVNIC), which supports higher network bandwidths from 50–100 Gbps. This parameter affects both MDS and OSS nodes.

This storage feature can be toggled independently of the gVNIC option on your cluster; your cluster’s configuration will not be affected.

For more information, see the Google documentation on gVNIC.

Tier 1

Use this toggle button to enable Tier_1, which increases maximum egress bandwidth (upload speed) to 50–100 Gps. If Tier_1 is off, the egress bandwidth will range from 10–32 Gbps. This parameter affects both MDS and OSS nodes.

Please note that Tier-1 is only supported if gVNIC is also active. If you try to start Tier-1 by itself, the PW platform will display the error message Tier_1 is only supported if gVNIC is on.

For more information, see the Google documentation on Tier_1.

Lustre Options: Lustre on Azure

The configuration parameters below are exclusive to Lustre on Azure.

Screenshot of configuration settings for Lustre on Azure.

Region

This field identifies which region the storage will be deployed in. Please note that this field is locked because the region is determined by which Cloud Infrastructure is selected.

Lustre Image

Use this dropdown menu to select the Lustre image that your storage will use. The image has prepackaged software that our engineers have installed to maximize performance.

We recommend selecting Latest for the best performance.

MDS Nodes

Use this field to enter the number of nodes on your storage’s MDS. This field is set to 1 by default and will be sufficient for most workloads.

About MDS

MDS stands for Metadata Service. In Lustre filesystems, the MDS manages the namespace hierarchy, records the layout of files, and handles object allocation on object storage targets (OSTs).

MDS fields largely affect data transfer speeds.

MDS Machine Type

Use this dropdown menu to select the machine type for your MDS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.

Please note that altering the MDS machine type may affect your estimated hourly cost.

MDS Boot Disk Type

Use this dropdown menu to select the type of disk your MDS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.

Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.

Please note that altering the boot disk type will slightly affect your estimated hourly cost.

MDS Boot Disk Size

Use this field to enter the total capacity of your MDS boot disk storage. This field is set to 20 by default.

Please note that altering the boot disk size will slightly affect your estimated hourly cost.

OSS Nodes

Use this field to enter the number of nodes on your storage’s OSS. This field is set to 2 by default.

Changing the number of OSS disk nodes changes your storage’s capacity and overall throughput; if you’re interested in tuning your Lustre filesystem, this parameter as well as OSS disk type will most strongly affect your storage’s performance.

About OSS

OSS stands for Object Storage Servers. In Lustre filesystems, OSS provide the bulk data storage for all file content.

OSS Machine Type

Use this dropdown menu to select the machine type for your OSS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.

Please note that altering the OSS machine type may affect your estimated hourly cost.

OSS Boot Disk Type

Use this dropdown menu to select the type of disk your OSS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.

Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.

Please note that altering the boot disk type will slightly affect your estimated hourly cost.

OSS Boot Disk Size

Use this field to enter the total capacity of your OSS boot disk storage. This field is set to 20 by default.

Please note that altering the boot disk size will slightly affect your estimated hourly cost.

Accelerated Networking

Use this toggle button to enable accelerated networking, which improves networking performance for large workloads on multiple cloud clusters.

For more information, see the Azure documentation on accelerated networking.

Cloud Object Stores: AWS S3 Bucket

The configuration parameters for AWS S3 Buckets are explained below.

Screenshot of configuration settings for AWS S3 Bucket.

Region

Use this dropdown menu to select the region that your storage will be deployed into. A region represents a geographic area.

Cloud Object Stores: Google Cloud Bucket

The configuration parameters for Google Cloud Buckets are explained below.

Screenshot of configuration settings for Google Cloud Bucket.

Region

Use this dropdown menu to select the region that your storage will be deployed into. A region represents a geographic area.