Building Workflows

A workflow is a set of jobs and steps that run to completion whenever the workflow is executed. A workflow may have one or more jobs, and a job may have one or more steps. The jobs run in parallel, while the steps run sequentially.

The input parameters that appear in the Run Workflow tab are defined by a YAML (.yaml) file. The YAML for local workflows is stored by the platform, so you can view and edit the YAML directly by using the Build tab. From there, you can define the jobs and inputs for your workflow.

YAML syntax

Since the workflows on ACTIVATE are written in YAML, it's worthwhile to get a grasp on YAML basics before diving into building workflows. For a a quick overview, check out the Red Hat website.

Navigate to Workflows > your workflow > the Build tab.

Newly created local workflows are loaded with a default YAML file configuration, as seen in the screenshot below.

jobs

The most essential part of the YAML configuration is jobs , which defines the jobs this workflow will run. Each job must define a list of steps to be executed using the steps field. By default, all jobs run in parallel. If a job needs to depend on the completion of another job, it can be specified using the needs field. You must have at least one job containing at least a single step for a workflow to be valid. Job names must be unique within a workflow.

jobs take the following shape in the YAML file:

jobs:
  job-name:
    steps:
      - name: Step 1
        run: ./folder/file1.sh
      - name: Step 2
        run: ./folder/file2.sh

jobs.<job-id>.needs

needs defines a list of jobs that must complete before this job can start. If a job fails, all jobs that depend on it will not run.

In the following example, the job first will run, and only after it successfully completes will the job second run.

jobs:
  first:
    steps:
      - run: echo hello world
  second:
    needs:
      - first
    steps:
      - run: echo hello again world

jobs.<job-id>.steps

A list of steps to be executed. These are executed in order, and if any step fails, the job will fail. Each step requires either a command to execute using the run field, or another workflow to execute by using the uses field. All other attributes are optional.

You can define steps with the attributes below.

jobs.<job-id>.steps[*].name

The name of the step.

Step names are useful when viewing a workflow's progress in the Runs tab. If a job fails at a certain step, you can quickly identify its name on the Runs graph.

If you don’t provide a name, the step's command will be shown instead.

jobs.<job-id>.steps[*].run

Defines the command to be executed, e.g. echo hello world. This cannot be used in conjunction with uses.

jobs.<job-id>.steps[*].uses

Runs another workflow as a step, e.g. marketplace/checkout. This cannot be used in conjunction with run.

uses calls an existing workflow either from another workflow or from the Marketplace. To reference another workflow, prefix the workflow name with workflow/. To reference a Marketplace workflow, prefix the marketplace workflow slug with marketplace/.

If the workflow requires inputs to run, use with to provide those inputs. The fields defined inside with are the same names as defined in the workflow's inputs section.

jobs.<job-id>.steps[*].with

Defines the inputs passed to the workflow defined by uses.

If you include uses in the YAML file to call a workflow, it may have inputs that must be defined in order to run. In that case, with is used to define those inputs. In the example below, we used the inputs from the default YAML configuration.

jobs:
  job-name:
    steps:
      - name: Sample Step
        uses: marketplace/sample-workflow
        with:
          resource: sample-cluster
          jobschedulertype: SLURM
          input_method: TEXT
          script_text: echo hello

jobs.<job-id>.steps[*].if

if prevents a step from running unless a conditional evaluates to true.

Example:

jobs:
  main:
    steps:
      - run: echo hello world
      - run: echo This ran because of an input!
        if: ${{ inputs.should-run }}
on:
  execute:
    inputs:
      should-run:
        type: boolean
        label: Run extra step?

In the above example, the second step will only run if the user selects Yes on "Run Extra Step?".

jobs.<job-id>.steps[*].env

Defines environment variables to be set for this step.

Example:

jobs:
  job-name:
    steps:
      - name: Print an environment variable
        run: echo $foo
        env:
          foo: bar

The above example will print bar.

jobs.<job-id>.steps[*].ignore-errors

If true, non-zero exit codes will not be counted as failures. By default, this attribute is set to false.

jobs.<job-id>.steps[*].timeout

Defines a maximum amount of time before the job automatically fails.

Supported units are:

• s (seconds)
• m (minutes)
• h (hours)
• d (days)

Example:

jobs:
  main:
    steps:
      - name: Sample Step
        run: sleep 60
        timeout: 10s

In the example above, the sleep command finishes after 1 minute. We set the timeout attribute to 10 seconds. Since the step has not completed in 10 seconds, the job will fail. Without a timeout value, the step will run until it finishes.

jobs.<job-id>.steps[*].cleanup

Defines a cleanup command to be run after job execution finishes, e.g. rm -rf /tmp/myapp.

This can be used to delete temporary files, terminate connections, remove credentials, or perform any any other clean up necessary when a job is shutting down. cleanup always runs at the end of a job in reverse order of definition. Clean up steps always run regardless of if a job was successful, failed, or was canceled. If a step did not run, it's cleanup step will not run.

For example, if you have a job with three steps that all have cleanup attributes, each cleanup will add a clean up step to the end of the job and the job will ultimately execute steps in this order:

• step 1 run
• step 2 run
• step 3 run
• step 3 cleanup
• step 2 cleanup
• step 1 cleanup

on

Use the on field to define the event that triggers the workflow. In a future release, the on field will support additional events. At this time, workflows only support the execute event, which is triggered when the workflow is manually executed via the UI or the REST API. When the workflow is manually triggered, the inputs context is populated with values from the input form.

on.execute.inputs

In addition to defining what your workflow does with jobs, you can define the workflow form users will fill out when running the workflow. This form is what's displayed in the Run Workflow tab, and the values of the inputs can be accessed from inside jobs using expressions. When defining the YAML file in the Build tab, the form is previewed on the right side of the screen.

See the example below for how to define the input form which users of the workflow will see:

on:
  execute:
    inputs:
      mystring:
        type: string

In the above example, the only field inside the inputs context is mystring, which would be referenced with ${{ inputs.mystring }}, for more information, see Expressions.

Types

Below is a list of each type and its modifying attributes. Unless otherwise noted, each type can be modified by any of the universal attributes.

boolean

A checkbox or toggle switch representing a boolean value (true/false).

Useful for optional settings, enabling/disabling features, or making binary choices (such as "Enable logging?").

color-picker

An input field that allows users to select a color.

Useful for customizing the appearance of visual elements, such as selecting colors for charts, backgrounds, or UI components.

compute-clusters

Represents a selection of compute clusters.

Attributes:

• csp: Specifies the CSP (cloud service provider). Options are aws, azure, and google.
• include-workspace: Indicates whether to include workspace clusters.
• provider: Specifies the compute resource provider.

Allows users to select the specific compute cluster where their job will run, essential for distributing workloads across available resources.

editor

A code editor input field with syntax highlighting.

Attributes:

• language: Specifies the language for syntax highlighting (such as yaml, json, etc.).

Useful for entering and editing code snippets, configuration files, or scripts directly within the workflow.

dropdown

A dropdown menu allowing the user to select from a list of predefined options.

Attributes:

• options: A list of options with label and value.
• option-key: A key to determine the selected option.

Ideal for selecting from multiple predefined options, such as job schedulers, input methods, or configuration settings.

group

A collapsible group of related input fields.

Attributes:

• items: A list of inputs that belong to this group.
• collapsed: Boolean to indicate if the group should be initially collapsed.

Organizes related inputs together. Especially useful for advanced settings or additional information that can be optionally expanded by the user.

header

Creates a header in the input form.

Attributes:

• text: The header text.
• bold: Indicates if the text should be bold.
• size: Font size of the header.

Adds titles or section headers in the input form for better organization and readability.

instance-type

Represents a selection of instance types for a specific CSP (cloud service provider) and region.

Attributes:

• csp: Specifies the CSP (aws, azure, or google).
• region: The region associated with the instance types.

Allows users to choose specific instance types for their compute tasks. Important for matching job requirements with appropriate resources.

list

A list of items, typically used for repeating input patterns.

Attributes:

• template: The template defining the type of items in the list.

Useful for inputs where multiple entries of the same type are needed, such as a list of regions, parameters, or configuration items.

multi-dropdown

A dropdown menu allowing the user to select multiple options.

Attributes:

• options: A list of options with label and value.

Ideal for selecting multiple items from a list, such as tags, categories, or resource groups.

number

An input field for entering numerical values.

Attributes:

• min: The minimum allowable value.
• max: The maximum allowable value.
• step: The increment step for the number input.
• slider: Boolean to indicate if the input should be displayed as a slider.

Useful for specifying numerical parameters, such as counts, thresholds, limits, or any other numeric settings.

organization-groups

Represents a selection of organization groups.

Attributes:

• csp: Specifies the CSP (cloud service provider). Options are aws, azure, and google.

Allows users to select organizational groups for permissions, access control, or grouping related resources.

region

Represents a geographic region associated with a CSP.

Attributes:

• csp: Specifies the CSP (cloud service provider). Options are aws, azure, and google.

Allows users to select a specific geographic region for their resources, essential for optimizing latency, compliance, and availability.

slurm-accounts

Represents a selection of SLURM accounts.

Attributes:

• resource: The compute resource associated with the SLURM accounts.

Allows users to select SLURM accounts for job submissions. Essential for managing resource allocation and accounting in SLURM environments.

slurm-partitions

Represents a selection of SLURM partitions.

Attributes:

• resource: The compute resource associated with the SLURM partitions.
• account: Specifies the SLURM account associated with the partitions.

Allows users to select SLURM partitions for job submissions. Essential for organizing and managing job queues in SLURM environments.

slurm-qos

Represents a selection of SLURM QoS (Quality of Service) settings.

Attributes:

• account: Specifies the SLURM account associated with the QoS settings.
• partition: Specifies the SLURM partition associated with the QoS settings.
• resource: The compute resource associated with the SLURM QoS.

Allows users to select SLURM QoS settings for job submissions. Essential for prioritizing and managing job performance in SLURM environments.

storage-image

Represents a selection of storage images.

Attributes:

• csp: Specifies the CSP (cloud service provider). Options are aws, azure, and google.
• subtype: Specifies the subtype of the storage image (such as lustre).

Allows users to select storage images for their jobs. Essential for specifying the data environment and ensuring compatibility with compute tasks.

string

A single-line text input or a multi-line text area for string values.

Attributes:

• placeholder: Placeholder text for the input field.
• textarea: Indicates if the input should be a text area.

Collects textual input from users. Suitable for names, descriptions, commands, or any other text-based configuration.

zone

Represents a selection of zones within a region for a specific CSP (cloud service provider).

Attributes:

• csp: Specifies the CSP. Options are aws, azure, and google.
• region: Specifies the region associated with the zone.

Allows users to select specific zones for their resources. Important for optimizing resource placement, redundancy, and compliance.

Universal Attributes

Most input types above can be modified by a set of common attributes, which help in configuring the basic behavior and presentation of the input fields. The only type without these universal attributes is header, which has its own unique set of attributes focused on display rather than interaction.

default

Sets the default value for the input field.

Usage: Provides an initial value if the user does not input one.

Example:

default: 'Enter description here'

disable

Disables the input field, making it read-only and uneditable by the user.

Usage: Use this when you want to display information without allowing the user to change it.

Example:

disable: true

hidden

A condition to hide or show the input field based on other inputs or logic.

Usage: Creates a dynamic form where inputs can be conditionally displayed.

Example:

hidden: ${{ inputs.show_advanced == false }}

ignore

When set to true, the input field's value is ignored and not sent to the backend.

Usage: Useful for fields that are needed for UI logic but should not be part of the final payload.

Example:

ignore: true

label

A human-readable name for the input field that will be displayed in the UI.

Usage: Helps users understand what data is expected.

Example:

label: Job Description

optional

Indicates whether the input field is optional or required.

Usage: Determines whether the user must provide a value.

Example:

optional: true

tooltip

A tooltip text providing additional information about the input field.

Usage: Assists users by offering more context or instructions when they hover over the input field.

Example:

tooltip: 'Provide a detailed description of the job'

Expressions

Expressions allow workflow authors to programatically control the input form based on other inputs and access contexts inside of jobs.

In order for expressions to be evaluated, they must be wrapped in ${{ }}.

Contexts

The following contexts are available for use in expressions:

• inputs: Contains the values of the input fields.
• needs: Contains the jobs that the current job depends on as well as a reference to the current job.
• app: Contains the target field which is a reference to an item of compute-clusters type which is only available for apps during execution.
• sessions: Contains a field for each session described in the workflow. Each session field evaluates to the name of the session.

The workflow form can only access the inputs context. All other contexts are only available during execution, so they can only be accessed inside of jobs.

Operators

The following operators are available for use in expressions:

• ( ): Logical grouping
• &&: Logical AND
• ||: Logical OR
• !: Logical NOT
• ==: Equal to
• !=: Not equal to
• >: Greater than
• <: Less than
• >=: Greater than or equal to
• <=: Less than or equal to
• +: Addition
• -: Subtraction
• *: Multiplication
• /: Division
• ^: Exponentiation
• //: Floor division
• .: Self referencing

Use Cases

Accessing outputs

Outputs from a job can be accessed using the needs context. The following example demonstrates how to access the outputs from within the current job:

jobs:
  main:
    steps:
      - name: fake output
        run: echo PORT=3001 >> $OUTPUTS
      - name: Print output
        run: echo ${{ needs.main.outputs.PORT }}

In this example, the second step will print 3001.

Conditional Visibility

Inputs can be dynamically hidden based on the value or attributes of other inputs. This is useful for creating more dynamic and user-friendly forms where advanced options are only shown when needed.

on:
  execute:
    inputs:
      header:
        type: header
        text: Starter Local Workflow
        size: 20
      show_advanced:
        type: boolean
        label: Show Advanced Settings
        default: false
      advanced_setting:
        type: string
        label: Advanced Setting
        hidden: ${{ !inputs.show_advanced }}
      advanced_setting2:
        type: string
        label: Advanced Setting 2
        hidden: ${{ inputs.advanced_setting[hidden] }}

In this example, advanced_setting and advanced_setting2 are hidden fields until the user selects Yes from the show_advanced toggle switch. advanced_setting2 is hidden based on the hidden attribute of advanced_setting, whereas advanced_setting is hidden based on the value of show_advanced.

Dynamic Default Values

Default values can be calculated based on other input values.

on:
  execute:
    inputs:
      header:
        type: header
        text: Starter Local Workflow
        size: 20
      base_value:
        type: number
        label: Base Value
        default: 10
      calculated_value:
        type: number
        label: Calculated Value
        hidden: ${{ inputs.base_value * 2 }}

In this example, the calculated_value input field has a default value that is twice the value of the base_value input field.