Ephemeral storage for HealthOmics workflow tasks

HealthOmics provides ephemeral storage for workflow tasks using the /tmp directory. This storage is temporary and unique to each task in a workflow. HealthOmics allocates 16 GiB of ephemeral storage to each task instance by default. You can increase the amount of ephemeral storage allocated to individual tasks in your workflow definition, up to a maximum of 3,072 GiB per task. All data stored in /tmp is encrypted at rest.

Key benefits

How ephemeral storage works

When you enable ephemeral storage, HealthOmics mounts a dedicated local storage volume at /tmp for each workflow task instance. Ephemeral storage is intended for temporary files generated during task execution. Ephemeral storage volumes are always deleted when the task terminates. Data written to /tmp is not persisted, exported, or accessible to other tasks or subsequent runs.

Where workflows write temporary files

Workflows benefit from ephemeral storage when task processes direct scratch I/O to /tmp. Bioinformatics workflow languages use the /tmp directory (and the $TMP, $TMPDIR environment variables) for temporary, short-lived intermediate files during task execution. Ensure your task commands have not mapped $TMPDIR to another location.

Workflow task processes that write to /tmp automatically use ephemeral storage when enabled, requiring no workflow changes. Workflows that do not explicitly direct scratch processes to /tmp will write scratch data to the working directory on the shared file system used for run storage, although tools used might take advantage of /tmp on ephemeral storage.

Encryption at rest

All ephemeral storage is encrypted at rest using a service-managed AWS KMS key. Accelerated computing instances are hardware-encrypted with a unique per-volume key that is destroyed when the instance terminates.

Permissions

HealthOmics manages the attachment and lifecycle of ephemeral storage volumes on your behalf. No additional IAM permissions are required in your task execution role.

Enabling ephemeral storage

You can redirect scratch I/O to local storage by setting scratchStorageMode in the StartRun API. The scratchStorageMode setting applies to CPU instances only and applies to all tasks in that run.

scratchStorageMode determines where your workflow writes scratch data. Possible values:

For more information, see Start a run in HealthOmics.

Opt in to ephemeral storage

To enable ephemeral storage for a run, set scratchStorageMode to LOCAL when you start the run.

aws omics start-run \
    --workflow-id workflow-id \
    --role-arn arn:aws:iam::123456789012:role/OmicsServiceRole \
    --output-uri s3://amzn-s3-demo-bucket/output-folder/ \
    --parameters file:///path/to/parameters.json \
    --scratch-storage-mode LOCAL

For batch runs, pass scratchStorageMode in defaultRunSetting. The setting applies to every run in the batch.

aws omics start-run-batch \
    --batch-name "my-batch" \
    --default-run-setting '{
      "workflowId": "workflow-id",
      "roleArn": "arn:aws:iam::123456789012:role/OmicsServiceRole",
      "outputUri": "s3://amzn-s3-demo-bucket/output-folder/",
      "storageType": "DYNAMIC",
      "parameters": {"referenceUri": "s3://amzn-s3-demo-bucket/reference.fasta"},
      "scratchStorageMode": "LOCAL"
    }' \
    --batch-run-settings '{
      "inlineSettings": [
        {
          "runSettingId": "sample-A",
          "parameters": {"inputUri": "s3://amzn-s3-demo-bucket/sampleA.fastq"}
        },
        {
          "runSettingId": "sample-B",
          "parameters": {"inputUri": "s3://amzn-s3-demo-bucket/sampleB.fastq"}
        }
      ]
    }'

Opt out of ephemeral storage

To disable ephemeral storage for a specific run (for example, to isolate a failure), set scratchStorageMode to SHARED.

aws omics start-run \
     --workflow-id workflow-id \
     --role-arn arn:aws:iam::123456789012:role/OmicsServiceRole \
     --output-uri s3://amzn-s3-demo-bucket/output-folder/ \
     --scratch-storage-mode SHARED

When scratchStorageMode is SHARED, all disk and equivalent directives in the workflow definition are ignored and /tmp is backed by the shared filesystem. This setting applies to CPU instances only. GPU tasks always use local NVMe ephemeral storage and cannot be opted out.

Check the effective mode

Ephemeral storage is off by default. When scratchStorageMode is omitted from a StartRun request, scratchStorageMode is set to SHARED (default).

scratchStorageMode is only returned in the GetRun response if it was explicitly passed in the StartRun request. SHARED is the default value if omitted. Call GetRun to confirm the effective storage mode for a run.

aws omics get-run --id run-id

Default storage allocation

The default ephemeral storage allocation is 16 GiB per task for all Standard, Compute- and Memory-optimized instance types. You do not need to specify a disk directive to receive this default. To configure additional storage, use the disk directive. You can increase the amount of ephemeral storage allocated to individual tasks in your workflow definition, up to a maximum of 3,072 GiB per task. You are billed for storage above the default 16 GiB.

Ephemeral storage can be configured in increments of 16 GiB. For more information, see Supported sizes.

Ephemeral storage for accelerated computing instances

GPU instance storage capacity is fixed per instance type and is provided at no additional cost.

GPU tasks always use local NVMe ephemeral storage. The scratchStorageMode setting on StartRun does not apply to GPU tasks and setting this value to SHARED will have no effect on GPU instances. Capacity is fixed per instance type and cannot be customized using the disk directive. The NVMe capacity is pre-determined by the instance type selected for the task's cpu, memory, and acceleratorType requirements.

Configuring ephemeral storage size

When scratchStorageMode is set to LOCAL, you can request increased per-task ephemeral storage using the disk directive (or equivalent) in your workflow definition. HealthOmics treats the disk directive as a hint and provisions a volume rounded to the next 16 GiB. The use of the disk directive does not affect instance type selection. Instance type is selected based solely on cpu, memory, and acceleratorType. For more information, see Task resources in a HealthOmics workflow definition.

If no disk directive is present, the task receives the default 16 GiB. Tasks cannot have less than the default ephemeral storage.

You do not need to size your ephemeral storage for pulled container images, which are accounted for separately. For more information, see Container images for private workflows.

When to use the disk directive

Use the disk directive in your task definition when the default ephemeral storage for your chosen instance type is not sufficient for your task requirements. For example, when a task writes large volumes of data to /tmp.

Common use cases for increased ephemeral storage

Directive syntax by engine

The following table shows the equivalent directive for each workflow language.

The following examples show how to configure a task that requests 700 GiB of ephemeral storage. HealthOmics rounds this up to the 704 GiB tier.

WDL

task sort_bam {
    runtime {
        cpu:  16
        disks: "700 GiB"
    }
    command <<<
        samtools sort -T /tmp/sort_buffer ~{input_bam} -o ~{output_bam}
    >>>
}

Nextflow

process sort_bam {
    disk '700 GB'
    script:
    """
    samtools sort -T /tmp/sort_buffer ${input} -o ${output}
    """
}

CWL

requirements:
  ResourceRequirement:
    tmpdirMin: 716800   # 700 GiB expressed in MiB

For more information on supported directive syntax, see WDL workflow definition specifics and Nextflow workflow definition specifics.

Common bioinformatics tools and ephemeral storage

Many bioinformatics tools write large temporary files during execution. When scratchStorageMode is set to LOCAL, redirect these tools to use /tmp so that scratch I/O goes to the fast local volume instead of the shared run filesystem. The following examples show the relevant flags for commonly used tools.

WDL

task sort_bam {
    runtime {
        cpu:   16
        disks: "700 GiB"
    }
    command <<<
        # samtools: -T sets the temp-file prefix
        samtools sort -T /tmp/sort_buffer ~{input_bam} -o ~{sorted_bam}

        # GATK / Picard: --TMP_DIR flag (older Picard uses TMP_DIR=/tmp)
        gatk MarkDuplicates -I ~{sorted_bam} -O ~{output_bam} --TMP_DIR /tmp

        # STAR: --outTmpDir (path must not pre-exist; STAR creates it)
        STAR --runThreadN 16 --readFilesIn ~{reads} --outTmpDir /tmp/star_tmp --outFileNamePrefix out_

        # bcftools sort: -T / --temp-dir
        bcftools sort -T /tmp ~{vcf} -o ~{output_vcf}

        # GNU sort: -T / --temporary-directory
        sort -T /tmp ~{big_table} -o ~{sorted_table}
    >>>
}
# HealthOmics rounds 700 GiB up to the 704 GiB tier

Nextflow

process sort_bam {
    disk '700 GB'
    script:
    """
    # samtools: -T sets the temp-file prefix
    samtools sort -T /tmp/sort_buffer input.bam -o sorted.bam

    # GATK / Picard: --TMP_DIR flag (older Picard uses TMP_DIR=/tmp)
    gatk MarkDuplicates -I sorted.bam -O dedup.bam --TMP_DIR /tmp

    # STAR: --outTmpDir (path must not pre-exist; STAR creates it)
    STAR --runThreadN 16 --readFilesIn reads.fastq --outTmpDir /tmp/star_tmp --outFileNamePrefix out_

    # bcftools sort: -T / --temp-dir
    bcftools sort -T /tmp input.vcf -o sorted.vcf

    # GNU sort: -T / --temporary-directory
    sort -T /tmp big_table.tsv -o sorted_table.tsv
    """
}
// HealthOmics rounds 700 GB up to the 704 GiB tier

CWL

class: CommandLineTool
cwlVersion: v1.2
requirements:
  ResourceRequirement:
    coresMin: 16
    tmpdirMin: 716800          # 700 GiB expressed in MiB
baseCommand: [bash, -c]
arguments:
  - |
    set -euo pipefail

    # samtools: -T sets the temp-file prefix
    samtools sort -T /tmp/sort_buffer input.bam -o sorted.bam

    # GATK / Picard: --TMP_DIR flag (older Picard uses TMP_DIR=/tmp)
    gatk MarkDuplicates -I sorted.bam -O dedup.bam --TMP_DIR /tmp

    # STAR: --outTmpDir (path must not pre-exist; STAR creates it)
    STAR --runThreadN 16 --readFilesIn reads.fastq --outTmpDir /tmp/star_tmp --outFileNamePrefix out_

    # bcftools sort: -T / --temp-dir
    bcftools sort -T /tmp input.vcf -o sorted.vcf

    # GNU sort: -T / --temporary-directory
    sort -T /tmp big_table.tsv -o sorted_table.tsv

Supported sizes

Requested sizes are rounded up to the nearest 16 GiB increment, starting from the default of 16 GiB (16, 32, 48, 64, ... up to 3,072 GiB). The maximum supported size is 3,072 GiB per task.

If a requested size exceeds 3,072 GiB, HealthOmics provisions 3,072 GiB and writes a warning to the run log. The task is not automatically failed.

Supported WDL disks forms

For the full list of accepted WDL disks forms, see Supported WDL disks forms.

Using the Nextflow scratch directive

For Nextflow workflows, you can use the scratch directive to control where processes write temporary working files. For information about supported values and recommended usage with ephemeral storage, see Using scratch storage efficiently in Nextflow.

Monitoring ephemeral storage

HealthOmics writes per-task ephemeral storage metrics to CloudWatch manifest logs. Metrics include per-task ephemeral storage data of provisioned volume size (as scratchStorageReservedGiB) and usage (as scratchStorageUtilizedGiB) for each task. Review the manifest log to determine whether tasks were over- or under-provisioned without querying CloudWatch directly. For details on manifest logs, see Monitoring HealthOmics with CloudWatch Logs.

How ephemeral storage is billed

You are billed only for the ephemeral storage provisioned above the default allocation. Requests above the default in your disk directive are rounded up to the nearest supported tier.

GPU instances have ephemeral storage already accounted for in instance pricing. There is no additional charge for ephemeral storage on GPU tasks.

Considerations and limitations

Troubleshooting ephemeral storage

Task fails with exhausted ephemeral storage

A task fails when ephemeral storage reaches capacity. Review your CloudWatch manifest logs to determine how much storage your task actually used, then add or increase the disk directive to request a larger tier.

# Before: 4-vCPU task using the 16 GiB default
runtime { cpu: 4 }

# After: explicitly request 400 GiB
runtime { cpu: 4, disks: "400 GiB" }

Ephemeral storage does not appear to be utilized

Call GetRun and check the scratchStorageMode field. If the value is SHARED, ephemeral storage is not enabled for that run. Set --scratch-storage-mode LOCAL on your next start-run call.

aws omics get-run --id run-id