13  Important Snakemake Embellishments

Thus far we have only scratched the surface of what Snakemake can do, and it could take a lifetime to realize the full potential of Snakemake as a workflow management system. At this point, we will point out that there is comprehensive documentation about how to use snakemake at https://snakemake.readthedocs.io/en/stable/. It is worth giving that a good-faith read-through at some point. However, it is not always clear which of the many gems in that documentation will be the most valuable for your bioinoformatic life.

Here we present a few embellishments upon what we have already learned that will really let you get a lot more out of Snakemake. There are three broad categories that will be discussed in different sections of this chapter.

  1. Embellishments critical for running Snakemake on a cluster. This includes the declaration of resources required for the jobs and also the use of profiles to create a snakemake command line that can send jobs to SLURM and also communicate with SLURM to check up on jobs while they are running. It also includes telling snakemake to generate benchmark information to record how much memory and time was required for every job in your workflow—this is super handy for predicting resource requirements for future data sets!

  2. Embellishments that make it easy to run your workflow on a different set of data, or a different species, or using different settings, options, or parameter values for some of the tasks/rules. The main improvement here is using snakemake config files, so that all the different settings and options for your workflow can be specified and stored and reviewed in one place. We will also see that configuration files can be very helpful for setting resource requirements for rules, which means that you only have to change things in one small config file to run your workflow on a different cluster with different partition names and machine capabilities, for example.

  3. More advanced processing of wildcards using snakemake input functions. This allows your rules to react to the wildcard values in very general ways, rather than just by substituting the values into input or log file paths.

13.1 Running Snakemake on a Cluster: resources, profiles, and benchmarks

13.1.1 Specify the resources required for each rule

Recall from our discussion of SLURM that the main axes upon which you will need to think about allocating computing resources for your bioinformatic jobs are:

  • Memory: what is the maximum amount of RAM the job will need while running?
  • Time: How long must the job run?
  • Cores: how many CPUs do you want to allocate to the job?

An additional resource that Snakemake might be concerned about is tmpdir the location of a directory for writing temporary files. Typically these files get written to a directory called /tmp on the hard drive that is local to the node that the job is running on. However, at least last summer, we found that we were often exceeding the amount of space that we were offered in /tmp on Alpine, so we will show you how to instruct snakemake to write temporary files into a directory in the results directory on scratch.

Within your Snakefile, you can specify the memory and/or time resources for each rule in a resources block for that rule. For example, if we wanted 6 GB of memory and 10 hours for our map_reads rule, then we would add a resources block so that it looks like:

rule map_reads:
  input:
    r1="results/trimmed/{sample}_R1.fastq.gz",
    r2="results/trimmed/{sample}_R2.fastq.gz",
    genome="resources/genome.fasta",
    idx=multiext("resources/genome.fasta", ".0123", ".amb", ".ann", ".bwt.2bit.64", ".pac")
  output:
    "results/bam/{sample}.bam"
  conda:
    "envs/bwa2sam.yaml"
  resources:
    mem_mb=6000,
    time="08:00:00"
  log:
    "results/logs/map_reads/{sample}.log"
  params:
    RG="-R '@RG\\tID:{sample}\\tSM:{sample}\\tPL:ILLUMINA' "
  shell:
    " (bwa-mem2 mem {params.RG} {input.genome} {input.r1} {input.r2} | "
    " samtools view -u | "
    " samtools sort - > {output}) 2> {log} "

In the above, we have named and specified our resources the way SLURM would.

Additionally, if for most of our rules we simply wanted a default time of 4 hours and a default amount of 3.74 Gb of memory, we don’t have to write a resources block for each of those rules. Rather, we could just specify that on the command line with the --default-resources option, like this:

--default-resources  mem_mb=3.74  time="04:00:00"

That is handy if most of your rules require the same resources.

We have not yet discussed how to request a certain number of cores/CPUs for each job of a rule. Remember, there is no point of requesting more cores/CPUs than the number of threads that can be used by the programs running the job. Consequently, you want the number of cores/CPUs to typically be set to the number of threads that the program(s) in your job are using. As a consequence, CPUs are typically set according to the threads specifier in the Snakemake rule. We haven’t talked about this yet, but it is simple—you set an integer value in a threads block, then you can use {threads} in the shell block to tell the program how many threads to use.

We will again illustrate this with our map_reads rule. bwa-mem2 mem uses the -t option to specify the number of threads to use, so, to make bwa-mem2 mem use 4 threads we could write:

rule map_reads:
  input:
    r1="results/trimmed/{sample}_R1.fastq.gz",
    r2="results/trimmed/{sample}_R2.fastq.gz",
    genome="resources/genome.fasta",
    idx=multiext("resources/genome.fasta", ".0123", ".amb", ".ann", ".bwt.2bit.64", ".pac")
  output:
    "results/bam/{sample}.bam"
  conda:
    "envs/bwa2sam.yaml"
  threads: 4
  resources:
    mem_mb=3740 * 4,
    time="08:00:00"
  log:
    "results/logs/map_reads/{sample}.log"
  params:
    RG="-R '@RG\\tID:{sample}\\tSM:{sample}\\tPL:ILLUMINA' "
  shell:
    " (bwa-mem2 mem -t {threads} {params.RG} {input.genome} {input.r1} {input.r2} | "
    " samtools view -u | "
    " samtools sort - > {output}) 2> {log} "

Note we have specified 4 threads, and we also bumped up the memory, because if we are asking for 4 cores, we might as well get the default amount of memory (3.74 Gb, on Alpine) that we would get with each core.

All of that is pretty straightforward; however, doing this does not automatically cause the rule to be dispatched by SLURM and granted those resources. Rather, Snakemake must be told to send jobs to SLURM and also to use the resources and threads to appropriately request the right amount of resources from SLURM. Snakemake versions < 8.0 had an older “officially sanctioned/supported” way of sending jobs to SLURM that can be found here. However, that approach is quite complex—it is implemented in a bunch of python scripts that I find hard to read, and I find it difficult to understand what all these scripts are doing.

Snakemake version 8.0 brought big changes, and snakemake versions >= 8.0 include an --executor option that uses snakemake-executor-plugins. These offer a relatively painless way of running Snakemake with SLURM. One of the advantages of this is that you should be able to set your workflow up to run in SLURM with the SLURM snakemake executor plugin, and then, with very few changes to your workflow you should be able to also run it using Google Batch or Amazon’s Cloud compute infrastructure using a different snakemake executor plugin. That has some big advantges, but it comes at the cost of flexibility. For example, when running the official Snakemake >= 8.0 SLURM executor plugin, the job names are completely meaningless and not useful at all.

By contrast, a different approach to having Snakemake send jobs to SLURM is implemented in the wonderfully readable and easy-to-understand approach taken by John D. Blischak, (find it here on GitHub) John was a postdoc with Matthew Stephens and has worked extensively in improving research reproducibility. His method is implemented almost entirely by using command line options (plus one little script to help check the status of jobs in the SLURM queue). However, to avoid having to type all these options at the command line every time you launch snakemake to use SLURM, these command line options can be saved and read from a snakemake profile. We discuss profiles in the next section and show how they can be used to allow snakemake to submit jobs to SLURM. This works on both Snakemake versions < 8.0 and >= 8.0, but the profiles are slightly different.

13.1.2 Snakemake profiles

We begin this section by quickly perusing all the different command line options that can be used when launching snakemake. If we type snakemake --help we get a listing of all the options. It starts with a synopsis of all the options, which is followed by longer explanations of each individual option. The synopsis looks like this:

usage: snakemake [-h] [--dry-run] [--profile PROFILE] [--workflow-profile WORKFLOW_PROFILE] [--cache [RULE ...]] [--snakefile FILE] [--cores N]
                 [--jobs N] [--local-cores N] [--resources NAME=INT [NAME=INT ...]] [--set-threads RULE=THREADS [RULE=THREADS ...]]
                 [--max-threads MAX_THREADS] [--set-resources RULE:RESOURCE=VALUE [RULE:RESOURCE=VALUE ...]]
                 [--set-scatter NAME=SCATTERITEMS [NAME=SCATTERITEMS ...]] [--set-resource-scopes RESOURCE=[global|local] [RESOURCE=[global|local]
                 ...]] [--default-resources [NAME=INT ...]] [--preemptible-rules [PREEMPTIBLE_RULES ...]]
                 [--preemptible-retries PREEMPTIBLE_RETRIES] [--config [KEY=VALUE ...]] [--configfile FILE [FILE ...]]
                 [--envvars VARNAME [VARNAME ...]] [--directory DIR] [--touch] [--keep-going]
                 [--rerun-triggers {code,input,mtime,params,software-env} [{code,input,mtime,params,software-env} ...]] [--force]
                 [--executor {local,dryrun,touch}] [--forceall] [--forcerun [TARGET ...]] [--prioritize TARGET [TARGET ...]]
                 [--batch RULE=BATCH/BATCHES] [--until TARGET [TARGET ...]] [--omit-from TARGET [TARGET ...]] [--rerun-incomplete]
                 [--shadow-prefix DIR] [--scheduler [{ilp,greedy}]] [--wms-monitor [WMS_MONITOR]] [--wms-monitor-arg [NAME=VALUE ...]]
                 [--scheduler-ilp-solver {PULP_CBC_CMD,COIN_CMD}] [--scheduler-solver-path SCHEDULER_SOLVER_PATH]
                 [--conda-base-path CONDA_BASE_PATH] [--no-subworkflows] [--precommand PRECOMMAND] [--groups GROUPS [GROUPS ...]]
                 [--group-components GROUP_COMPONENTS [GROUP_COMPONENTS ...]] [--report [FILE]] [--report-stylesheet CSSFILE] [--reporter PLUGIN]
                 [--draft-notebook TARGET] [--edit-notebook TARGET] [--notebook-listen IP:PORT] [--lint [{text,json}]]
                 [--generate-unit-tests [TESTPATH]] [--containerize] [--export-cwl FILE] [--list-rules] [--list-target-rules] [--dag] [--rulegraph]
                 [--filegraph] [--d3dag] [--summary] [--detailed-summary] [--archive FILE] [--cleanup-metadata FILE [FILE ...]] [--cleanup-shadow]
                 [--skip-script-cleanup] [--unlock] [--list-changes {params,code,input}] [--list-input-changes] [--list-params-changes]
                 [--list-untracked] [--delete-all-output | --delete-temp-output] [--keep-incomplete] [--drop-metadata]
                 [--deploy-sources QUERY CHECKSUM] [--version] [--printshellcmds] [--debug-dag] [--nocolor] [--quiet [{all,progress,rules} ...]]
                 [--print-compilation] [--verbose] [--force-use-threads] [--allow-ambiguity] [--nolock] [--ignore-incomplete]
                 [--max-inventory-time SECONDS] [--latency-wait SECONDS] [--wait-for-files [FILE ...]] [--wait-for-files-file FILE]
                 [--queue-input-wait-time SECONDS] [--notemp] [--all-temp] [--unneeded-temp-files FILE [FILE ...]] [--keep-storage-local-copies]
                 [--target-files-omit-workdir-adjustment] [--allowed-rules ALLOWED_RULES [ALLOWED_RULES ...]]
                 [--target-jobs TARGET_JOBS [TARGET_JOBS ...]] [--local-groupid LOCAL_GROUPID] [--max-jobs-per-second MAX_JOBS_PER_SECOND]
                 [--max-status-checks-per-second MAX_STATUS_CHECKS_PER_SECOND] [--seconds-between-status-checks SECONDS_BETWEEN_STATUS_CHECKS]
                 [--retries RETRIES] [--attempt ATTEMPT] [--wrapper-prefix WRAPPER_PREFIX] [--default-storage-provider DEFAULT_STORAGE_PROVIDER]
                 [--default-storage-prefix DEFAULT_STORAGE_PREFIX] [--local-storage-prefix LOCAL_STORAGE_PREFIX]
                 [--shared-fs-usage {input-output,persistence,software-deployment,source-cache,sources,storage-local-copies,none} [{input-output,persistence,software-deployment,source-cache,sources,storage-local-copies,none} ...]]
                 [--scheduler-greediness SCHEDULER_GREEDINESS] [--no-hooks] [--debug] [--runtime-profile FILE] [--mode {default,remote,subprocess}]
                 [--show-failed-logs] [--log-handler-script FILE] [--log-service {none,slack,wms}] [--job-deploy-sources] [--container-image IMAGE]
                 [--immediate-submit] [--jobscript SCRIPT] [--jobname NAME] [--flux]
                 [--software-deployment-method {apptainer,conda,env-modules} [{apptainer,conda,env-modules} ...]] [--container-cleanup-images]
                 [--use-conda] [--conda-not-block-search-path-envvars] [--list-conda-envs] [--conda-prefix DIR] [--conda-cleanup-envs]
                 [--conda-cleanup-pkgs [{tarballs,cache}]] [--conda-create-envs-only] [--conda-frontend {conda,mamba}] [--use-apptainer]
                 [--apptainer-prefix DIR] [--apptainer-args ARGS] [--use-envmodules] [--report-html-path VALUE]
                 [--report-html-stylesheet-path VALUE]
                 [targets ...]

In this synopsis, command line options that are truly optional (which is basically all of them!) are wrapped in [ ]. You will notice that some of these options, like --dry-run (look at the top line of the synopsis) do not take any arguments (there is nothing after --dry-run in the [ ]), while other options, like --profile take an argument, in the above listing, of PROFILE. Clearly if you use even a small fraction of these optional command-line options to snakemake, the stuff you type on the command line could grow long, typing it in on the command line could become quite ponderous, and the chance to make a mistake would grow, the longer the command line got. Not only that, but if you were just putting these options on the command line, it would be easy to forget exactly which options you used!

This is where snakemake profiles come in. A snakemake profile is a collection of files in a directory, one of which is named config.yaml. So, if I have directory ./my-profile and within that a file ./my-profile/config.yaml, then, I can store the command line options I want to use in YAML format in ./my-profile/config.yaml and those options will be added to the command line if I invoke snakemake with:

snakemake --profile ./my-profile

Note that it is good practice to add the . in front of your profile name, because otherwise Snakemake might search for system-wide or user-wide profiles.

Profiles for versions < 8.0 and >= 8.0

The changes introduced in Snakemake version 8.0 were large enough that older profiles might not run on versions >= 8.0 and profiles meant for versions >= 8.0 might not run on versions < 8.0. To handle this, Snakemake’s developers allow for there to be two config files withing the profile directory. In our repo, for example:

Snakemake-Example/slurm/jdb
├── alpine
│   ├── config.v8+.yaml
│   ├── config.yaml
│   └── status-sacct-robust.sh
└── sedna
    ├── config.v8+.yaml
    ├── config.yaml
    └── status-sacct-robust.sh

There is a config.v8+.yaml and a config.yaml in these profile directories for alpine and sedna. Snakemake Version >= 8.0 will preferentially use the config.v8+.yaml file, and versions of Snakemake < 8.0 will use config.yaml because they don’t even recognize config.v8+.yaml.

So, how do you write command line options in the YAML-formatted Snakemake profile config.yaml? It is easy:

  • You can specify any command line option that is preceded by two dashes, like --option, but you cannot specify options that you might specify with a single dash, like -n or -p (or -np as we often type the two together). This is OK, because every single-dash option has a two-dash name. For example:

    • -n is the same as --dry-run
    • -p is the same as --printshellcmds
    • -v is the same as --version and so forth.
  • If the option does not take an argument, for example --dry-run or --printshellcmds, then you enter them into the config.yaml file by writing the option name, without the dashes, following it with a colon and then a space and then True, like:

    dry-run: True
    printshellcmds: True
  • If the option does take an argument, for example --cores 20 or --rerun-triggers mtime, then you just include the argument after the colon (and a space), like:

    cores: 20
    rerun-triggers: mtime

And that is all there is to it!

13.1.3 An Alpine Slurm Profile for Snakemake

If you are using Snakemake >= 8.0, you have to do one step to get the JDB’s simple-slurm-smk config to work:

pip install snakemake-executor-plugin-cluster-generic

Do not do that if you are using Snakemake versions < 8.0.

We have a profile for sending jobs to Alpine in the directory Snakemake-Example/slurm/jdb/alpine. The config.v8+.yaml file within that looks like this:

Snakemake-Example/slurm/jdb/alpine/config.v8+.yaml

executor: cluster-generic
cluster-generic-submit-cmd:
  mkdir -p results/slurm_logs/{rule} &&
  sbatch
    --partition=amilan,csu
    --cpus-per-task={threads}
    --mem={resources.mem_mb}
    --time={resources.time}
    --job-name=smk-{rule}-{wildcards}
    --output=results/slurm_logs/{rule}/{rule}-%j-{wildcards}.out
    --error=results/slurm_logs/{rule}/{rule}-%j-{wildcards}.err
    --parsable
cluster-generic-status-cmd: status-sacct-robust.sh
cluster-generic-cancel-cmd: scancel
cluster-generic-cancel-nargs: 400
# warning, time here is very small.  It works for the small
# example data set, but should be increased for production jobs
default-resources:
  - time="00:30:00"
  - mem_mb=3740
  - tmpdir="results/snake-tmp"
restart-times: 0
max-jobs-per-second: 10
max-status-checks-per-second: 25
local-cores: 1
latency-wait: 60
cores: 2400
jobs: 950
keep-going: True
rerun-incomplete: True
printshellcmds: True
software-deployment-method: conda
rerun-trigger: mtime

We see here that the “heavy-lifting” is done by snakemake’s --cluster-generic-submit-cmd option. This option takes as an argument a command line that should be used to send off individual jobs for execution. As you can see, the command it uses does two main things:

  1. First it creates a directory in which to place the error and output files that sbatch will write.
  2. If that first command is successful, it then runs sbatch with a variety of options.

Various values known to snakemake can be substituted into that sbatch command line by enclosing them within curly braces, thereby passing them as arguments to various sbatch options. In particular:

  • {rule} expands into the rule name
  • {threads} expands into the number of threads to be used for the rule
  • {wildcards} expands into a comma-separate string with the values of the wildcards in effect for this job, like sample=A,chromo=NC103567.1
  • {resources.mem_mb} and {resources.time} print the values given for mem_mb and time in the resources: block of the rule.

Each of these values are used as arguments to different sbatch options in a plainly readable way. The last option given is --parsable which tells sbatch to return the job number of the submitted job in an easily parsable fashion.

Snakemake takes this sbatch command and uses it to submit a job script. Snakemake itself writes that job script (and stores it in the .snakemake directory for as long as it is needed) according to the code that it finds in the shell block of the rule. This is why you never need to write another sbatch script in your life if you use snakemake—Snakemake writes those scripts for you and submits them via sbatch.

You will also notice that the profile defines the --default-resources option that we described above. Note that the default memory is 3740, which is what you get with one core on Alpine, and the default time is 8 hours.

Some of the remaining options in the profile are not really specific to SLURM, but many are. One of those,

cluster-generic-status-cmd: status-sacct-robust.sh

gives the name of a shell script in the profile directory that snakemake uses to check on the status of jobs that have been submitted to the cluster. This is a script that queries SLURM using sacct to ask about the status of running jobs according to their job number. For each job, the script tells Snakemake either:

  • success: the job finished successfully.
  • running: the job is still running or waiting in the queue
  • failed: the job failed

When your jobs are running, Snakemake is continually checking on their status, but won’t check more than max-status-checks-per-second. (In the case of this profile that is 25 checks per second.)

You can unfold the following code block to see the shell code for status-sacct-robust.sh.

Click the triangle to see the code
Snakemake-Example/slurm/alpine/status-sacct-robust.sh
#!/usr/bin/env bash

# Check status of Slurm job. More robust because runs `sacct` multiple times if
# needed

jobid="$1"

if [[ "$jobid" == Submitted ]]
then
  echo smk-simple-slurm: Invalid job ID: "$jobid" >&2
  echo smk-simple-slurm: Did you remember to add the flag --parsable to your sbatch call? >&2
  exit 1
fi

function get_status(){
  sacct -j "$1" --format State --noheader | head -n 1 | awk '{print $1}'
}

for i in {1..20}
do
  output=`get_status "$jobid"`
  if [[ ! -z $output ]]
  then
    break
  else
    sleep 3
  fi
done

if [[ -z $output ]]
then
  echo sacct failed to return the status for jobid "$jobid" >&2
  echo Maybe you need to use scontrol instead? >&2
  exit 1
fi

if [[ $output =~ ^(COMPLETED).* ]]
then
  echo success
elif [[ $output =~ ^(RUNNING|PENDING|COMPLETING|CONFIGURING|SUSPENDED).* ]]
then
  echo running
else
  echo failed
fi

Much of the code is there to make sure the script tries at least 20 times to get the status of the job, because sometimes SLURM gets busy and fails to return a result for a job, which causes snakemake to do bad things…

Other SLURM-relevant options in this profile are:

  • --cluster-generic-cancel-cmd: if the Snakemake run is terminated (with <cntrl>-c for example) then kill all jobs running under SLURM using the scancel command.
  • --cluster-generic-cancel-nargs: the maximum number of arguments to pass to scancel when cancelling jobs. This should be high enough to kill all jobs currently running or queued up in SLURM.
  • --jobs: don’t submit so many jobs that you have more than this number of jobs running or waiting in the SLURM queue. This is quite important in Alpine, because SLURM on Alpine doesn’t let you have more the 1000 jobs running or queued up.
  • --max-jobs-per-second: don’t submit more than this jobs each second to SLURM

So, in summary, if you are using snakemake and you had the alpine directory containing config.yaml and status-sacct-robust.sh in the directory you were running snakemake from, you could get snakemake to submit jobs to the cluster using SLURM by doing:

snakemake  [some options or targets here] --profile ./alpine  [other options or targets here]

Also, we will end this section by noting that you can override options in the profile by adding them actually on the command line, just like with sbatch and its #SBATCH directives.

13.1.4 Simple-slurm-smk profile for Snakemake versions < 8.0

If you are running snakemake < 8.0, you don’t have the executor option, and some of the command line options are a little shorter:

Click the triangle to see the code with changes from v8+ highlighted
Snakemake-Example/slurm/jdb/alpine/config.yaml
cluster:
  mkdir -p results/slurm_logs/{rule} &&
  sbatch
    --partition=amilan,csu
    --cpus-per-task={threads}
    --mem={resources.mem_mb}
    --time={resources.time}
    --job-name=smk-{rule}-{wildcards}
    --output=results/slurm_logs/{rule}/{rule}-%j-{wildcards}.out
    --error=results/slurm_logs/{rule}/{rule}-%j-{wildcards}.err
    --parsable
cluster-status: status-sacct-robust.sh
cluster-cancel: scancel
cluster-cancel-nargs: 400
# warning, time here is very small.  It works for the small
# example data set, but should be increased for production jobs
default-resources:
  - time="00:30:00"
  - mem_mb=3740
  - tmpdir="results/snake-tmp"
restart-times: 0
max-jobs-per-second: 10
max-status-checks-per-second: 25
local-cores: 1
latency-wait: 60
cores: 2400
jobs: 950
keep-going: True
rerun-incomplete: True
printshellcmds: True
use-conda: True
rerun-trigger: mtime

13.1.5 The officially supported slurm executor approach for Snakemake >= 8.0

One must first download a snakemake-executor-plugin like this:

pip install snakemake-executor-plugin-slurm

After that, you can use a profile that looks like this:

executor: slurm
default-resources:
  - runtime=20
  - mem_mb=3740
  - tmpdir="results/snake-tmp"
restart-times: 0
max-jobs-per-second: 10
max-status-checks-per-second: 50
local-cores: 1
latency-wait: 60
cores: 2400
jobs: 950
keep-going: True
rerun-incomplete: True
printshellcmds: True
use-conda: True
rerun-trigger: mtime

13.1.6 A note on local rules

You can tell Snakemake to execute rules locally by putting them in a localrules: directive near the top of the Snakefile, like:

localrules: all, genome_faidx, genome_dict

13.2 Snakemake config files

We have already seen that we might add some variables to our Snakefile to run a workflow on certain samples, etc.

13.2.1 YAML configuration

Helpful snippet for testing:

from snakemake.common.configfile import _load_configfile

13.2.2 Tabular configuration via pandas

13.3 Snakemake benchmarks

Compared the previous two topics, this is really simple.

Recommended convention, put them in:

results/benchmarks/<rule_name>/<first_wildcard>/.../<last_wildcard>.bmk

13.4 Some additional snakemake topics