The sbatch command takes an --array=<array_spec> option that let’s you quickly launch—and easily manage—different instances of a single job that are all differentiated by a unique integer.
That integer is accessible within the script as the environment variable SLURM_ARRAY_TASK_ID.
We will see a little shell/awk script that is very helpful in this context.
8.1 Prepare for this session
Get on your cluster and navigate to the top level (con-gen-csu) of your fork of the class repository. Then, to make sure that you have all the latest updates from the repository, sync the main branch of your fork with the main branch of eriqande/con-gen-csu on GitHub, then in your shell, use git switch main to make sure you are on the main branch of your fork, and then use git pull origin main to pull down those changes.
8.2sbatch’s --array option
When you give sbatch the --array=1-10 option, say, then it runs your job 10 separate times, each time with the environment variable SLURM_ARRAY_TASK_ID set to a different number between 1 and 10.
Let’s see a quick example, using the following script:
Contents of scripts/array_example.sh
#!/bin/bash#SBATCH --time=00:05:00#SBATCH --output=my_output_%A_%a#SBATCH --error=my_error_%A_%a#SBATCH --array=1-10ODIR=results/array_examplemkdir -p $ODIR( echo "The SLURM_ARRAY_JOB_ID is : $SLURM_ARRAY_JOB_ID" echo "The SLURM_ARRAY_TASK_ID is: $SLURM_ARRAY_TASK_ID" echo "The SLURM_JOB_ID is: $SLURM_JOB_ID" echo echo "You can refer to this individual SLURM array task as: ${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}" echo) >$ODIR/output_$SLURM_ARRAY_TASK_ID.txtsleep 20
This is helpful to see a few of the variables that are defined in the environment of an array job:
SLURM_ARRAY_JOB_ID: the overall JOB_ID for the whole array
SLURM_ARRAY_TASK_ID: the index that runs from 1 to 10 in this case
SLURM_JOB_ID: The underlying job_id
One can run this script, but we don’t want a whole classroom full of people throwing all these jobs on the cluster. So, get into groups of three or four, talk amongst yourselves about what you think this script will do, and then have just one person from each group launch the job with the following command:
One person from each group, paste this into your shell
sbatch scripts/array_example.sh
And then use myjobs and alljobs to watch what is happening on the cluster.
When that job is done, or when it is running, look at the values written to the first output files:
Paste this into you shell
head results/array_example/output_{1..10}.txt
Cool syntax interlude: {1..10}
On the shell, if you do something like:
{2..7}: that will expand to 2 3 4 5 6 7
{a..g}: that will expand to a b c d e f g
{0001..0015}: that will expand to 0001 0002 0003 0004 0005 0006 0007 0008 0009 0010 0011 0012 0013 0014 0015
{F..M}: that will expand to F G H I J K L M
From looking at the array_example output files, we can infer that:
SLURM_ARRAY_JOB_ID: is a unique number that refers to the entire set of jobs in the array
SLURM_ARRAY_TASK_ID: is the integer that is being cycled over in the array job
SLURM_JOB_ID: is a unique SLURM_JOB_ID of the specific array task.
Also, in alljobs and myjobs you see that jobs can be referred to like 331989_4. That is SLURM_ARRAY_JOB_ID + underscore + SLURM_ARRAY_TASK_ID. For example: scancel 376578_12.
In the slurm output files specified like:
SBATCH--output=my_output_%A_%a
%A expands to SLURM_ARRAY_JOB_ID
%a expands to SLURM_ARRAY_TASK_ID
If you need to cancel a particular array task using scancel you would typically use SLURM_ARRAY_JOB_ID + underscore + SLURM_ARRAY_TASK_ID, unless you wanted to cancel all remaining instances of an array task, in which case you would use
scancel SLURM_ARRAY_JOB_ID
8.3 Variations on the <array_spec>
There are some important variations to how you can specify those array numbers:
--array=1-50: simple, consecutive numbers
--array=1-10:3: 1 through 10 by threes, (so 1,4,7,10)
--array=1,2,3,6,9,15 non-consecutive numbers
--array=1-21:10,100-200:50: non-consecutive ranges. It becomes (1,11,21,100,150,200)
--array=1-10,4: WARNING, this becomes 1,2,3,4,5,6,7,8,9,10,4. SLURM does not check that the array numbers are unique, so task array 4 would be run twice (possibly concurrently overwriting the output.)
--array=1-20%5 VERY IMPORTANT SYNTAX: Run the jobs, but don’t ever have more the 5 running at a time. This is useful for making sure your jobs don’t consume every last CPU on Sedna.
Let’s try putting all these together. Read the following command and, talking among the members of your group, figure out what the array spec is doing. Then have one person from each group submit the job with the following command:
One person from each group, aste this into your shell
Then use myjobs and alljobs to see what is going on in the cluster.
Do you ever have more than 5 jobs running?
8.4 Translating Array Indexes to Job Instances
The user is left to translate what a job array index of, say, 7, means in terms of what actions that array task should take.
Quite often you will want to map an array index to a different file to analyze, or perhaps a different region of a chromosome to do variant calling on, etc.
A flexible and generic way of doing this mapping from array indexes to job specifics is to first define the variables (things like filenames, etc.) required by each array task in a simple TAB-delimited text file in which the first row holds the names of the variables in different TAB-separated columns, and each row below that holds the values that those variables should take for different values of the array index.
The array index itself should be listed in the first column, whose name is index.
8.4.1 An example TAB-delimited file
Let’s explore the following file, which is at data/sample-array-info.tsv:
An easier way to view this might be to look at in on GitHub at: here
Something that would be really handy would be a little shell script that would pick out a particular line of that file that corresponds to the value in the index column and then define some shell variables according to the columns names: index, sample, library, flowcell, platform, lane, barcode, fq1, fq2 so that they could be used in an array script.
We have already seen some of the bash and awk machinery that would make that possible, and we have wrapped it up in the line-assign.sh script.
8.4.2 The line-assign.sh script
Within the repository is a script called scripts/line-assign.sh, that looks like this:
Contents of scripts/line-assign.sh
#!/bin/bash# simple script. Arguments are:# 1. The index to pick out (like 1, or 2, or 7, etc)# 2. Path to a TAB delimited file with the first column named index# and subsequent columns named valid shell variable names.## This script will pick out the line that matches $1, and return a command# line assigning the column header names as shell variables whose values are the# values in each cell in the file $2.## Note: this is not written with a whole lot of error checking or catching.#if[$#-ne 2 ];thenecho"Wrong number of arguments in $0 "> /dev/stderrfiawk-F"\t"-v LINE=$1' $1 == "index" {for(i=1; i<=NF; i++) vars[i]=$i; next} $1 == LINE {for(i=1; i<=NF; i++) printf("%s=\"%s\"; ", vars[i], $i); printf("\n");}'$2
Whoa! It returns a command line that assigns values to a lot of shell variables.
So, if we wanted to run that command line, we would have to precede it with the eval keyword (because the command line itself includes special characters like =). Let’s do that like this:
Holy Smokes! These are variables that we could use in a job array script.
In this case, we can assign values to the pesky Read Group string for bwa mem.
8.5 Putting it all together: a read-mapping job array
We can now elaborate on our simple bwa_map.sh shell script and turn that into a SLURM job array script. The key here is that when SLURM runs the script as a slurm array, there will be an environment variable called SLURM_ARRAY_TASK_ID that we can use to pick out the desired sample from the data/sample-array-info.tsv file.
Here is what it looks like:
Contents of scripts/bwa_map_array.sh
#!/bin/bash#SBATCH --cpus-per-task=1#SBATCH --time=1:00:00#SBATCH --output=bwa_map_array-%A_%a.out#SBATCH --error=bwa_map_array-%A_%a.err#SBATCH --array=1-16# load the module that gives us the bwa softwaremodule load bwamodule load samtools# make a directory for log output if it does not already existLDIR=results/log/bwa_map_arrayODIR=results/mappedmkdir-p$LDIR$ODIR# get shell variables for this array task:COMM=$(./scripts/line-assign.sh$SLURM_ARRAY_TASK_ID data/sample-array-info.tsv)eval$COMM# run bwa mem on the input and pipe it to samtools to sort itbwa mem \-R"@RG\tID:${sample}.${library}.${flowcell}.${lane}\tSM:${sample}\tLB:${library}\tBC:${barcode}\tPU:\tID:${sample}.${library}.${flowcell}.${lane}.${barcode}\tPL:ILLUMINA"\ data/genome/genome.fasta \$fq1$fq22>$LDIR/bwa_mem_$sample.log |\(samtools view -u-|\samtools sort -o$ODIR/$sample.bam)2>$LDIR/samtools_$sample.log
The main differences from our simple bwa_map.sh script are:
The SLURM_ARRAY_TASK_ID is used to pick out the right line from our TSV file of information
The slurm output and error have been changed to use %A_%a notation.
There is a #SBATCH --array=1-16 directive, to run it over the 16 different files.
The shell code that runs bwa and samtools uses the variables that were defined by eval-ing the results of the call to the line-assign.sh script.
You can launch that job array and see how it goes:
Paste this into your shell
sbatch scripts/bwa_map_array.sh
Then check with myjobs and alljobs to see what is happening on Sedna.
That runs really fast.
Afterward you can check that the results exist:
Paste this into your shell
ls-l results/mapped
And you can look at the logs for bwa mem:
Paste this into your shell
tail results/log/bwa_map_array/bwa_mem_*
To see what one of the output files looks like, you can use samtools. On Alpine, you have to get onto a compute node to do that, so you would first do:
Use the left and right arrows, and space bar and backspace to see all parts of the file.
Remember to hit q to get out of the less viewer.
8.6 Wrap Up
So, that was a quick tour of the capabilities of sbatch.
Oddly enough, I haven’t directly launched any jobs using sbatch (apart from the example jobs for this course) in many months, because I drive all my bioinformatics projects using Snakemake, which sends my jobs to SLURM for me.
Eventually we will have an introduction to Snakemake!