6  Sedna and SLURM intro

6.1 Why do we need SLURM?

  • The fundamental problem of cluster computing.
  • A cluster does not operate like your laptop.
  • Most compute-intensive jobs run most efficiently on a dedicated processor or processors.

6.2 HPCC Architecture in a Nutshell

  • Nodes: the closest thing to what you think of as a computer. (“Pizza-boxes” with no displays).
    • Each node is attached via a fast connection to centralized attached storage (A big set of hard drives attached via “Infiniband.”)
    • Within each node are some numbers of Cores or CPUs
      • Cores/CPUs are the actual processing units within a node. (Usually 20 to 24)
  • Sedna, like almost all other HPCCs has a login node
    • The login node is dedicated to allowing people to communicate with the HPCC.
    • DO NOT do computationally intensive, or input/output-intensive jobs on the login node
    • Not surprisingly, when you login to Sedna you are on the login node.
Hot tip!

In the default configuration on Sedna, your command prompt at the shell tells you which node you are logged into. The default command prompt looks like:

[username@computer directory]

So, for example, mine at the moment is:

[eanderson@sedna playground]$

Which tells me that I am user eanderson and I am logged in to sedna in the directory whose basename is playground.

The login node for Sedna is named sedna. So, this is telling me that I am logged into the login node of Sedna.

If you don’t have such an informative prompt…

On Alpine, I am not sure what the default command prompt is, but you can always change yours by setting the PS1 variable. Try doing:

export PS1='[\h: \W]--% '

That will give you a command prompt with the hostname (\h) and the working directory basename (\W), which is helpful. Mine, right now, looks like:

[login11: con-gen-csu]--%

Of course, you can put the export PS1='[\h: \W]--% ' into your ~/.bashrc file and then always have an informative prompt.

Self study

Investigate your own command prompt and make sure you understand what the different parts in it mean.

6.3 Say it again: Don’t do heavy computing on the login nodes!

If you run a big job that is computationally intensive on the login node, it can interfere with other people being able to access the cluster.

Don’t do it!!

That is all fine and well, but how do we avoid computing on the login nodes?

6.4 Do your computation on the compute nodes

  • To run jobs on the compute node you need to ask SLURM to give you “compute resources” to run the job.
  • The basic axes of “compute resources” are:
    • The number of cores to use.
    • The maximum amount of RAM memory you might need.
    • The amount of time that you will need those resources for.
  • To prepare for the next part of the course on SEDNA, each one of us is going to “check out” 2 cores for 3 hours for interactive work.
    • Interactive here means that we will have a Unix shell that has access to compute power on the cores that we checked out.

    • Here is the command to checkout 2 cores for 3 hours for interactive use:

    Paste this into your shell on SEDNA
    srun -c 2 -t 03:00:00 --pty /bin/bash
  • If you are working on ALPINE then you can get on a compute node differently:
    • Load the slurm/alpine module
    • Request a shell on the oversubscribed atesting partition.
    • The commands for that look like:
    Paste this into your shell on ALPINE
    module load slurm/alpine
    srun --partition atesting --pty /bin/bash
Self study:

Have a look at your command prompt now. If you are on SEDNA, it should show that you are logged in to nodeXX where XX is a number like 04 or 33. If you are on Alpine, then the node you are on will be named something more cryptic, like c3cpu-a5-u34-3.

6.5 What does Sedna have under the hood?

Sedna is not an outrageously large cluster, but it is well-maintained and provides a lot of computing power.

The nodes in Sedna are broken into three different partitions.

A partition is a collection of nodes that tend to be similar.

  • node partition: This is a collection of 36 nodes:
    • node01 – node28: “standard compute nodes”
    • node29 – node36: “standard compute nodes with twice as much memory”
  • himem partition: Four nodes with a boatload of memory for high-memory jobs
    • himem01 – himem04
  • bionode partition: legacy machines from an older NWFSC cluster
    • I’m not sure if all of us have access to this.
    • When I check, it seems like a lot of the nodes are down, occasionally.
    • But I have been able to checkout resources on it.
    • Might be an option, but it doesn’t perform at the same level as the node partition

6.6 What does ALPINE have under the hood?

Alpine is a pretty darn large cluster, with a boatload of different machines, many of them with 64 cores. It also has some GPU machines, etc.

We will have a look at it with the commands, like sinfo described next.

6.7 Ask SLURM about what nodes are available, and how busy they are

When I login to Sedna, before launching and jobs or working on anything, I always like to get a summary of how hard Sedna is currently working.

For this we have the sinfo command:

Type this at your command prompt
sinfo

The answer will look somethign like this:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
node*        up   infinite      5    mix node[01,29-31,33]
node*        up   infinite      1  alloc node32
node*        up   infinite     30   idle node[02-28,34-36]
himem        up   infinite      1    mix himem01
himem        up   infinite      3   idle himem[02-04]
bionode      up   infinite      8  down* bionode[10-11,13-18]
bionode      up   infinite      1  drain bionode12
bionode      up   infinite     10   idle bionode[01-09,19]

This is a hugely informative summary for being so compact.

  • The output is sorted by partition.
  • Within each partition it tells us how many nodes are in different states:
  • The main states:
    • idle: just sitting there with all cores available for checkout,
    • mix: some, but not all, cores are checkout out and in use,
    • alloc: all the cores on the node are checkout out and in use.
  • Other states you might see:
    • down: node is unavailable because it is not working properly,
    • drain: node is not available because the sys-admins are not letting any new jobs to start because they are going to be working on the system soon, etc.
  • The node partition is starred node* because it is the default partition.
  • Notation like node[01,29-31,33] gives the specific node numbers, compactly.

If you type sinfo on ALPINE you get a similar output, but it is much larger and more complex because there are more partitions and more, different nodes in each partition.

The CURC lists the partitions on a web page here.

If you are going to be working on Alpine, it is worth reading over the entire CURC documentation for it, starting from here.

Self-study

Review how many nodes are currently in use.

6.8 More detailed sinfo output

Sometimes you want more information about what is going on with the cluster. sinfo can do that, too, but the syntax is hard to remember and even harder to type.

Here is a bash function we will define called slinfo for (s-long-info) that uses sinfo to provide more detailed output.

Paste this into your shell
function slinfo {
  sinfo -N -O nodelist,cpusstate,memory,allocmem
}

Once you have done that:

Type this at your command prompt
slinfo

This gives information about each node.

  • In CPUS(A/I/O/T):
    • A = allocated
    • I = idle
    • O = offline
    • T = total
  • MEMORY is total RAM available on the node (in Megabytes)
  • ALLOCMEM is the total RAM allocated to jobs.

This view shows each node just once. But it might be useful to also see what partition(s) each of those nodes are in (especially on ALPINE). So, here is a quick function to do that:

Paste this into your shell
function slinfop {
  sinfo -N -O nodelist,partition,cpusstate,memory,allocmem
}

Once you have done that:

Type this at your command prompt
slinfop

Note that many nodes are included in multiple partitions.

Since the main partition on Alpine for general computing is amilan you can look specifically at nodes in that partition with:

Copy this into your command prompt
slinfop | awk '$2~/amilan/'
ALPINE Self-study

Quickly scan the output of the above command and estimate the number of idle (i.e. available) cores in the amilan partition.

Then, read about the amilan partition (here)[https://curc.readthedocs.io/en/stable/clusters/alpine/alpine-hardware.html#partitions]. In particular, note how much memory is available per core, and the default and max wall times.

We will discuss this.

6.9 Summarize what the cluster is doing by job

SLURM keeps track of individual jobs that get submitted.

For SLURM, each job is basically a request for resources.

If SLURM finds the requested resources are available, it provides the resources and starts the job.

If resources are not available because other jobs are running, the reqested job enters the “queue” in a WAITING state.

We can see how many jobs are running and how many are waiting, by using the SLURM squeue command:

(By the way, notice the pattern? All SLURM command start with an s).

Type this at your command prompt
squeue

This tells us a little about all the jobs that are currently allocated resources or are waiting for resources.

One thing to note: - Every job is assigned a SLURM_JOB_ID. Which is a unique integer (that gets assigned successively). - And jobs might have a job NAME, that is assigned by the user, which is the name of the script that the job runs, or is defined in the SBATCH directives (we will talk about that later!)

6.10 I find the standard, default, squeue output sorely lacking

The standard way that squeue presents information is not super informative, especially if the job names are not very short, or if you are interested in how many cores (not just nodes) each job involves.

Once again, we make a bash function, alljobs, that runs squeue but provides more information.

Paste this at your command prompt
function alljobs {
  if [ $# -ne 1 ]; then
    JL=10;
  else
    JL=$1;
  fi;
  squeue -o  "%.12i %.9P %.${JL}j %.15u %.2t %.15M %.6D %.18R %.5C %.12m %.12l %.12L";
}

Also, if you follow the command with an integer like 20, that is how many spaces the output will allow for the job NAME.

For example, try:

Type this at the command prompt
alljobs 30

This is kind of fun to see the full names of the jobs.

Hot Tip!

It is worth taking special note of the columns, NODELIST(REASON) and CPUS:

  • NODELIST(REASON): If the job is running, then this gives the name(s) of the node(s) upon which it is running. If it is not running, it says why it is not yet running. The most typical reason it:
    • Priority: Too many other users are already using the cluster. Your job is waiting for resources to become available.
  • CPUS: the number of CPUs the job is using

It is worth noting that, when I look at alljobs on Alpine, most of the jobs that are waiting due to Priority are jobs that are requesting all or most of the CPUs (like 64 or 32) on a single node. Many of the jobs that are not waiting are those that are requesting only one or a few cores.

In general, if you can break your workflows down into a small chunks that run on just one or a few cores, you have a better chance of getting compute time, than if you request all the cores on a node.

6.11 Just tell me about my jobs!

We can make a similar function that only tells us about our own jobs that are running or waiting in the queue:

Paste this into your terminal
function myjobs {
    if [ $# -ne 1 ]; then
      JL=10;
  else
      JL=$1;
  fi;
    squeue -u $(whoami) -o  "%.12i %.9P %.${JL}j %.10u %.2t %.15M %.6D %.18R %.5C %.12m %.12l %.12L %p";
}

You can run that the same way as alljobs, and you should see the 2-core job you started with srun:

Type this into your shell
myjobs

6.12 Finding Software on a Cluster

  • Analysis software is not automatically available on a cluster
  • On many clusters you have to install it yourself
  • On Sedna, we are very lucky to have a great support crew that will install software for us, and also make it available for everyone else to use.

However, software can be complicated:

  • Some software might conflict with other software.
  • Some users might want different software versions

The solution: maintain software (and its dependencies) in (somewhat) isolated modules.

Bioinformatic modules on Sedna are only available if the MODULEPATH is specified

Here we check to see if the bioinformatics paths are in the MODULEPATH:

Paste this in your shell
echo $MODULEPATH | grep bioinformatics

If that command did not return anything to stdout, then you need to add a command to your ~/.bashrc that will add /opt/bioinformatics/modulefiles to your MODULEPATH. You can do that (as detailed in the Sedna Google Doc) like this:

If you need to, paste this into the shell
echo 'export MODULEPATH=${MODULEPATH}:/opt/bioinformatics/modulefiles' >> ~/.bashrc

And when you are done with that, reload your .bashrc to get the change to take effect:

If you need to, paste this into the shell
source ~/.bashrc

Most modules on Alpine are only available if you are on a compute node.

6.13 Example: Let’s try running samtools

samtools is a common bioinformatic utility.

Typically, if you run it with no arguments, it gives you some usage information.

However, on Sedna, unless you have installed samtools yourself, or have set your environment up to always have it available, when you type samtools at the shell, you will get a response saying that it is not available:

Type this at the command prompt
samtools

The response I get from the computer is:

bash: samtools: command not found...

You will likely get that response, too.

On Alpine, you could install samtools via mamba or you could see if it is available in the system modules.

6.14 Check which software programs are available in the modules

The command module and its subcommands let you interact with the software modules.

module avail tells you what software is available in the modules.

Type this into your shell
module avail

On SEDNA, there is a lot of tasty bioinformatics software there. On Alpine, there is now a dedicated “Bioinformatics” section in the output of module avail, which is a nice change from Alpine’s predecessor, SUMMIT, which was mostly loaded up with software for geoscientists.

The numbers are the version numbers. The (D) after some of them tells us that version is the default.

6.15 Load the samtools module

We will load the module for version 1.15.1 of samtools. That can be done like this:

SEDNA

On SEDNA, Paste this into your shell
module load bio/samtools/1.15.1

Alpine

On Alpine, Paste this into your shell
module load samtools

(using the load subcommand of module)

About the default versions

The same would have been achieved with

module load bio/samtools

because 1.15.1 is the default samtools version.

Nonetheless, it is typically best from a reproducibility standpoint to be explicit about version numbers of software you are using.

Now, that it is loaded, you can run the samtools command and get the usage message.

Type this at your shell
samtools
Inventory and Unload modules

To know which modules you have loaded:

module list

To unload all the modules you have loaded

module purge

There are many more module commands, but those are the only ones that you really need to know.

6.16 Two notes about modules

Loaded modules are only active in the current session

If you open a new shell, or allocate an interactive session on a new core, or logout and log back in again…

…You will have to reload the modules that you need.

So: Any time you write a script that needs some software, you should load the module for that software in the initial lines of the script.

Modules are not always portable

Not every cluster that you use will have the same modules set up in the same way. (Though it does seem that Giles has set the modules up on Sedna according to the accepted best practices!).

So, if you use code you developed for Sedna (or Alpine) on another cluster, you might have to modify things to ensure that software is available.

The conda/mamba package manager is an option if you are going to be running your scripts on different clusters.

6.17 If you need more software or newer versions on Sedna

You can request software installations by using the Sedna work request form

6.18 If you need more software or newer versions on Alpine

You probably can request it from the sys admins, but you will almost indubitably have your required software up and running faster if you install it with mamba.

6.19 Before we go, make our convenience functions available whenever we are on our cluster

We defined four convenience functions in bash: slinfo, slinfop, alljobs, and myjobs.

Once we log out of the current session, or login to a different session, those convenience functions will be lost.

To make sure that they are accessible the next time we login, we will put them into our own .bashrc files.

The .bashrc file is a file in your home directory in which you can put function definitions and other configurations that will be applied whenever you open a new bash shell.

To add our four convenience functions to your .bashrc file, first start editing your ~/.bashrc file with nano.

Paste this into your shell
nano ~/.bashrc

Then come back to this web page and copy the following text:

Copy this onto your clipboard
function slinfo {
  sinfo -N -O nodelist,cpusstate,memory,allocmem
}

function alljobs {
    if [ $# -ne 1 ]; then
      JL=10;
  else
      JL=$1;
  fi;
    squeue -o  "%.12i %.9P %.${JL}j %.15u %.2t %.15M %.6D %.18R %.5C %.12m %.12l %.12L";
}

function myjobs {
    if [ $# -ne 1 ]; then
      JL=10;
  else
      JL=$1;
  fi;
    squeue -u $(whoami) -o  "%.12i %.9P %.${JL}j %.10u %.2t %.15M %.6D %.18R %.5C %.12m %.12l %.12L %p";
}

Then paste those function definitions into the editor in your .bashrc file (before your conda/mamba block if you have one), and then do cntrl-X, Y, RETURN to save the file and get out of nano.

Now, the next time you login you will have all those functions.

6.20 Bonus Thoughts — What are other people doing?

Especially when you are working on a smaller workgroup cluster and you are wondering if other users are using it efficiently, there are a few ways you can check up on what they are doing.

Using alljobs you can see what other people are doing. Then you can ssh to the node that they are running jobs on, and then do ps -auxww | grep user (where user is their username) and that will show the currently running command lines. This can be useful, for example, to see if their jobs are multithreaded while they are using many cores or not.

You can also get information about failures on different nodes with dmesg -T.