2  Quick Unix Review

We are going to take 15 or 20 minutes to do a whirlwind review of some crucial Unix topics

2.1 What is bash?

We start by acknowledging that there are many different flavors of Unix and Linux. I will refer to them all simply as Unix or unix.

Also, there are a number of different shells for Unix. The shell is the part that interprets commands.

We will be talking about the bash shell. This is the default shell on Sedna, and it is also the most popular shell for bioinformatics.

Bash stands for “Bourne-again shell”. It is an update to an earlier shell called the Bourne shell.

2.2 Setting up our workspace

  • I have prepared a repository with a few different example data files that we be using.
  • It also contains all these notes.
  • I want everyone to download it to their home directory and then cd into its playground directory, where we will be playing today and tomorrow.

After logging onto Sedna:

Paste this into your shell
cd ~
git clone https://github.com/eriqande/nmfs-bioinf-2022.git
cd nmfs-bioinf-2022/playground
Important

When we say “Paste this into your shell” or “Type this at your command prompt” we also implicitly mean “Hit RETURN afterward.”

  • This is where our working directory will be for the next two days.
  • Use the tree utility to see the files that we have to play with within this playground:
Type this command at your prompt
tree

The data directory has a few things that we will be using for examples.

  • cd: change directories
  • git: run git subcommands, like clone with this. In the above case it clones the repository that is found at the GitHub URL.
  • tree: Super cool “text-graphical” directory listing

2.3 A motivating example

  • The data/samtools_stats directory has gzipped output from running the samtools stats program on 30 different samples.
  • This provides information about reads that have been mapped to a reference genome in a BAM file.

To see what those files look like:

Paste this into your terminal
gzip -cd data/samtools_stats/s001_stats.tsv.gz | less
  • Hit the SPACE-bar to go down a screenful, the b key to go back up
  • Most terminal emulators let you use up-arrow and down-arrow to go one line at a time, too.
  • Hit the q key to quit out of the less viewer.

To see it without lines wrapping all over the place try this:

Paste this into your terminal
gzip -cd data/samtools_stats/s001_stats.tsv.gz | less -S
  • Now you can use the left and right arrows to see different parts of lines that are not wrapped on the screen.
  • gzip -cd: decompress gzipped file to stdout (this is what zcat does, but zcat is not portable).
  • less: page view. Great to pipe output into. (SPACE-bar, b, q, down-arrow, up-arrow)
    • less -S: option to not wrap lines. (left-arrow, right-arrow)

2.3.1 (One of) Our Missions…

It is pretty typical that Bioinformatic outputs will be spread as small bits of information across multiple files.

One motivating example is summarizing the total number of reads mapped, the number of properly paired reads, and the total number of bases mapped, in all 30 samples, in a table.

2.4 The anatomy of a Unix command

Nearly every line in a bash script, or every line you type when banging away at the Unix terminal is a command that has this structure:

command options arguments

  • The command is the name of the command itself (like cd or less).
  • The options are often given:
    • with a dash plus a single character, like -l or -S or -a, -v, -z.
      • In most commands that are part of Unix, these single options can be combined, so -cd, is the same as -c -d.
    • with two dashes and a word, like --long or --version
    • Sometimes options take arguments, like --cores 20, but sometimes, they stand alone.
      • When they stand alone they are sometimes called flags.
  • The arguments are typically file or paths.
Self-study question

Identify the command, options, and arguments in:

tree -d ..

# and

gzip -cd data/samtools_stats/s001_stats.tsv.gz

First case:

  • tree is the command
  • -d is the option (print only directory names, not files)
  • .. is the argument (one directory level up)

Second case:

  • gzip is the command
  • The options are -c and -d, contracted into -cd
  • data/samtools_stats/s001_stats.tsv.gz is the argument

2.4.1 What do all these options mean?

Everything you need to know about any Unix command will typically be found with the man command. For example:

type this at your terminal
man tree
  • That gives you more information than you will ever want to know.
  • It starts with a synopsis of the syntax, which can feel very intimidating.
  • Man uses the less viewer for presenting contents of the man pages.

  • When you are viewing man pages, you can scroll down with SPACE-bar and up with b, and get out with q, just like in less

  • To search for patterns in the manual pages, you can type / then the string you want and then RETURN.

    • When in pattern-searching mode, use n to go to the next occurrence, and N to the previous.
    • If searching for a single letter option try searching with [, ] afterward.
    • For example, to search for the -d flag you would type /, then -d[, ], then hit RETURN. Try it on the tree man page.
Self-study questions
  1. Learn about gzip
    • Use man to read information about the gzip command
    • Find information about the -c and the -d options.
      • Maybe even search for those using the “slash-pattern” bonus tip from above.
  2. Learn about the ls command
    • Use man to see information about the ls command, which lists directories and their contents
    • Find out what the -R option does. Maybe even look for it using the Bonus Tip above.
    • Do the same for the -Q option.
    • Look at what those do by doing ls -RQ on your terminal.
    • man gzip
    • To search for -c, type /-c + return. You might have better results with /-c[, ].
      • use n or N to go forward or backward through the occurrences of -c.
    • You would do man ls
    • To search for -R in the man pages, a good way to do it would be to type /-Q + RETURN, or maybe /-Q[, ] + RETURN.

2.5 Streams and redirection

  • When you’ve executed the unix commands above, they have typically responded by writing text or data to the terminal screen.
  • The command is actually writing to a stream that is called stdout, which is short for “standard output.”
  • It turns out that, by default, the stdout stream gets written to the terminal.

Aha! But here is where it gets fun:

  • You can redirect the stdout stream to a file by using > or >> after the command, options, and arguments.

For example:

Paste this into your terminal
mkdir outputs
tree -d .. > outputs/repo-tree.txt

Now, you can use the less viewer to see what got into the file outputs/repo-tree.txt:

Type this at the terminal
less outputs/repo-tree.txt

Aha! Instead of writing the output to the screen, it just puts it in the file outputs/repo-tree.txt, as we told it to.

Danger!

If you redirect stdout into a file that already exists, the contents of that file will get erased!!!

For example, if you now do:

Paste this into the shell
echo "New content coming through..." > outputs/repo-tree.txt

Then you will no longer have the output of the tree command in the file outputs/repo-tree.txt. Check it out with the less command.

If you want to merely append stdout to an existing file, you can use >>. For example:

Paste this into your terminal
echo "Add this line" >> outputs/repo-tree.txt
echo "And then add another line" >> outputs/repo-tree.txt
  • mkdir: make a new directory
    • (check out the -p option, which means “make any necessary parent directories and don’t complain if the directory already exists.”)
  • echo: print the argument (usually a string) to stdout.

2.6 Pipes: redirecting into another Unix command

As we have said, many Unix utilities take files as their arguments, and they operate on the contents of that file. They can also receive input from streams, and almost all Unix utilities are set up to accept input from the stream called stdin, which is short for standard input.

  • The most important way to pass the stdin stream to a Unix command is by piping the stdout from one command in as the stdin to the next command.

  • This uses the | which is called the “pipe”.

We have already used the pipe when we did:

gzip -cd data/samtools_stats/s001_stats.tsv.gz | less

Pipe syntax is pretty simple:

command1 | command2

means pipe the stdout output of command1 in as stdin input for command2.

2.7 stderr: The stream Unix uses to yell at you

  • If a Unix command fails, typically the program/command will bark at you to tell you why it failed. This can be very useful.
  • The stream it writes this information to is called stderr, which is short for standard error.
  • Some bioinformatics programs write progess and log output to stderr, in addition to actual error messages.

If you are running a program non-interactively, it is extremely valuable and important to redirect stderr to a file, so you can come back later to see what went wrong, if your job failed.

  • stderr is redirected with 2>.
  • Think of the 2 as meaning that stderr is the second-most important stream, after stdout.

As you might imagine, you could redirect stdout by using 1> instead of >, since stdout is stream #1.

For example, try this:

Paste this into your shell
tree -d --not-a-real-option .. > outputs/repo-tree.txt
  • Aha! We get a warning note printed on the screen,
  • Because, stderr gets printed to the terminal by default.
  • Also outputs/repo-tree.txt has been overwritten and is now a file with nothing in it.

So, try this:

Paste this into your shell
tree -d --not-a-real-option .. > outputs/repo-tree.txt 2>outputs/error.txt

Now, look at the contents of both outputs/error.txt and outputs/repo-tree.txt:

Paste this into your shell
head outputs/repo-tree.txt outputs/error.txt
  • > path/to/file: redirect stdout to file at path/to/file. This overwrites any file already at path/to/file.
  • >> path/to/file: redirect stdout to append to file at path/to/file. If path/to/file does not exist, it creates it and then adds the contents of stdout to it.
  • 2> path/to/file: redirect stderr to the file at path/to/file.
  • |: the uber-useful Unix pipe. (Just as an aside, when R finally got a similar construct—the %>% from the ‘magrittr’ package—it became much easier for Unixy people to enjoy coding in R).
  • head: print the first ten lines of a file to stdout. If multiple file arguments are given, they are separated by little ==> filename <== lines, which is super convenient if you want to look at the top of a lot of files.
    • head -n XX: print the first XX lines (instead of 10).
Self-study questions
  1. Decompress data/samtools_stats/s001_stats.tsv.gz onto stdout using the gzip -cd command and pipe the output into wc to count how many lines words, and characters are in the file.
  2. Do the same that you did above, but redirect the stdout to a file so.txt and stderr to a file se.txt in the current working directory.

These could be done like this:

  1. gzip -cd data/samtools_stats/s001_stats.tsv.gz | wc
  2. gzip -cd data/samtools_stats/s001_stats.tsv.gz | wc > so.txt 2>se.txt

As an interesting side note, this will only redirect stderr for the wc command into se.txt. If the first command fails, its stderr will to to the screen. Try this:

Paste this into your shell
gzip -cd --bogus-option data/samtools_stats/s001_stats.tsv.gz | wc > so.txt 2>se.txt

An interesting fact is that you can redirect stderr from the first command before the pipe. So, to redirect stderr for the gzip command into a file called ze.txt, we could do:

Paste this into your shell
gzip -cd --bogus-option data/samtools_stats/s001_stats.tsv.gz 2>ze.txt | wc > so.txt 2>se.txt

Have a look at the contents of ze.txt.

The astute reader might note that if you redirect stdout to a file, and then redirect stderr to the same file, you might end up overwriting the contents of stdout with stderr.

If you want to redirect stdout and stderr to the same place then you first redirect stdout to a file, and then after that, you say “redirect stderr to wherever stdout has been redirected to,” by using 2>&1.

So it looks like this:

command > file.out 2>&1