2 Quick Unix Review
We are going to take 15 or 20 minutes to do a whirlwind review of some crucial Unix topics
2.1 What is bash?
We start by acknowledging that there are many different flavors of Unix and Linux. I will refer to them all simply as Unix or unix.
Also, there are a number of different shells for Unix. The shell is the part that interprets commands.
We will be talking about the bash shell. This is the default shell on Sedna, and it is also the most popular shell for bioinformatics.
Bash stands for “Bourne-again shell”. It is an update to an earlier shell called the Bourne shell.
2.2 Setting up our workspace
- I have prepared a repository with a few different example data files that we be using.
- It also contains all these notes.
- I want everyone to download it to their home directory and then
cdinto itsplaygrounddirectory, where we will be playing today and tomorrow.
After logging onto Sedna:
Paste this into your shell
cd ~
git clone https://github.com/eriqande/nmfs-bioinf-2022.git
cd nmfs-bioinf-2022/playgroundWhen we say “Paste this into your shell” or “Type this at your command prompt” we also implicitly mean “Hit RETURN afterward.”
- This is where our working directory will be for the next two days.
- Use the
treeutility to see the files that we have to play with within this playground:
Type this command at your prompt
treeThe data directory has a few things that we will be using for examples.
cd: change directoriesgit: run git subcommands, likeclonewith this. In the above case it clones the repository that is found at the GitHub URL.tree: Super cool “text-graphical” directory listing
2.3 A motivating example
- The
data/samtools_statsdirectory has gzipped output from running thesamtools statsprogram on 30 different samples. - This provides information about reads that have been mapped to a reference genome in a BAM file.
To see what those files look like:
Paste this into your terminal
gzip -cd data/samtools_stats/s001_stats.tsv.gz | less- Hit the SPACE-bar to go down a screenful, the
bkey to go back up - Most terminal emulators let you use
up-arrowanddown-arrowto go one line at a time, too. - Hit the
qkey to quit out of thelessviewer.
To see it without lines wrapping all over the place try this:
Paste this into your terminal
gzip -cd data/samtools_stats/s001_stats.tsv.gz | less -S- Now you can use the left and right arrows to see different parts of lines that are not wrapped on the screen.
gzip -cd: decompress gzipped file to stdout (this is whatzcatdoes, butzcatis not portable).less: page view. Great to pipe output into. (SPACE-bar,b,q,down-arrow,up-arrow)less -S: option to not wrap lines. (left-arrow,right-arrow)
2.3.1 (One of) Our Missions…
It is pretty typical that Bioinformatic outputs will be spread as small bits of information across multiple files.
One motivating example is summarizing the total number of reads mapped, the number of properly paired reads, and the total number of bases mapped, in all 30 samples, in a table.
2.4 The anatomy of a Unix command
Nearly every line in a bash script, or every line you type when banging away at the Unix terminal is a command that has this structure:
command options arguments
- The
commandis the name of the command itself (likecdorless). - The options are often given:
- with a dash plus a single character, like
-lor-Sor-a,-v,-z.- In most commands that are part of Unix, these single options can be combined, so
-cd, is the same as-c -d.
- In most commands that are part of Unix, these single options can be combined, so
- with two dashes and a word, like
--longor--version - Sometimes options take arguments, like
--cores 20, but sometimes, they stand alone.- When they stand alone they are sometimes called flags.
- with a dash plus a single character, like
- The arguments are typically file or paths.
Identify the command, options, and arguments in:
tree -d ..
# and
gzip -cd data/samtools_stats/s001_stats.tsv.gzFirst case:
treeis the command-dis the option (print only directory names, not files)..is the argument (one directory level up)
Second case:
gzipis the command- The options are
-cand-d, contracted into-cd data/samtools_stats/s001_stats.tsv.gzis the argument
2.4.1 What do all these options mean?
Everything you need to know about any Unix command will typically be found with the man command. For example:
type this at your terminal
man tree- That gives you more information than you will ever want to know.
- It starts with a synopsis of the syntax, which can feel very intimidating.
Man uses the
lessviewer for presenting contents of the man pages.When you are viewing
manpages, you can scroll down with SPACE-bar and up withb, and get out withq, just like inlessTo search for patterns in the manual pages, you can type
/then the string you want and then RETURN.- When in pattern-searching mode, use
nto go to the next occurrence, andNto the previous. - If searching for a single letter option try searching with
[, ]afterward. - For example, to search for the
-dflag you would type/, then-d[, ], then hit RETURN. Try it on thetreeman page.
- When in pattern-searching mode, use
- Learn about
gzip- Use man to read information about the
gzipcommand - Find information about the
-cand the-doptions.- Maybe even search for those using the “slash-pattern” bonus tip from above.
- Use man to read information about the
- Learn about the
lscommand- Use
manto see information about thelscommand, which lists directories and their contents - Find out what the
-Roption does. Maybe even look for it using the Bonus Tip above. - Do the same for the
-Qoption. - Look at what those do by doing
ls -RQon your terminal.
- Use
man gzip- To search for
-c, type/-c+ return. You might have better results with/-c[, ].- use
norNto go forward or backward through the occurrences of-c.
- use
- You would do
man ls - To search for
-Rin the man pages, a good way to do it would be to type/-Q+ RETURN, or maybe/-Q[, ]+ RETURN.
- You would do
2.5 Streams and redirection
- When you’ve executed the unix commands above, they have typically responded by writing text or data to the terminal screen.
- The command is actually writing to a stream that is called
stdout, which is short for “standard output.” - It turns out that, by default, the
stdoutstream gets written to the terminal.
Aha! But here is where it gets fun:
- You can redirect the
stdoutstream to a file by using>or>>after the command, options, and arguments.
For example:
Paste this into your terminal
mkdir outputs
tree -d .. > outputs/repo-tree.txtNow, you can use the less viewer to see what got into the file outputs/repo-tree.txt:
Type this at the terminal
less outputs/repo-tree.txtAha! Instead of writing the output to the screen, it just puts it in the file outputs/repo-tree.txt, as we told it to.
If you redirect stdout into a file that already exists, the contents of that file will get erased!!!
For example, if you now do:
Paste this into the shell
echo "New content coming through..." > outputs/repo-tree.txtThen you will no longer have the output of the tree command in the file outputs/repo-tree.txt. Check it out with the less command.
If you want to merely append stdout to an existing file, you can use >>. For example:
Paste this into your terminal
echo "Add this line" >> outputs/repo-tree.txt
echo "And then add another line" >> outputs/repo-tree.txtmkdir: make a new directory- (check out the
-poption, which means “make any necessary parent directories and don’t complain if the directory already exists.”)
- (check out the
echo: print the argument (usually a string) tostdout.
2.6 Pipes: redirecting into another Unix command
As we have said, many Unix utilities take files as their arguments, and they operate on the contents of that file. They can also receive input from streams, and almost all Unix utilities are set up to accept input from the stream called stdin, which is short for standard input.
The most important way to pass the
stdinstream to a Unix command is by piping thestdoutfrom one command in as thestdinto the next command.This uses the
|which is called the “pipe”.
We have already used the pipe when we did:
gzip -cd data/samtools_stats/s001_stats.tsv.gz | lessPipe syntax is pretty simple:
command1 | command2means pipe the stdout output of command1 in as stdin input for command2.
2.7 stderr: The stream Unix uses to yell at you
- If a Unix command fails, typically the program/command will bark at you to tell you why it failed. This can be very useful.
- The stream it writes this information to is called
stderr, which is short for standard error. - Some bioinformatics programs write progess and log output to
stderr, in addition to actual error messages.
If you are running a program non-interactively, it is extremely valuable and important to redirect stderr to a file, so you can come back later to see what went wrong, if your job failed.
stderris redirected with2>.- Think of the 2 as meaning that
stderris the second-most important stream, afterstdout.
As you might imagine, you could redirect stdout by using 1> instead of >, since stdout is stream #1.
For example, try this:
Paste this into your shell
tree -d --not-a-real-option .. > outputs/repo-tree.txt- Aha! We get a warning note printed on the screen,
- Because,
stderrgets printed to the terminal by default. - Also
outputs/repo-tree.txthas been overwritten and is now a file with nothing in it.
So, try this:
Paste this into your shell
tree -d --not-a-real-option .. > outputs/repo-tree.txt 2>outputs/error.txtNow, look at the contents of both outputs/error.txt and outputs/repo-tree.txt:
Paste this into your shell
head outputs/repo-tree.txt outputs/error.txt> path/to/file: redirectstdoutto file atpath/to/file. This overwrites any file already atpath/to/file.>> path/to/file: redirectstdoutto append to file atpath/to/file. Ifpath/to/filedoes not exist, it creates it and then adds the contents ofstdoutto it.2> path/to/file: redirectstderrto the file atpath/to/file.|: the uber-useful Unix pipe. (Just as an aside, when R finally got a similar construct—the%>%from the ‘magrittr’ package—it became much easier for Unixy people to enjoy coding in R).head: print the first ten lines of a file tostdout. If multiple file arguments are given, they are separated by little==> filename <==lines, which is super convenient if you want to look at the top of a lot of files.head -n XX: print the first XX lines (instead of 10).
- Decompress
data/samtools_stats/s001_stats.tsv.gzontostdoutusing thegzip -cdcommand and pipe the output intowcto count how many lines words, and characters are in the file. - Do the same that you did above, but redirect the
stdoutto a fileso.txtandstderrto a filese.txtin the current working directory.
These could be done like this:
gzip -cd data/samtools_stats/s001_stats.tsv.gz | wcgzip -cd data/samtools_stats/s001_stats.tsv.gz | wc > so.txt 2>se.txt
As an interesting side note, this will only redirect stderr for the wc command into se.txt. If the first command fails, its stderr will to to the screen. Try this:
Paste this into your shell
gzip -cd --bogus-option data/samtools_stats/s001_stats.tsv.gz | wc > so.txt 2>se.txtAn interesting fact is that you can redirect stderr from the first command before the pipe. So, to redirect stderr for the gzip command into a file called ze.txt, we could do:
Paste this into your shell
gzip -cd --bogus-option data/samtools_stats/s001_stats.tsv.gz 2>ze.txt | wc > so.txt 2>se.txtHave a look at the contents of ze.txt.
stdout and stderr to the same place
The astute reader might note that if you redirect stdout to a file, and then redirect stderr to the same file, you might end up overwriting the contents of stdout with stderr.
If you want to redirect stdout and stderr to the same place then you first redirect stdout to a file, and then after that, you say “redirect stderr to wherever stdout has been redirected to,” by using 2>&1.
So it looks like this:
command > file.out 2>&1