R:
# numeric vector
x <- 1:10
# character vector
y <- c("a", "b", "c")Snakefiles are YAML formatted files which are interpreted by Python, and, as such, considerable value can be gained from knowing how to use Python in this context. Many biologists I know are quite familiar with R, but fewer are equally well-versed in Python. So the purpose of this chapter is to provide a little background on Python (from the eyes of an R user) that is most useful for embellishing one’s Snakefiles and workflows. The key goal for this chapter is to introduce several simple Python constructs that can be particularly useful in Snakemake, allowing the user to competently use several central Snakemake features: config files, input functions, and pandas input.
The second main goal of this chapter is to illustrate an approach to working interactively with Python (the way R users might be familiar with working with R) within the RStudio environment. As we will see in the next chapter, this will allow users to run Python code associated with their Snakefile in the context of an interactive Python session, which can be considerably more illuminating—and more helpful for tracking down bugs/errors/inconsistencies—than only getting to run that code by running Snakemake on your Snakefile. Being able to work interactively with the Python code as Snakemake might see it is very helpful for learning how to write workflows. The next chapter has most of the gory details for this second goal, like some little hacks for seeing input function parameters as Snakemake might see them, etc., primarily as a learning tool to aid in figuring out how to write Python code within Snakemake. In this chapter we talk about how to set up RStudio to work interactively in the python environment that snakemake uses.
The order of events here is thus:
Use the R package ‘reticulate’ to run python, interactively, in our Snakemake conda environment.
Once we can work interactively in python easily, we will discuss a few of the main data structures and operations in python by comparing to their equivalents in R and the tidyverse, talk about defining functions, and take a brief tour of python’s ‘pandas’ package, which is similar in its functionality to R’s tidyverse.
With that background we will tackle our exploration of Snakemake’s use of
The folks at Posit (the company that started as RStudio) make a great integrated development environment (IDE) for working with R; however they also recognized the need for being a great IDE for other languages, like python and julia. As I understand it, that was part of the reason that they changed their name to Posit—to be less tied to the R language. At any rate, they have tools for running python code, much like you would run R code in RStudio.
You can do this on your laptop, rather than upon the cluster; however, if you have been working on Alpine, you should skip the rest of this section and get an RStudio session on Alpine following the directions in Chapter 1. Returning to here to get the reticulate package and proceed.
If you are doing this on your laptop, fortunately, mamba works on all platforms, and so does Snakemake. But you might need to first install mamba on your laptop. This is pretty straightforward. If you do not already have mamba on your laptop, find the instructions for your platform here and follow them. You should be able to just use the default location for your conda/mamba library.
Once you are done with that, you will want to create a snakemake environment on your laptop. Do that by following the directions that we used on our clusters, here, without bothering to be on a compute node, of course, since you are on your laptop!
When both of the above are done, we can move on to working with python in RStudio.
The reticulate package allows you to run python in a comfortable, interactive way within an R (and RStudio) session. Importantly, it allows you to run the exact python version that Snakemake uses by letting you load the snakemake conda environment.
So, Sync your fork, then open up the RStudio project for your fork of the con-gen-csu repo, make sure you are on the main branch, and then pull, to make sure that you have all the recent materials from the course.
If you do not already have the ‘reticulate’, R-package, get it with this command at the R console in Rstudio. If you are Alpine you must do this:
# first do this
install.packages("reticulate")
# then do this:
install.packages("reticulate", repos = "https://cran.rstudio.com/")Note that, on Alpine, it is critical to use that second line where you add the repos = "https://cran.rstudio.com/ part in there. The RStudio Server maintained by CURC in OpenOnDemand is configured by default to draw its packages from a repository full of binary versions, and it may end up givng you versions of packages which are much older than the most recent versions. (At the time of this writing, it installed ‘reticulate’ version 1.28 instead of 1.35, and all the following steps failed…I lost a good several hours because of that!!) We do the initial install.packages("reticulate") to install binary versions of some of the dependencies for the ‘reticulate’ package.
If you are not on Alpine, you should be able to simply do:
install.packages("reticulate")to get the latest version of reticulate.
Then, you can load the package with:
library(reticulate)Now, we want to use python from a particular conda environment—our snakemake conda environment, in fact. For that there is a reticulate function called use_condaenv(). We can tell it which conda environment to use most reliably by passing the absolute path of the environment to the function. This path will be the path to your conda library plus envs plus the name of the environment. For example on my laptop that would translate into:
use_condaenv(condaenv = "~/mambaforge-arm64/envs/snakemake-8.5.3")However, if you are on Alpine, your conda library would probably be in /project/user@colostate.edu/miniforge.
While on my RStudio Server session on Alpine I would do:
use_condaenv(condaenv = "/projects/eriq@colostate.edu/miniforge3/envs/snakemake-8.5.3")At any rate, it should be easy to find the path to your snakemake environment using tab-completion for path names in your R console.
Once you have successfully given the above command, you can start a python REPL (Read Evaluate Print and Loop) session in your R console with:
repl_python()After you have done that, you should see something that looks like this:
> repl_python()
Python 3.12.2 (/Users/eriq/mambaforge-arm64/envs/snakemake-8.5.3/bin/python)
Reticulate 1.35.0 REPL -- A Python interpreter in R.
Enter 'exit' or 'quit' to exit the REPL and return to R.
>>> The command prompt has now changed to >>> and anything you type at it will be interpreted as python code.
So, you can’t just type R code into it any longer. For example, see what happens if you type this in there:
x <- 1:4But you can type python code, for example:
x = [1,2,3,4]Then try:
xNote, to get out of the python REPL loop, you can type:
exit
which will drop you back into R.
You can consider this a sort of Berlitz language course for R-speakers on how to communicate in Python. It assumes that y’all know something about R. This is really just scratching the surface of both langauges, but it will supply you with a few important things for working in Snakemake.
In R, we are all familiar with vectors. Each element of a vector is of the same type, i.e., integer, numeric, or character/string, etc. Here we make a numeric vector and a character vector in R:
R:
# numeric vector
x <- 1:10
# character vector
y <- c("a", "b", "c")One nice feature of having vectors is that we can do array processing of them—i.e., many operations/functions in R are vectorized: they do the same thing to each element of the vector. For example:
R:
x^2 [1] 1 4 9 16 25 36 49 64 81 100
paste(y, "!", sep = "")[1] "a!" "b!" "c!"
By contrast, vectors and vectorized operations, as such, do not exist natively in Python. Some packages provide that type of functionality, but it is not “baked into the guts of the language” the way it is in R.
Though python does not have a native vector class, like R’s, it does have something that is very similar to an R list, and, in fact, in python these are called “lists.”
In R you make a list, which is a special type of vector that can include elements of different types (i.e., one element could be an integer, another a character/string, another a numeric vector, and another a list!, etc.), like this:
R:
L_R <- list(1, 2, "R", "word", 10)and you could find its length with:
R:
length(L_R)[1] 5
The equivalent data structure in Python is a list and it is formed with [ ]:
python:
L_Py = [1, 2, "R", "word", 10]and you can get its length with:
python:
len(L_Py)5
You can access the elements of an R list using the [[ ]] operator and the index of each element. In R, lists are “base-1 indexed,” which means the first element has the index 1, and the second the index 2, and so forth. For example:
R:
# get the first element of the R list L_R
L_R[[1]][1] 1
# get the fourth element of the R list L_R
L_R[[4]][1] "word"
To access elements of a python list you use [ ], but (THIS IS EXTREMELY IMPORTANT!) python lists are “base-0 indexed”— the indexes of a python list start from 0, instead of 1. So, if you want the first element of a python list, you subscript it with 0:
python:
L_Py[0]1
While, if you wanted the 4-th element of a python list you would subscript it with 3:
python:
L_Py[3]'word'
We also note in the above output that python seems to have a preference for using single quotes for strings in its output; however just as in R, you can use " and ' somewhat interchangeably in Python. According to StackOverflow:
In Python there is no difference between both quotations of a string.
As you have noticed both result in the same value stored in a variable.
There is an important advantage of being able to use these quotation marks
interchangeably - you can include in the string the quotation mark that you
are not using to enquote the string, e.g.:
single = "This string variable contains 'single' quotation marks."
double = 'This string variable contains "double" quotation marks.'So, that is sort of like R, though R makes internal substitutions and some backslash-escaping of double quotes. Here is the R version:
R:
# R
heck_double <- "What the 'heck'?"
heck_single <- 'What the "heck"?'
heck_double[1] "What the 'heck'?"
heck_single[1] "What the \"heck\"?"
Here is the Python version:
python:
# python
heck_double = "What the 'heck'?"
heck_single = 'What the "heck"?'
heck_double"What the 'heck'?"
heck_single'What the "heck"?'
That said, every Snakefile I have seen out in the wild tends to use double quotes to enquote path names and simple command lines.
In R, you know that you can change one element of a list with a simple assignment to that element:
R:
L_R[[5]] <- "New Last Element"
# now we print the whole list
L_R[[1]]
[1] 1
[[2]]
[1] 2
[[3]]
[1] "R"
[[4]]
[1] "word"
[[5]]
[1] "New Last Element"
In python we can do exactly the same thing:
python:
L_Py[4] = "New Last Element"
# now print the whole python list
L_Py[1, 2, 'R', 'word', 'New Last Element']
Since vectorized operations do not occur automatically over the elements of lists in R, you may be familiar with the lapply() function in R, which applies a function to every element of a list. So, if we wanted to append “Weird!” to every element of list L_R, we could do something like this:
R:
lapply(L_R, function(x) paste(x, "Weird!"))[[1]]
[1] "1 Weird!"
[[2]]
[1] "2 Weird!"
[[3]]
[1] "R Weird!"
[[4]]
[1] "word Weird!"
[[5]]
[1] "New Last Element Weird!"
Here, we have, on the fly, defined a new function using the function() function in R. This is called an “anonymous” function because we do not give it a name, but it returns a function in the context where R is expecting a function (i.e., the FUN argument of lapply()). We see above that lapply() returns a list.
In python, there is a function very similar to lapply() in R, and it is called map. Additionally, there is a notion of an “anonymous” function, but, in python, these are called “lambda” functions. The analogous syntax in python for appending “Weird!” to each element of a list would be:
python:
map(lambda x: str(x) + " Weird!", L_Py)<map object at 0x1252f06a0>
map is like lapply() but the order of its arguments is reversed—it takes the function first and the list second—and it doesn’t automatically return a list; rather, as you can see, it returns a <map object at some_location> where some some_location is the address of some computer memory, like 0x292826170.
If you want to get the actual list back from map(lambda x: str(x) + " Weird!", L_Py) you have to pass it into python’s list() function:
python:
list(map(lambda x: str(x) + ' Weird!', L_Py))['1 Weird!', '2 Weird!', 'R Weird!', 'word Weird!', 'New Last Element Weird!']
In R, some type conversion is automatic if it goes from a more restricted to a more general type. In other words, if you are passing an integer or a numeric variable into a function or context that is expecting a string, R will happily convert the numeric variable to a string. Hence you can do:
R:
paste("Pi is", 3.1415)[1] "Pi is 3.1415"
In the above, 3.1415 is a numeric variable, but R is content to convert it to a string to paste() it to the string “Pi is”.
Python does not automatically convert types to the more general type. As we saw above, to concatenate two strings in python we can use the + operator:
python:
'string1' + 'string2''string1string2'
However python won’t automatically convert an numeric type, say, to a character type to get this job done. In other words, if we do this in python:
python:
x = "Pi is "
y = 3.1415
x + yTypeError: can only concatenate str (not "float") to str
We get an error, rather than getting “Pi is 3.1415”.
Therefore, when mixing types in an operation, but desiring a particular type as the output, you may need to coerce variables to that output type. For example:
python:
x + str(y)'Pi is 3.1415'
The five data types in python that correspond to R’s five, main, atomic data types are given in the following tables, along with the functions used for coercion to that data type.
| type | coercion | examples |
|---|---|---|
| logical | as.logical() |
TRUE, FALSE |
| integer | as.integer() |
3L, 16L |
| numeric | as.numeric() |
3, 16, 4.2 |
| complex | as.complex() |
3.2 + 4i |
| character | as.character() |
"foo", 'bar' |
| type | coercion | examples |
|---|---|---|
| bool | bool() |
True, False |
| int | int() |
3, 16 |
| float | float() |
3.0, 16.0, 4.2 |
| complex | complex() |
3 + 4j, 8.2 + 2.3j |
| str | str() |
"foo", 'bar' |
We emphasize a few important points here:
True and False for its Boolean types, while R uses TRUE and FALSE. The capitalization differences are very important. The variations don’t work across languages!class() function to tell you what class any object is, python has the type() function that tells you what data type an object is. This can be quite useful for familiarizing yourself with what is going on in your own code, or someone else’s:R:
list2 <- list(FALSE, 3L, 3.1415, 3 + 4.2i, "Boing!")
lapply(list2, class)[[1]]
[1] "logical"
[[2]]
[1] "integer"
[[3]]
[1] "numeric"
[[4]]
[1] "complex"
[[5]]
[1] "character"
python:
list2 = [False, 3, 3.1415, 3 + 4.2j, "Boing!"]
list(map(type, list2))[<class 'bool'>, <class 'int'>, <class 'float'>, <class 'complex'>, <class 'str'>]
list is itself a data type in python, and when we write list(map(type, list2)) we are essentially coercing an object of class map to be a list.Previously, we saw the R lapply() command. We note here that you could have applied a function to every element of a list in R using a for loop. We could have done something like this:
R:
L_R2 <- list()
for(i in 1:length(L_R)) L_R2[[i]] <- paste(L_R[[i]], "Weird!")
# now print it out:
L_R2[[1]]
[1] "1 Weird!"
[[2]]
[1] "2 Weird!"
[[3]]
[1] "R Weird!"
[[4]]
[1] "word Weird!"
[[5]]
[1] "New Last Element Weird!"
The above is not considered good R style, it is not much fun to look at and it will operate more slowly than lapply() in most circumstances. However, it is good to see this to motivate an approach in python for operating on every element of a list (and returning a list) that is considered good python style. This is called the syntax of “list comprehension”, and it looks like this:
python, but not meant to be run:
["do something to x" for x in list]where “do something to x” is replaced by some operation on a variable called x (or whatever you might want to call it) and list is an existing list object.
So, to paste “Weird!” onto the elements of the list L_Py via list comprehension we would do:
python:
[str(x) + " Weird!" for x in L_Py]['1 Weird!', '2 Weird!', 'R Weird!', 'word Weird!', 'New Last Element Weird!']
See that it returns a list. Unlike in R, for loops, and especially those within a list comprehension, are not eschewed.
There is actually a further subtlety here. The for in there can cycle over the elements in a list, but in fact, it is much more general. It can cycle over all the elements in any “iterable” (i.e., able to be iterated over), object. For example a single string like "ABCDEFG" is iterable—python will iterate over the different letters in the string. Thus you could make a list of the letters in a string like:
python:
[x for x in "ABCDEFG"]['A', 'B', 'C', 'D', 'E', 'F', 'G']
In fact, you can go for even more economy and note that list() in python applied to any iterable object, will automatically iterate over the iterable elements in the object. Hence, this does the same as the previous line:
python:
list("ABCDEFG")['A', 'B', 'C', 'D', 'E', 'F', 'G']
That is good to keep in mind. If you want a list of length 1 that includes the string "ABCDDEFG", then, rather that list("ABCDEFG"), you would want to do:
python:
["ABCDEFG"]['ABCDEFG']
+ in python is like c()—it catenates the listsIn R, if you want to add one list onto the end of another, essentially catenating them, you would use c() like this:
R:
c(L_R, list("add", "some", "more", 3))[[1]]
[1] 1
[[2]]
[1] 2
[[3]]
[1] "R"
[[4]]
[1] "word"
[[5]]
[1] "New Last Element"
[[6]]
[1] "add"
[[7]]
[1] "some"
[[8]]
[1] "more"
[[9]]
[1] 3
In python, you use + for that type of operation:
python:
L_Py + ["add", "some", "more", 3][1, 2, 'R', 'word', 'New Last Element', 'add', 'some', 'more', 3]
dicts in pythonIn R, the separate elements of a list can be named and we can use the names to access them. For exmaple:
R:
dinners <- list(
Monday = "Arctic char",
Tuesday = "Chicken soup",
Wednesday = "Pizza",
Thursday = "Stir fry",
Friday= "Fend for yourselves, kids!"
)And then we can access each element of the list by its name in two ways:
R:
# either with the dollar sign
dinners$Tuesday[1] "Chicken soup"
# or with double brackets
dinners[["Friday"]][1] "Fend for yourselves, kids!"
And of course we can assign values to that list in either way:
R:
# make the changes
dinners$Friday <- "Go out to brewery"
dinners[["Tuesday"]] <- "Burritos"
# then see what that did:
dinners$Monday
[1] "Arctic char"
$Tuesday
[1] "Burritos"
$Wednesday
[1] "Pizza"
$Thursday
[1] "Stir fry"
$Friday
[1] "Go out to brewery"
Interestingly, the elements of a named list in R can still be accessed by their index. We could get “Pizza” out of the dinners list in three different ways:
R:
dinners$Wednesday[1] "Pizza"
dinners[["Wednesday"]][1] "Pizza"
dinners[[3]][1] "Pizza"
Lists in python do not take names like lists in R; however, there is another data type in python that does behave something like a named list. This data type is called a dict, which is, I believe, short for “dictionary.” We can make a dict of our dinners like this:
python:
dinners = {
"Monday": "Arctic char",
"Tuesday": "Chicken soup",
"Wednesday": "Pizza",
"Thursday": "Stir fry",
"Friday": "Fend for yourselves, kids!"
}And we can print it like this:
python:
dinners{'Monday': 'Arctic char', 'Tuesday': 'Chicken soup', 'Wednesday': 'Pizza', 'Thursday': 'Stir fry', 'Friday': 'Fend for yourselves, kids!'}
which is not much fun to look at, and gets really ugly when the dict has multiple nested levels (which is definitely possible, and we will see examples of that when we start playing with Snakemake config files).
As a side note, there is a python module called pprint that will pretty-print a variety of python objects. It can be used like this:
python:
# import only needs to happen once per session
import pprint
pprint.pp(dinners){'Monday': 'Arctic char',
'Tuesday': 'Chicken soup',
'Wednesday': 'Pizza',
'Thursday': 'Stir fry',
'Friday': 'Fend for yourselves, kids!'}
That is way more palatable!
Elements of a dictionary can be accessed much like elements of a named list in R, but python uses [ ] for that (rather than $ or [[ ]]) like this:
python:
dinners["Thursday"]'Stir fry'
In R, lists can be nested with multiple levels, and the same is true with dicts in python. (In fact, it is true with python lists, as well…)
python:
meals = {
"Monday": {"breakfast": "eggs", "lunch": "salad", "dinner": "Arctic char"},
"Tuesday": {"breakfast": "oatmeal", "lunch": "sandwich", "dinner": "Chicken soup"}
}Now, each element at the top level (Monday or Tuesday) is itself a dict with three elements (breakfast, lunch, and dinner). If you want to get the value for Tuesday’s lunch, you just drill down into the dict, much like you would in a named R list:
meals["Tuesday"]["lunch"]'sandwich'
We will come back to the dict data struture a lot when we talk about how Snakemake parses and stores the information in YAML formatted snakemake config files.
R has a system called S4 that allows for very general kinds of objects to be defined. Most people don’t fiddle with that too much, so it isn’t much help by analogy. Python also has a mechanism to define certain classes of objects that can be very general. The programmer can define a class of objects, and then create an object which is an instance of a class. For our purposes working with snakemake, the important part is that instances of a class can hold attributes that can be just about anything.
We are discussing this because Snakemake, when it is executing, carries around values of wildcards in these sorts of objects, and you can access the wildcard values as attributes of a wildcards object (e.g. you can get their values using wildcards.sample or wildcards.chromo) and use those values in what are called input functions which we will see in the next chapter.
Here we just present a simple example of defining a class, creating an object that is an instance of that class, and then setting and accessing attribute values in it.
python:
# Define class. Can be named anything. We name this Foo.
# pass is just a placeholder. There could/should be some code there
# but our silly example class, here, does nothing, so we just give
# it "pass" which is code that does nothing
class Foo(object):
pass
# create an instance of class Foo called myfoo
myfoo = Foo()
# assign some values to some attributes. Uses the . (dot)
myfoo.attrib1 = "bing"
myfoo.attrib2 = "bong"
myfoo.hello = ["bonjour", "hola", "hallo"]
# access the value of those attributes
myfoo.attrib1'bing'
myfoo.attrib2'bong'
myfoo.hello['bonjour', 'hola', 'hallo']
So, we just wanted people to see this because it turns out it is helpful when writing snakemake input functions to be able to create a fake wildcards object for testing those input functions.
Here we will compare and contrast how functions are defined in R and python, and in the process we will see how blocks of code in python are defined by indentation, rather than by, say, curly braces, as in R.
If you are familiar with R functions, then you know you can define one like this:
R:
add2 <- function(x) {
x + 2
}That is a function that adds 2 to its argument. Like this:
R:
add2(6)[1] 8
Not very exciting, but we see that the code of the function is enclosed in curly braces, and, in R, the value produced by the last line of code gets returned by the function.
In python, the equivalent function would be defined this way:
python:
def add2(x):
return x + 2And you can use it like this:
python:
add2(6)8
Instead of putting the code between curly braces, python uses indentation to know which code belongs to the function. Any lines indented to the same degree (here, two spaces) after the colon on the first line are understood to be part of the function definition. Importantly, unlike R, python does not automatically return the value produced by the last line of code in the function. If you want your python function to return a value, you must prepend the value or object that you want returned with the keyword return
We might also point out here that anywhere you might typically use curly braces in R to denote a block of code, python accomplishes the same thing with a colon and indentation—for loops, if and else statements, etc.
The parameters of a python function can be any valid python objects and you can have multiple parameters and they can have default values, if desired. For example
python:
# here is a function that adds text ("Fun!", by default) to the elements of a list
# and returns a new list:
def fun(list_of_things, text="Fun!"):
return [ x + ", " + text for x in list_of_things]If we call it with no value for the text parameter we get the default value for text.
python:
fun(list_of_things = ["dogs", "kittens", "frisbees"])['dogs, Fun!', 'kittens, Fun!', 'frisbees, Fun!']
But we could add the text argument to change it:
python:
fun(list_of_things = ["snakes", "lizards", "squid"], text = "Slithery!")['snakes, Slithery!', 'lizards, Slithery!', 'squid, Slithery!']
In R, when you want to include a bunch of useful functions from an R package, you can use the library command:
R:
library(tidyverse)In python, you can do something similar by importing “modules.” We have already seen that with the pprint module for pretty printing things—we used:
python:
import pprintThat statement imports a pprint object, and once that is done we can use all the functions from the pprint module by naming them as attributes of the pprint object:
python:
pprint.pp(meals){'Monday': {'breakfast': 'eggs', 'lunch': 'salad', 'dinner': 'Arctic char'},
'Tuesday': {'breakfast': 'oatmeal',
'lunch': 'sandwich',
'dinner': 'Chicken soup'}}
In python, we can also be more fine-grained about what we import from different modules. At its most fine-grainedness, we could import just a single function from a module so that we can call that function by name, without calling it as an attribute of an object. For example, if we wanted to play with the multiext function from snakemake (which happens to reside in a module named snakemake.io) we can import it like this:
python:
from snakemake.io import multiextNow we can use the multiext function to see how it works:
python:
multiext("path/dir/file", ".pdf", ".jpg")['path/dir/file.pdf', 'path/dir/file.jpg']
Note that this works because we are using python in the conda environment that includes snakemake. (Remember the line like use_condaenv(condaenv = "/projects/eriq@colostate.edu/miniforge3/envs/snakemake-8.5.3")?) Otherwise, python typically would not find the snakemake module.
One can also import a module, but give its namespace a shorter name. For example, to prepare for the next section, we will import the ‘pandas’ module, but we will name it pd for brevity.
python:
import pandas as pdMost R users are now familiar with the ‘tidyverse’ and its packages like ‘readr’ and ‘dplyr’. We give an example here, reading a small tab-separated file with information about the samples in our example Chinook salmon data set, which is in the repo at: data/config/units.tsv. The easiest way to peruse it is to view it on GitHub here.
If we wanted to read it into R using the tidyverse, that would be pretty familiar (at least if we are tidyverse aficionados):
R:
library(tidyverse)
units <- read_tsv("./data/config/units.tsv")
# and then we print it to see the familiar tibble format
units# A tibble: 16 × 12
sample unit library flowcell platform lane sample_id barcode fq1 fq2
<chr> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr>
1 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
2 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
3 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
4 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
5 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
6 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
7 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
8 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
9 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
10 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
11 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
12 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
13 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
14 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
15 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
16 DPCh_pla… 1 plate1 HTYYCBB… ILLUMINA 1 DPCh_pla… DPCh_p… data… data…
# ℹ 2 more variables: kb1 <dbl>, kb2 <dbl>
The python world has a similar package, called pandas. We imported it above, using import pandas as pd, so now we can use it to do a comparable read operation on our tab-separated file using the pandas function read_table(). We will also force each column to be stored as a string, too (which is often helpful if we are going to use these values in snakemake later…) Pandas also offers indexing of data frame for faster access, so, in this case we will index each row by the sample name
python:
units = pd.read_table("./data/config/units.tsv", dtype=str).set_index(
"sample", drop=False
)
# and then we can print it out:
units sample unit ... kb1 kb2
sample ...
DPCh_plate1_B10_S22 DPCh_plate1_B10_S22 1 ... 10822.571 12438.261
DPCh_plate1_B11_S23 DPCh_plate1_B11_S23 1 ... 10838.002 12026.601
DPCh_plate1_B12_S24 DPCh_plate1_B12_S24 1 ... 11437.975 12892.735
DPCh_plate1_C10_S34 DPCh_plate1_C10_S34 1 ... 14139.54 15767.478
DPCh_plate1_C11_S35 DPCh_plate1_C11_S35 1 ... 9705.477 11108.45
DPCh_plate1_C12_S36 DPCh_plate1_C12_S36 1 ... 10467.881 12064.275
DPCh_plate1_D09_S45 DPCh_plate1_D09_S45 1 ... 12167.214 13628.883
DPCh_plate1_D11_S47 DPCh_plate1_D11_S47 1 ... 12337.821 13929.613
DPCh_plate1_F10_S70 DPCh_plate1_F10_S70 1 ... 10805.055 11871.849
DPCh_plate1_F11_S71 DPCh_plate1_F11_S71 1 ... 12583.866 14308.931
DPCh_plate1_F12_S72 DPCh_plate1_F12_S72 1 ... 14848.51 16313.417
DPCh_plate1_G09_S81 DPCh_plate1_G09_S81 1 ... 20816.204 22987.841
DPCh_plate1_G10_S82 DPCh_plate1_G10_S82 1 ... 24775.084 27340.307
DPCh_plate1_G12_S84 DPCh_plate1_G12_S84 1 ... 18082.229 20757.831
DPCh_plate1_H09_S93 DPCh_plate1_H09_S93 1 ... 9702.925 10835.209
DPCh_plate1_H10_S94 DPCh_plate1_H10_S94 1 ... 11179.913 12277.416
[16 rows x 12 columns]
That looks pretty similar. The first column in that output shows that sample is the index column (which we set with .set_index(), after reading it) of this pandas data frame.
Now, in R, if we wanted to pull out the values in some of the columns that correspond to a particular sample, we could do like this:
R:
# get fq1 value for DPCh_plate1_C12_S36
units %>%
filter(sample == "DPCh_plate1_C12_S36") %>%
pull(fq1)[1] "data/fastqs/DPCh_plate1_C12_S36_R1.fq.gz"
# get kb2 value for DPCh_plate1_C12_S36
units %>%
filter(sample == "DPCh_plate1_C12_S36") %>%
pull(kb2)[1] 12064.27
In pandas you can do a similar thing with the .loc method (short for “location”) of units, leveraging the fact that we are using sample as the index. The syntax is:
data_frame.loc( value of index, Column )So, analagous to the above we could do, in python:
python:
# get fq1 value for DPCh_plate1_C12_S36
units.loc[ "DPCh_plate1_C12_S36", "fq1"]'data/fastqs/DPCh_plate1_C12_S36_R1.fq.gz'
# get kb2 value for DPCh_plate1_C12_S36
units.loc[ "DPCh_plate1_C12_S36", "kb2"]'12064.275'
Note that if you want to extract values based on multiple values (both the samples and unit column, for example) you could do
python:
units2 = units.set_index(["sample", "unit"], drop = False)
units2.loc[ ("DPCh_plate1_C12_S36", "1"), "kb2" ]'12064.275'
These are handy ways of populating parameters of rules with values that are taken from tabular data, for example, getting read group information for different samples.
We should point out that we can also use the .loc method to test more general conditions in the pandas data frame. For example, if we wanted a list of all the samples which have an F in their name we could put that condition test into place where the index went before:
python:
units.loc[ units["sample"].str.contains("F"), "sample" ]sample
DPCh_plate1_F10_S70 DPCh_plate1_F10_S70
DPCh_plate1_F11_S71 DPCh_plate1_F11_S71
DPCh_plate1_F12_S72 DPCh_plate1_F12_S72
Name: sample, dtype: object
We see that this sort of construct returns the sample column in the correct rows, but it is still a pandas object. If we want to just get a list of those sample names, we use .tolist() on the end:
python:
units.loc[ units["sample"].str.contains("F"), "sample" ].tolist()['DPCh_plate1_F10_S70', 'DPCh_plate1_F11_S71', 'DPCh_plate1_F12_S72']
Those things come in handy when writing more complex workflows.