15  Basic Handling of VCF files

As always, sync your fork and pull down any changes before trying to run through this

As we saw in the section on bioinformatic formats, VCF files can be large and unwieldy. The format specification is also such that fields might have different numbers of subfields, depending, for example, on the number of alleles found at a variant. Both of these features make it hard to directly read a VCF file into, say, R, or some other program that may wish to treat it purely as tabular data.

This is not to say that you couldn’t just read a VCF file into R directly as a TAB delimited text file, and then start splitting fields up on it. However, there are specialized tools for doing operations on VCF files, and becoming familiar with them can relieve a lot of the pain of dealing with VCF files.

To have an example VCF file to play with, the course repository contains a vcf.gz file at data/vcf/all.vcf.gz. This turns out to be the unfiltered VCF file you get after running our course example data through a standard GATK pipeline. The course repository also includes the file data/vcf/all-hard-filtered-miss-marked.bcf which was obtained by running standard hard-filtering data/vcf/all.vcf.gz and also properly marking individuals with no reads (or a flat genotype likelihood) as missing. This second file is in BCF format, which is a compressed binary format. We have both formats here so that you get comfortable with the fact that you can handle all three formats, vcf, vcf.gz, and bcf in the same way using bcftools. Note that when people say “VCF file” they could be referring to a file in any of those three formats.

We have also indexed each of those files using bcftools like this, for example:

bcftools index data/vcf/all-hard-filtered-miss-marked.bcf

Indexing creates a file with the same name but with a .csi extension, which stands for “coordinate-sorted index.” An index is required if you want to make fast access to variants in particular regions of the genome, and it is also sometimes required for other operations with bcftools.

There are two main, well-known programs for handling VCF files: vcftools and bcftools. Both of these grew out of the 1000 Genomes effort starting around the late 2000s. It seems that vcftools may have been developed first, but, currently, bcftools is being more actively developed, with new versions and new features being added to it regularly. vcftools provides some very specific commands for particular analyses or operations, some of which are not available from bcftools. On the other hand, bcftools provides a more general interface for operations on VCF files. By this interface, a great number of the operations done in vcftools are available, but a little extra knowledge is required to implement them. That said, the range of possible operations seems much larger in bcftools than in vcftools.

Further, bcftools behaves like a typical Unix utility, allowing data to be streamed to stdout, and data can be streamed into bcftools (by using the - as the input file name) from stdin. This lets you pipe output into it the way you can with most Unix tools. This makes it far more versatile than vcftools.

15.1 Installing bcftools

Both SEDNA and Alpine have bcftools in a module, and you can use that, if you like:

# on SEDNA
module load bio/bcftools

# on Alpine (from a compute node or acompile)
module load bcftools

Alternatively you can use mamba to create a conda environment called bcftools that has it:

mamba create -n bcftools -c conda-forge -c bioconda bcftools
conda activate bcftools

(Note the use of -c conda-forge in the above. This is present because some of the dependencies for bcftools are not found on the bioconda channel. Rather they are on the conda-forge channel. If you conda/mamba environment is not set up to search conda-forge by default, then the -c conda-forge is required to get all the dependencies.)

15.2 bcftools basic interface

Like samtools (which is maintained by the same group of people), bcftools possesses a number of different subcommands. So, the syntax is always like:

  • bcftools subcommand options file(s)

Also like samtools, bcftools will take input from stdin rather than from a file—you just pass it - instead of a file name.

The full documentation/manual for bcftools is maintained at: http://samtools.github.io/bcftools/bcftools.html. It is well worth reading through this whole documentation, though it can be quite terse and intimidating. A friendlier “tutorial” introduction to the software can be found at https://samtools.github.io/bcftools/howtos/index.html.

Here we are going to get our feet with with a few operations.

First, we will look at the “onboard” documentation. By just entering bcftools you get a list of all the subcommands that are available:

bcftools

If you want the onboard documentation for any of the particular subcommands, you can just give a naked bctools subcommand command, like:

bcftools index

or, for a more daunting set of documentation:

bcftools roh

15.3 Index my VCF file!

The first thing we are going to do is index our VCF files. We create the default index, a coordinate sorted index which has the .csi extension. If your VCF file is not in coordinate-sorted order, you might have to sort it before you do this. However, our example files are already sorted. In fact, they are already indexed, so we don’t have to do the following:

bcftools index data/vcf/all.vcf.gz
bcftools index data/vcf/all-hard-filtered-miss-marked.bcf

The version of bcftools I am using notes that the index is already there so it doesn’t re-index it.

The index allows for rapid access to different parts of the files that correspond to specific locations in the genome.

15.4 Tell me about my VCF file!

VCF files are a little daunting. Especially when they are gzipped (or are BCF files!) they can seem particularly opaque—learning anything about them in the traditional manner of uncompressing them and then searching for lines within them or counting up the number of records can be time consuming. Here are some bcftools solutions to a few different questions you might have.

Who is in this file? You can always try to find the last header line in a VCF file using grep or awk and parse the individuals out yourself, but it turns out to be faster and safer to use the query subcommand from bcftools with the -l option. Do it here:

bcftools query -l data/vcf/all.vcf.gz

# And, of course, you can do the same with the BCF file
bcftools query -l data/vcf/all-hard-filtered-miss-marked.bcf

Then read about it on the manual page. Find the part that describes it.

How many variants are in this file? This question can be answered quickly with bcftools stats, which also returns to you a plethora of information about the variants.

bcftools stats  data/vcf/all.vcf.gz | less -S

The top part of the output tells you how many SNPs and indels (and other types of variants) there are. Then it tells you about Ts/Tv ratios, then it essentially gives histogram summaries for allele frequencies, variant quality scores (QUAL), insertion-deletion sizes, substitution types, read depths, etc.

We can do the same for the bcf file:

bcftools stats data/vcf/all-hard-filtered-miss-marked.bcf  | less -S

Where are these variants? There are several ways to answer this question. One might be simply to print the CHROM and the POS for each row in the VCF file:

bcftools query -f '%CHROM\t%POS\n' data/vcf/all.vcf.gz | less

This uses bcftools query telling it to print the CHROM and POS for each variant.

If you want to see where it starts and where it finishes you can do:

bcftools query -f '%CHROM\t%POS\n' data/vcf/all.vcf.gz | head 
bcftools query -f '%CHROM\t%POS\n' data/vcf/all.vcf.gz | tail

If we wanted to quickly see how many variants were on each of the chromosomes/scaffolds, sorted by number of variants, we could do:

bcftools query -f '%CHROM\t%POS\n' data/vcf/all.vcf.gz | awk '{n[$1]++} END {for(i in n) print i, n[i]}' | sort -nbr -k 2

The subcommand query has as its main purpose simply extracting fields of information from a VCF file and spitting them out in a new, user-specified, typically tabular format. It is super useful.

Give me a glimpse of the file You can use bcftools view for a number of things, but at its simplest, you can merely look at the file in VCF format. (In this manner, it behaves much like samtools view for VCF files).

# show the whole file from the top
bcftools view data/vcf/all-hard-filtered-miss-marked.bcf | less -S

# show just the header with -h.  Here we look at just the last 10 lines of the header
bcftools view -h data/vcf/all-hard-filtered-miss-marked.bcf | tail

# show the variants themselves (no header) with -H
bcftools view -H data/vcf/all-hard-filtered-miss-marked.bcf | less -S

Just like you can with samtools view you can convert formats with bcftools view. Pipe a VCF into it and then use the -O (Big-O, not a zero) option:

  • -O z: bgzipped VCF (vcf.gz)
  • -O v: uncompressed VCF (the default)
  • -O u: uncompressed BCF
  • -O b: compressed BCF

15.4.1 Rename the samples/individuals in the file

We saw above that the names of the samples in the file are like DPCh_plate1_B10_S22. The names in the VCF file are set by the SM field of the read groups in the BAM files from which variants are called. Let’s imagine that we really want the names of those individuals to be different. If we wanted different names for those samples, we don’t have to go all the way back to the beginning and remap everything and call variants. We simply rename the samples in the VCF file. For this we can use bcftools reheader. First, look at the documentation for that, both on the web, and with:

bcftools reheader

Aha! we see that the web-based documentation is a little more complete, and it tells us what format to use for a sample-renaming file for the -s option. Copy the following contents (using nano, for example) into a file called sample-renames.txt

DPCh_plate1_F11_S71 T145171
DPCh_plate1_B12_S24 T145193
DPCh_plate1_F12_S72 T145241
DPCh_plate1_G12_S84 T145242
DPCh_plate1_B11_S23 T145152
DPCh_plate1_C11_S35 T145153
DPCh_plate1_C12_S36 T145200
DPCh_plate1_D11_S47 T145163
DPCh_plate1_F10_S70 T144996
DPCh_plate1_D09_S45 T144967
DPCh_plate1_H10_S94 T145003
DPCh_plate1_G10_S82 T144997
DPCh_plate1_H09_S93 T144979
DPCh_plate1_C10_S34 T144989
DPCh_plate1_B10_S22 T144981
DPCh_plate1_G09_S81 T144974

That file shows the NMFS_DNA_ID (like T144979) that we would like to use for sample names, rather than the names currently in the VCF file, like DPCh_plate1_H09_S93.

We can make a renamed bcf file with:

bcftools reheader -s sample-renames.txt data/vcf/all-hard-filtered-miss-marked.bcf  > renamed.bcf

and a renamed vcf.gz file with:

bcftools reheader -s sample-renames.txt data/vcf/all.vcf.gz  > renamed.vcf.gz

In this case, the type of output file (bcf or vcf.gz) is the same as the type of the input file.

Exercise Use bcftools view and tail to see that the names have really been changed. Then use bcftools query to do the same.

15.4.2 Print/Extract fragments/parts of my VCF file

There are lots of ways to extract desired bits of information from a VCF file into a more manageable format.

Extract keyed values from the INFO field

When we did this:

bcftools view -H | less

we saw that there is a lot of information in the INFO field. What if we wanted to extract that? It looks like it could be hard to parse because the fields are in semi-colon-separated key-value pairs.

This is another job for bcftools query. You pass a format string to the -f option that tells the program which fields you want to extract and how you want to format it. In general, the values are preceded by a % and subfields of the INFO column can be named described like %INFO/SUBFIELD. You can ask for TABs between fields with \t and for line endings with \n. In general you need to wrap all of these format specifications in single quotes so that the shell does not get confused by them.

Check out some examples:

# extract CHROM POS and BaseQRankSum, separated by TABs
bcftools query -f '%CHROM\t%POS\t%INFO/BaseQRankSum\n' data/vcf/all.vcf.gz | less

# extract CHROM POS and total read depth DP
bcftools query -f '%CHROM\t%POS\t%INFO/DP\n' data/vcf/all-hard-filtered-miss-marked.bcf | less

You can even extract information from each of the genotype columns. If you want to print CHROM and POS and then all of the PHRED-scaled genotype likelihoods for all the samples, separated by TABs, you can do:

bcftools query -f '%CHROM\t%POS\t[%PL\t]\n' data/vcf/all.vcf.gz | less -S

Note that FORMAT values (i.e., those in the genotype columns) must be wrapped in [ ] to get all the values to be printed out.

EXERCISE: Extract the CHROM, POS, Maximum Likelihood-estimated Allele Frequency (MLEAF in the INFO column) for each variant, along with the allele depths (AD) of each individual, all TAB separated, from the file all.vcf.gz.

View data from specified regions

What if we want to look at variants only in two 5 Kb regions, like NC_037122.1f5t9:1-5000 and NC_037123.1f10t14:1000000-1005000? Pass those, separated by commas, to the -r option (which is an option that applies to many of the subcommands):

bcftools view -H -r NC_037122.1f5t9:1-5000,NC_037123.1f10t14:1000000-1005000 data/vcf/all.vcf.gz | less -S

You can also specify those regions in a file with the -R option.

This typically requires that an index be available for the VCF file, which allows direct and efficient access to different parts of the genome in the file.

View data from specified individuals

You can give the sample names (comma separated) to the -s option:

bcftools view -s DPCh_plate1_C11_S35,DPCh_plate1_C12_S36,DPCh_plate1_D09_S45 data/vcf/all-hard-filtered-miss-marked.bcf | less -S

Or, if you wanted to view all but those two individuals, precede them with a ^:

bcftools view -s ^DPCh_plate1_C11_S35,DPCh_plate1_C12_S36,DPCh_plate1_D09_S45 data/vcf/all-hard-filtered-miss-marked.bcf | less -S

You can also supply a text file with sample names (one-per-line) to the capital letter -S option.

You can combine options, like -r and -s, as well.

15.5 Combine VCF files in various ways

Catenate VCF files

If you have VCF files called from the same reference genome filled with the same samples, it is easy to catenate them together with bcftools concat:

# make two files from different regions
bcftools view -O z -r NC_037122.1f5t9:1-5000 data/vcf/all.vcf.gz  > A.vcf.gz
bcftools view -O z -r NC_037123.1f10t14:1000000-1005000 data/vcf/all.vcf.gz  > B.vcf.gz

# how many variants in each of those?
bcftools stats A.vcf.gz | awk '/^SN/'
bcftools stats B.vcf.gz | awk '/^SN/'

# catenate the back together
bcftools concat -Oz  A.vcf.gz B.vcf.gz > CAT.vcf.gz

# how many variants in that?
bcftools stats CAT.vcf.gz | awk '/^SN/'

Note that when using the -O (capital “o”) option to specify the output type: v = VCF, b = BCF, u = uncompressed BCF, z = bgzipped VCF, you don’t need a space after the -O.

Merge VCF files

If you have files with different samples in them you can easily combine them:

# make file with first three samples
bcftools view -Oz \
  -s DPCh_plate1_B10_S22,DPCh_plate1_B11_S23,DPCh_plate1_B12_S24 \
  data/vcf/all.vcf.gz > first3.vcf.gz

# make another with the last three samples
bcftools view -Oz \
  -s DPCh_plate1_G12_S84,DPCh_plate1_H09_S93,DPCh_plate1_H10_S94 \
  data/vcf/all-hard-filtered-miss-marked.bcf > last3.vcf.gz

# merging requires that the files be indexed
bcftools index first3.vcf.gz
bcftools index last3.vcf.gz

# merge those into a file with 6 samples
bcftools merge -Oz first3.vcf.gz last3.vcf.gz > 6-samples.vcf.gz

15.6 Filter out variants for a variety of reasons

There are a lot of ways to filter out variants. bcftools leaves things very general here, and so just about anything is possible. Some simple ones appear below. Remember, we are piping the result to bcftools stats just so that we can see the result. If you really wanted to make a filtered file, you would typically just redirect it to a file.

Only the sites that pass filters

When we did the hard-filtering with GATK, we ended up with (mostly) the same variants in the file, but there were entries in the FILTER column: PASS if the variants passed filters and a descriptive term for the filters that were not passed if they didn’t pass at least one of the filters. This is in keeping with the general GATK philosophy of not discarding any data, but merely marking data for inclusion (i.e. PASS), or not, in downstream analyses. However, if you were to use data/vcf/all-hard-filtered-miss-marked.bcf directly in an analysis, you might, if you didn’t remove the sites that did not pass filter, inadvertently end up using them.

So, how do we make a file that doesn’t have the variants that did not pass filter? We can use bcftools view with the -i option (which stands for “include”)

bcftools view -i 'FILTER="PASS"' data/vcf/all-hard-filtered-miss-marked.bcf > passed.bcf

And, you can also exclude the ones that pass, to look just at those that didn’t pass:

bcftools view -e 'FILTER="PASS"' data/vcf/all-hard-filtered-miss-marked.bcf > exclude_passed.bcf

Have a look at those with bcftools view -H exclude_passed.bcf | less -S and see why they didn’t pass filters.

Just the biallelic SNPs please

Hey Folks, we are adding | bcftools stats - | awk '/^SN/' to many of the following commands.

However, that is just so we can quickly see what effect different filtering options have in terms of numbers of variants. If you really want to have a filtered file for further use, just redirect the output into a file (and, you would also probably want to change -Ou to -Oz or -Ob).

Get things with no more than 2 alleles and no fewer than two alleles, and of a type = SNP:

# do it and summarize the result to look at it, all in one line:
bcftools view -Ou -m 2 -M 2 --types=snps data/vcf/all-hard-filtered-miss-marked.bcf | bcftools stats - | awk '/^SN/'

Just the biallelic indels please

# do it and see the result all in one line:
bcftools view -Ou -m 2 -M 2 --types=indels data/vcf/all.vcf.gz | bcftools stats - | awk '/^SN/'

Note the use of -Ou in order to pipe uncompressed BCF output directly into bcftools stats using the - for a filename.

Fraction of missing sites less than X

If you want to make sure that 60% of your individuals have at least one read at the genotype, you can do this:

bcftools view -i 'F_MISSING < 0.40' data/vcf/all-hard-filtered-miss-marked.bcf | bcftools stats - | awk '/^SN/'

Play with setting the F_MISSING to different values and see how that affects the number of variants retained.

Here’s why GATK’s decision to leave missing data as 0/0 is such a disaster:

The F_MISSING variable is something that bcftools calculates “on the fly” by counting up the number of individuals with missing genotypes that are denoted by ./.. Since GATK doesn’t reliably mark missing data as such (but probably will in a newer version…), if you use the vcf file in which missing data has not been marked as ./., then not much happens when you change the filters on F_MISSING. Try it, by changing 0.40 to different values to see that hardly anything changes:

bcftools view -i 'F_MISSING < 0.40' data/vcf/all.vcf.gz | bcftools stats - | awk '/^SN/'

Exclude based on various features of the data

Here we circle back around to say that -i and -e are not just for filtering on the FILTER column.

You can use the -e option to bcftools view or bcftools filter to exclude sites that meet certain criteria. (You can use -i to include those sites and no others).

The syntax for use with -i and -e is very powerful, but somewhat complex. It is described (in a terrifyingly terse and dense fashion) in the FILTERING EXPRESSIONS section of the bcftools manual.

A somewhat more approachable description of the filtering options in bcftools can be found in the bcftools filtering tutorial.

We look at a few examples here.

To only keep things with a maximum-likelihood-estimated allele frequency between 0.4 and 0.6:

bcftools view -i 'INFO/MLEAF >= 0.4 && INFO/MLEAF <= 0.6' data/vcf/all-hard-filtered-miss-marked.bcf | bcftools query -f '%INFO/MLEAF\n' | less

Note we are piping the result to bcftools query in order to see what the actual MLEAFs are after filtering. For the most part, this has worked, except for cases in which there are more than two allele freqencies. If we wanted to filter those out, we could filter to only biallelic sites, or, for the sake of illustration, we could retain only those sites at which the MLEAF value for the first alternate allele is between 0.4 and 0.6:

bcftools view -i 'INFO/MLEAF[0] >= 0.4 && INFO/MLEAF[0] <= 0.6' data/vcf/all-hard-filtered-miss-marked.bcf | bcftools query -f '%INFO/MLEAF\n' | less

Cool!

How about excluding those sites in which any individual had a DP less than 5. We can test each of the DP columns in the FORMAT columns. We name these FMT/DP when we are doing filtering. Note that each test (from each sample) is combined with an OR by default, so:

bcftools view -H -e 'FMT/DP < 5' data/vcf/all-hard-filtered-miss-marked.bcf | less

To make it easier to see what the DPs are there, let’s print them:

bcftools view -e 'FMT/DP < 5' data/vcf/all-hard-filtered-miss-marked.bcf | bcftools query -f '%CHROM\t%POS\t[%DP\t]\n' | less

This has just scratched the surface of the things that can be done with bcftools, but I hope it has encouraged you to read through the manual, including the section on filtering expressions, as well as the filtering tutorial.

15.7 Make a “beagle” file for input to ANGSD from a VCF file

We can use bcftools query and awk to get this done well. The first step is to get the header line:

bcftools query -l data/vcf/all-hard-filtered-miss-marked.bcf | awk '
  BEGIN {printf("marker\tallele1\tallele2");} 
  {printf("\t%s\t%s\t%s", $1, $1, $1);} 
  END {printf("\n");}
'

Then, after that, for the data, we can use bcftools view to restrict ourselves to biallelic SNPs (as required for ANGSD) and pipe the result to bcftools query to print the CHROM, POS, REF and ALT and the Phred scaled likelihoods. We pipe that to less -S to see what we get:

 bcftools view  -m 2 -M 2 --types=snps data/vcf/all-hard-filtered-miss-marked.bcf | bcftools query -f '%CHROM:%POS\t%REF\t%ALT\t[%PL\t]\n' -  | less -S

Now we could just pass that stream into an awk script that computes the raw genotype likelihoods from the Phred scaled ones and formats the chromosome and position appropriately. I had an awk script to do that, but we can actually just put it that awk script into a shell script that also runs the steps of printing the header ande filtering the VCF file.

Such a script is included in this repository, in scripts/vcf2beagle.sh. It looks like this:

Listing of scripts/vcf2beagle.sh

if [ $# != 1 ]; then
    echo "Error! Need to supply vcf, bcf, or vcf.gz file."
    echo
    echo "Syntax:"
    echo "  vcf2beagle  File.bcf "
    exit 1
fi


file=$1

(
    bcftools query -l  $file; 
    bcftools view -m 2 -M 2 --types=snps  data/vcf/all-hard-filtered-miss-marked.bcf | \
    bcftools query -f '%CHROM:%POS\t%REF\t%ALT[\t%PL]\n' -
    ) | awk '

# this is a simple script that takes lines that are have comma-separated
# Phred-scaled genotype liklelihoods starting in column three, like:
#
# CHROMPOS  REF ALT 0,12,34 0,3,16
#
# (i.e., the CHROM and POS are smooshed into a single column.)
#
# and it turns each of those comma-containing columns into three columns
# of genotype likelihoods scaled to sum to one.

BEGIN {
    IFS="\t"
    base2int["A"] = 0
    base2int["C"] = 1
    base2int["G"] = 2
    base2int["T"] = 3
    printf("marker\tallele1\tallele2");
}

NF==1 {
    printf("\t%s\t%s\t%s", $1, $1, $1);
    next;
}

{   
    if(header_done == 0) {
        printf("\n");
        header_done = 1;
    }
    chrompos=$1
    gsub(/_/, "-", chrompos)
    gsub(/:/, "_", chrompos)
    printf("%s\t%s\t%s", chrompos, base2int[$2], base2int[$3]);
    for(i=4;i<=NF;i++) {
        if($i == ".") {  # deal with old VCFs that use a single dot here
            printf("\t%.6f\t%.6f\t%.6f", 1.0/3.0, 1.0/3.0, 1.0/3.0);
        }
        else {
            n=split($i,a,/,/);
            if(n!=3) {
                print "Not 3 GL values at ", $1  > "/dev/stderr";
                exit;
            }
            x=10^(-a[1]/10);
            y=10^(-a[2]/10);
            z=10^(-a[3]/10);
            sum=x+y+z;
            printf("\t%.6f\t%.6f\t%.6f",x/sum, y/sum,z/sum);
        }
    }
    printf("\n");
}
'