Overview of additions

Jon Hess and friends at CRTFC are kicking ass using SNPPIT for PBT in the Columbia River. They have now gone to \(\approx 275\) SNPs which are doing great, but there are some legacy data sets that have fewer SNPs. In these cases, especially with populations that don’t have as much genetic variation as some of the others there end up being disastrously high numbers of parent pairs that remain unexcluded using the Mendelian incompatibility criterion. This means a boatload of pairs to compute posterior probabilities for, and, in some data sets, it has the unfortunate consequence of causing segmentation faults (likely due to memory problems) when the backward simulation step starts up.

I have implemented a workaround for this that involve two additional filters achieved with two new options. --min-logl and --max-par-pair.

Here is what happens. First all of the parent pairs with sufficiently few Mendelian incompatibilities to an individual are recorded, and their posterior probabilities and log-likelihoods are computed and put into sorted order from largest log-likelihood ratio (the ratio of their genotype probabilities given they form a parental trio divided by the genotype probabilities given that they are not a parental trio) to smallest. Then when --min-logl R is issued, all those parent pairs with log-likelihood ratio \(> R\) are retained while the rest are discarded, where \(R\) is some real number. If you also choose --max-par-pair J, then if there remain more than \(J\) parent pairs after the LogL filter, then it retains only \(J\) of them. You can also use the --max-par-pair filter without using the --min-logl filter, and vice-versa.

When either of these options is used there is a chance that the resulting p-values and FDR values for assigned parent pairs will be lower than they would be without those options, because there are fewer chances in the backward simulation to simulate a non-parental parent pair that has a log-likelihood ratio as high or higher than the observed candidate parent pair.

A few quick tests

I ran a few tests to see how much of an effect we see. First, I ran snppit a few different times with different filter parameters on a data set that was causing problems for CRTFC (though I randomly discarded some offspring so it didn’t take quite so long to run.) The command lines used were:

snppit -f ../data/BON2018_SY11-17PBT93_SNPPIT_inputfix-Ex257assExGack3_fewer_offspring.txt  --min-logl 0.0
snppit -f ../data/BON2018_SY11-17PBT93_SNPPIT_inputfix-Ex257assExGack3_fewer_offspring.txt  --min-logl 0.0 --max-par-pair 500
snppit -f ../data/BON2018_SY11-17PBT93_SNPPIT_inputfix-Ex257assExGack3_fewer_offspring.txt  --min-logl 3.0
snppit -f ../data/BON2018_SY11-17PBT93_SNPPIT_inputfix-Ex257assExGack3_fewer_offspring.txt  --min-logl 5.0
snppit -f ../data/BON2018_SY11-17PBT93_SNPPIT_inputfix-Ex257assExGack3_fewer_offspring.txt  --min-logl 5.0 --max-par-pair 50
snppit -f ../data/BON2018_SY11-17PBT93_SNPPIT_inputfix-Ex257assExGack3_fewer_offspring.txt  --min-logl 10.0

We can read in the output from those runs and compare the FDRs between the case with --min-logl 0.0, which is the least amount of filtering, with all the other cases to see how much of an effect there is.

First, load up some packages:

library(tidyverse)
Registered S3 method overwritten by 'dplyr':
  method           from
  print.rowwise_df     
Registered S3 methods overwritten by 'ggplot2':
  method         from 
  [.quosures     rlang
  c.quosures     rlang
  print.quosures rlang
── Attaching packages ───────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
✔ ggplot2 3.1.1     ✔ purrr   0.3.2
✔ tibble  2.1.1     ✔ dplyr   0.8.1
✔ tidyr   0.8.3     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.4.0
── Conflicts ──────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
library(viridis)
Loading required package: viridisLite

Then read the parentage assignments into a big data frame.

prefixes <- c(
  "LOGL_0",
  "LOGL_0_TOP_500",
  "LOGL_3",
  "LOGL_5",
  "LOGL_5_TOP_50",
  "LOGL_10"
)
names(prefixes) <- prefixes

Results <- lapply(prefixes, function(p) {
  read_tsv(str_c("inputs/", p, "_snppit_output_ParentageAssignments.txt"), na = "---")
}) %>%
  bind_rows(.id = "filter_condition")

And then reformat that so we can compare all of these to the LOGL_0 score, but also keep all the other information in there in a long, tidy format. And only keep ones that were C_Se_Se, and only those that had a parent found in the LOGL_0 condition.

compared_0 <- Results %>%
  filter(filter_condition == "LOGL_0") %>%
  select(OffspCollection:PopName, MaxP.Pr.Relat, FDR, Pvalue) %>%
  rename(LOGL0_max_relat = MaxP.Pr.Relat,
         LOGL0_FDR = FDR,
         LOGL0_Pvalue = Pvalue) %>%
  filter(LOGL0_max_relat == "C_Se_Se" & !is.na(LOGL0_FDR)) %>%
  left_join(Results)
Joining, by = c("OffspCollection", "Kid", "Pa", "Ma", "PopName")

And now we can make a quick plot of the Pvalues:

That is about what we expect to see. If you have a super high LogL cutoff (like 10) then your P-values are much lower than with a low cutoff (of 0, for example).

This is because there are many more retained individuals with lower filter criteria. We can color each point by the Log10 of the number of non-excluded parent pairs.

Let’s look at the portion where the FDR is less than 1

g2 <- ggplot(compared_0, aes(x = LOGL0_Pvalue, y = Pvalue, colour = log10(TotPairsNonExc))) +
  geom_point() +
  scale_color_viridis_c()
g2

This will ultimately affect the FDRs.

It might also be interesting to compare the total number of parent pairs that are not excluded by Mendelian incompatibility, and the number that are non-excluded after the LogL and max-par-pair filters. We limit our focus to pairs that have a MaxP.Pr.Relat of C_Se_Se, and plot it on a log scale:

ggplot(compared_0, aes(x = TotPairsMendCompat, y = TotPairsNonExc, colour = filter_condition)) +
  geom_point() +
  geom_abline(intercept = 0, slope = 1, linetype = "dashed") +
  scale_y_log10() + 
  scale_x_log10()

NA
NA

That just serves to show that even when you filter on Logl > 0, we are still, in some cases, discarding quite a few parent pairs that are not excluded by Mendelian incompatibility.

So, what this all points out is that these two filtering options are probably best to implement for a first pass to catch any of the “SNPPIT killers”: offspring that cause problems because they have so many parent pairs that are not filtered out on the basis of Mendelian incompatibility. These individuals could be removed and then SNPPIT re-run with no Logl or max-par-pair filtering.

A small change to the output file

I’ve added three columns to the snppit_output_ParentageAssignments.txt file:

The total number of parent pairs used int the backward step is, as before, in the column TotPairsNonExc.

Note that these extra columns still appear even if the min-logl and max-par-pair options are not used. If users have scripts that pick out the values in columns of snppit_output_ParentageAssignments.txt by position, rather than column name, then those scripts will have to be updated.

LS0tCnRpdGxlOiAiTG9nTCBhbmQgUmFuayBUaHJlc2hvbGRzIGZvciBOb24tZXhjbHVkZWQgUGFyZW50IFBhaXJzIGluIFNOUFBJVCIKYXV0aG9yOiBFcmljIEMuIEFuZGVyc29uCmRhdGU6IDIwIEpVTkUgMjAxOQpvdXRwdXQ6IAogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHRydWUKLS0tCgojIE92ZXJ2aWV3IG9mIGFkZGl0aW9ucwoKSm9uIEhlc3MgYW5kIGZyaWVuZHMgYXQgQ1JURkMgYXJlIGtpY2tpbmcgYXNzIHVzaW5nIFNOUFBJVCBmb3IgUEJUIGluIHRoZQpDb2x1bWJpYSBSaXZlci4gVGhleSBoYXZlIG5vdyBnb25lIHRvICRcYXBwcm94IDI3NSQgU05QcyB3aGljaCBhcmUgZG9pbmcgZ3JlYXQsCmJ1dCB0aGVyZSBhcmUgc29tZSBsZWdhY3kgZGF0YSBzZXRzIHRoYXQgaGF2ZSBmZXdlciBTTlBzLiAgSW4gdGhlc2UgY2FzZXMsCmVzcGVjaWFsbHkgd2l0aCBwb3B1bGF0aW9ucyB0aGF0IGRvbid0IGhhdmUgYXMgbXVjaCBnZW5ldGljIHZhcmlhdGlvbiBhcyBzb21lIG9mCnRoZSBvdGhlcnMgdGhlcmUgZW5kIHVwIGJlaW5nIGRpc2FzdHJvdXNseSBoaWdoIG51bWJlcnMgb2YgcGFyZW50IHBhaXJzCnRoYXQgcmVtYWluIHVuZXhjbHVkZWQgdXNpbmcgdGhlIE1lbmRlbGlhbiBpbmNvbXBhdGliaWxpdHkgY3JpdGVyaW9uLiBUaGlzIG1lYW5zCmEgYm9hdGxvYWQgb2YgcGFpcnMgdG8gY29tcHV0ZSBwb3N0ZXJpb3IgcHJvYmFiaWxpdGllcyBmb3IsIGFuZCwgaW4gc29tZSAKZGF0YSBzZXRzLCBpdCBoYXMgdGhlIHVuZm9ydHVuYXRlIGNvbnNlcXVlbmNlIG9mIGNhdXNpbmcgc2VnbWVudGF0aW9uIGZhdWx0cwoobGlrZWx5IGR1ZSB0byBtZW1vcnkgcHJvYmxlbXMpIHdoZW4gdGhlIGJhY2t3YXJkIHNpbXVsYXRpb24gc3RlcCBzdGFydHMgdXAuCgpJIGhhdmUgaW1wbGVtZW50ZWQgYSB3b3JrYXJvdW5kIGZvciB0aGlzIHRoYXQgaW52b2x2ZSB0d28gYWRkaXRpb25hbCBmaWx0ZXJzCmFjaGlldmVkIHdpdGggdHdvIG5ldyBvcHRpb25zLiAgYC0tbWluLWxvZ2xgIGFuZCBgLS1tYXgtcGFyLXBhaXJgLgoKSGVyZSBpcyB3aGF0IGhhcHBlbnMuIEZpcnN0IGFsbCBvZiB0aGUgcGFyZW50IHBhaXJzIHdpdGggc3VmZmljaWVudGx5CmZldyBNZW5kZWxpYW4gaW5jb21wYXRpYmlsaXRpZXMgdG8gYW4gaW5kaXZpZHVhbCBhcmUgcmVjb3JkZWQsIGFuZCB0aGVpciBwb3N0ZXJpb3IKcHJvYmFiaWxpdGllcyBhbmQgbG9nLWxpa2VsaWhvb2RzIGFyZSBjb21wdXRlZCBhbmQgcHV0IGludG8gc29ydGVkIG9yZGVyIGZyb20KbGFyZ2VzdCBsb2ctbGlrZWxpaG9vZCByYXRpbyAodGhlIHJhdGlvIG9mIHRoZWlyIGdlbm90eXBlIHByb2JhYmlsaXRpZXMgZ2l2ZW4KdGhleSBmb3JtIGEgcGFyZW50YWwgdHJpbyBkaXZpZGVkIGJ5IHRoZSBnZW5vdHlwZSBwcm9iYWJpbGl0aWVzIGdpdmVuIHRoYXQgdGhleQphcmUgbm90IGEgcGFyZW50YWwgdHJpbykgdG8gc21hbGxlc3QuIFRoZW4gd2hlbiBgLS1taW4tbG9nbCBSYCBpcyBpc3N1ZWQsIGFsbCB0aG9zZQpwYXJlbnQgcGFpcnMgd2l0aCBsb2ctbGlrZWxpaG9vZCByYXRpbyAkPiBSJCBhcmUgcmV0YWluZWQgd2hpbGUgdGhlIHJlc3QgYXJlCmRpc2NhcmRlZCwgd2hlcmUgJFIkIGlzIHNvbWUgcmVhbCBudW1iZXIuICBJZiB5b3UgYWxzbyBjaG9vc2UgYC0tbWF4LXBhci1wYWlyIEpgLAp0aGVuIGlmIHRoZXJlIHJlbWFpbiBtb3JlIHRoYW4gJEokIHBhcmVudCBwYWlycyBhZnRlciB0aGUgTG9nTCBmaWx0ZXIsIHRoZW4KaXQgcmV0YWlucyBvbmx5ICRKJCBvZiB0aGVtLiAgWW91IGNhbiBhbHNvIHVzZSB0aGUgYC0tbWF4LXBhci1wYWlyYCBmaWx0ZXIKd2l0aG91dCB1c2luZyB0aGUgYC0tbWluLWxvZ2xgIGZpbHRlciwgYW5kIHZpY2UtdmVyc2EuCgpXaGVuIGVpdGhlciBvZiB0aGVzZSBvcHRpb25zIGlzIHVzZWQgdGhlcmUgaXMgYSBjaGFuY2UgdGhhdCB0aGUgcmVzdWx0aW5nIHAtdmFsdWVzIGFuZCAKRkRSIHZhbHVlcyBmb3IgYXNzaWduZWQgcGFyZW50IHBhaXJzIHdpbGwgYmUgbG93ZXIgdGhhbiB0aGV5IHdvdWxkIGJlIHdpdGhvdXQgdGhvc2Ugb3B0aW9ucywKYmVjYXVzZSB0aGVyZSBhcmUgZmV3ZXIgY2hhbmNlcyBpbiB0aGUgYmFja3dhcmQgc2ltdWxhdGlvbiB0byBzaW11bGF0ZSBhIG5vbi1wYXJlbnRhbCBwYXJlbnQKcGFpciB0aGF0IGhhcyBhIGxvZy1saWtlbGlob29kIHJhdGlvIGFzIGhpZ2ggb3IgaGlnaGVyIHRoYW4gdGhlIG9ic2VydmVkIGNhbmRpZGF0ZSBwYXJlbnQKcGFpci4KCiMgQSBmZXcgcXVpY2sgdGVzdHMKCkkgcmFuIGEgZmV3IHRlc3RzIHRvIHNlZSBob3cgbXVjaCBvZiBhbiBlZmZlY3Qgd2Ugc2VlLiAgRmlyc3QsIEkgcmFuIHNucHBpdCBhIGZldyBkaWZmZXJlbnQKdGltZXMgd2l0aCBkaWZmZXJlbnQgZmlsdGVyIHBhcmFtZXRlcnMgb24gYSBkYXRhIHNldCB0aGF0IHdhcyBjYXVzaW5nIHByb2JsZW1zIGZvciBDUlRGQwoodGhvdWdoIEkgcmFuZG9tbHkgZGlzY2FyZGVkIHNvbWUgb2Zmc3ByaW5nIHNvIGl0IGRpZG4ndCB0YWtlIHF1aXRlIHNvIGxvbmcgdG8gcnVuLikgIFRoZSBjb21tYW5kCmxpbmVzIHVzZWQgd2VyZToKYGBgc2gKc25wcGl0IC1mIC4uL2RhdGEvQk9OMjAxOF9TWTExLTE3UEJUOTNfU05QUElUX2lucHV0Zml4LUV4MjU3YXNzRXhHYWNrM19mZXdlcl9vZmZzcHJpbmcudHh0ICAtLW1pbi1sb2dsIDAuMApzbnBwaXQgLWYgLi4vZGF0YS9CT04yMDE4X1NZMTEtMTdQQlQ5M19TTlBQSVRfaW5wdXRmaXgtRXgyNTdhc3NFeEdhY2szX2Zld2VyX29mZnNwcmluZy50eHQgIC0tbWluLWxvZ2wgMC4wIC0tbWF4LXBhci1wYWlyIDUwMApzbnBwaXQgLWYgLi4vZGF0YS9CT04yMDE4X1NZMTEtMTdQQlQ5M19TTlBQSVRfaW5wdXRmaXgtRXgyNTdhc3NFeEdhY2szX2Zld2VyX29mZnNwcmluZy50eHQgIC0tbWluLWxvZ2wgMy4wCnNucHBpdCAtZiAuLi9kYXRhL0JPTjIwMThfU1kxMS0xN1BCVDkzX1NOUFBJVF9pbnB1dGZpeC1FeDI1N2Fzc0V4R2FjazNfZmV3ZXJfb2Zmc3ByaW5nLnR4dCAgLS1taW4tbG9nbCA1LjAKc25wcGl0IC1mIC4uL2RhdGEvQk9OMjAxOF9TWTExLTE3UEJUOTNfU05QUElUX2lucHV0Zml4LUV4MjU3YXNzRXhHYWNrM19mZXdlcl9vZmZzcHJpbmcudHh0ICAtLW1pbi1sb2dsIDUuMCAtLW1heC1wYXItcGFpciA1MApzbnBwaXQgLWYgLi4vZGF0YS9CT04yMDE4X1NZMTEtMTdQQlQ5M19TTlBQSVRfaW5wdXRmaXgtRXgyNTdhc3NFeEdhY2szX2Zld2VyX29mZnNwcmluZy50eHQgIC0tbWluLWxvZ2wgMTAuMApgYGAKV2UgY2FuIHJlYWQgaW4gdGhlIG91dHB1dCBmcm9tIHRob3NlIHJ1bnMgYW5kIGNvbXBhcmUgdGhlIEZEUnMgYmV0d2VlbiB0aGUgY2FzZSB3aXRoIGAtLW1pbi1sb2dsIDAuMGAsIHdoaWNoCmlzIHRoZSBsZWFzdCBhbW91bnQgb2YgZmlsdGVyaW5nLCB3aXRoIGFsbCB0aGUgb3RoZXIgY2FzZXMgdG8gc2VlIGhvdyBtdWNoIG9mIGFuIGVmZmVjdCB0aGVyZSBpcy4KCkZpcnN0LCBsb2FkIHVwIHNvbWUgcGFja2FnZXM6CmBgYHtyfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKbGlicmFyeSh2aXJpZGlzKQpgYGAKClRoZW4gcmVhZCB0aGUgcGFyZW50YWdlIGFzc2lnbm1lbnRzIGludG8gYSBiaWcgZGF0YSBmcmFtZS4KYGBge3IsIG1lc3NhZ2U9RkFMU0V9CnByZWZpeGVzIDwtIGMoCiAgIkxPR0xfMCIsCiAgIkxPR0xfMF9UT1BfNTAwIiwKICAiTE9HTF8zIiwKICAiTE9HTF81IiwKICAiTE9HTF81X1RPUF81MCIsCiAgIkxPR0xfMTAiCikKbmFtZXMocHJlZml4ZXMpIDwtIHByZWZpeGVzCgpSZXN1bHRzIDwtIGxhcHBseShwcmVmaXhlcywgZnVuY3Rpb24ocCkgewogIHJlYWRfdHN2KHN0cl9jKCJpbnB1dHMvIiwgcCwgIl9zbnBwaXRfb3V0cHV0X1BhcmVudGFnZUFzc2lnbm1lbnRzLnR4dCIpLCBuYSA9ICItLS0iKQp9KSAlPiUKICBiaW5kX3Jvd3MoLmlkID0gImZpbHRlcl9jb25kaXRpb24iKQpgYGAKCkFuZCB0aGVuIHJlZm9ybWF0IHRoYXQgc28gd2UgY2FuIGNvbXBhcmUgYWxsIG9mIHRoZXNlIHRvIHRoZSBMT0dMXzAgc2NvcmUsIGJ1dCBhbHNvIGtlZXAgYWxsIHRoZQpvdGhlciBpbmZvcm1hdGlvbiBpbiB0aGVyZSBpbiBhIGxvbmcsIHRpZHkgZm9ybWF0LiAgQW5kIG9ubHkga2VlcCBvbmVzIHRoYXQgd2VyZSBDX1NlX1NlLCBhbmQgb25seSB0aG9zZSB0aGF0CmhhZCBhIHBhcmVudCBmb3VuZCBpbiB0aGUgTE9HTF8wIGNvbmRpdGlvbi4KYGBge3J9CmNvbXBhcmVkXzAgPC0gUmVzdWx0cyAlPiUKICBmaWx0ZXIoZmlsdGVyX2NvbmRpdGlvbiA9PSAiTE9HTF8wIikgJT4lCiAgc2VsZWN0KE9mZnNwQ29sbGVjdGlvbjpQb3BOYW1lLCBNYXhQLlByLlJlbGF0LCBGRFIsIFB2YWx1ZSkgJT4lCiAgcmVuYW1lKExPR0wwX21heF9yZWxhdCA9IE1heFAuUHIuUmVsYXQsCiAgICAgICAgIExPR0wwX0ZEUiA9IEZEUiwKICAgICAgICAgTE9HTDBfUHZhbHVlID0gUHZhbHVlKSAlPiUKICBmaWx0ZXIoTE9HTDBfbWF4X3JlbGF0ID09ICJDX1NlX1NlIiAmICFpcy5uYShMT0dMMF9GRFIpKSAlPiUKICBsZWZ0X2pvaW4oUmVzdWx0cykKYGBgCgpBbmQgbm93IHdlIGNhbiBtYWtlIGEgcXVpY2sgcGxvdCBvZiB0aGUgUHZhbHVlczoKYGBge3J9CmcgPC0gZ2dwbG90KGNvbXBhcmVkXzAsIGFlcyh4ID0gTE9HTDBfUHZhbHVlLCB5ID0gUHZhbHVlLCBjb2xvdXIgPSBmaWx0ZXJfY29uZGl0aW9uKSkgKwogIGdlb21fcG9pbnQoKQpnCmBgYAoKVGhhdCBpcyBhYm91dCB3aGF0IHdlIGV4cGVjdCB0byBzZWUuICBJZiB5b3UgaGF2ZSBhIHN1cGVyIGhpZ2ggTG9nTCBjdXRvZmYgKGxpa2UgMTApCnRoZW4geW91ciBQLXZhbHVlcyBhcmUgbXVjaCBsb3dlciB0aGFuIHdpdGggYSBsb3cgY3V0b2ZmIChvZiAwLCBmb3IgZXhhbXBsZSkuIAoKVGhpcyBpcyBiZWNhdXNlIHRoZXJlIGFyZSBtYW55IG1vcmUgcmV0YWluZWQgaW5kaXZpZHVhbHMgd2l0aCBsb3dlciBmaWx0ZXIKY3JpdGVyaWEuICBXZSBjYW4gY29sb3IgZWFjaCBwb2ludCBieSB0aGUgTG9nMTAgb2YgdGhlIG51bWJlciBvZiBub24tZXhjbHVkZWQKcGFyZW50IHBhaXJzLgoKTGV0J3MgbG9vayBhdCB0aGUgcG9ydGlvbiB3aGVyZSB0aGUgRkRSIGlzIGxlc3MgdGhhbiAxCmBgYHtyfQpnMiA8LSBnZ3Bsb3QoY29tcGFyZWRfMCwgYWVzKHggPSBMT0dMMF9QdmFsdWUsIHkgPSBQdmFsdWUsIGNvbG91ciA9IGxvZzEwKFRvdFBhaXJzTm9uRXhjKSkpICsKICBnZW9tX3BvaW50KCkgKwogIHNjYWxlX2NvbG9yX3ZpcmlkaXNfYygpCmcyCmBgYAoKVGhpcyB3aWxsIHVsdGltYXRlbHkgYWZmZWN0IHRoZSBGRFJzLgoKSXQgbWlnaHQgYWxzbyBiZSBpbnRlcmVzdGluZyB0byBjb21wYXJlIHRoZSB0b3RhbCBudW1iZXIgb2YgcGFyZW50IHBhaXJzIHRoYXQKYXJlIG5vdCBleGNsdWRlZCBieSBNZW5kZWxpYW4gaW5jb21wYXRpYmlsaXR5LCBhbmQgdGhlIG51bWJlciB0aGF0IGFyZQpub24tZXhjbHVkZWQgYWZ0ZXIgdGhlIExvZ0wgYW5kIG1heC1wYXItcGFpciBmaWx0ZXJzLiBXZSBsaW1pdCBvdXIgZm9jdXMgdG8gcGFpcnMKdGhhdCBoYXZlIGEgTWF4UC5Qci5SZWxhdCBvZiBDX1NlX1NlLCBhbmQgcGxvdCBpdCBvbiBhIGxvZyBzY2FsZToKYGBge3J9CmdncGxvdChjb21wYXJlZF8wLCBhZXMoeCA9IFRvdFBhaXJzTWVuZENvbXBhdCwgeSA9IFRvdFBhaXJzTm9uRXhjLCBjb2xvdXIgPSBmaWx0ZXJfY29uZGl0aW9uKSkgKwogIGdlb21fcG9pbnQoKSArCiAgZ2VvbV9hYmxpbmUoaW50ZXJjZXB0ID0gMCwgc2xvcGUgPSAxLCBsaW5ldHlwZSA9ICJkYXNoZWQiKSArCiAgc2NhbGVfeV9sb2cxMCgpICsgCiAgc2NhbGVfeF9sb2cxMCgpCiAgCgpgYGAKClRoYXQganVzdCBzZXJ2ZXMgdG8gc2hvdyB0aGF0IGV2ZW4gd2hlbiB5b3UgZmlsdGVyIG9uIExvZ2wgPiAwLCB3ZSBhcmUgc3RpbGwsIGluIHNvbWUKY2FzZXMsIGRpc2NhcmRpbmcgcXVpdGUgYSBmZXcgcGFyZW50IHBhaXJzIHRoYXQgYXJlIG5vdCBleGNsdWRlZCBieSBNZW5kZWxpYW4gaW5jb21wYXRpYmlsaXR5LgoKU28sIHdoYXQgdGhpcyBhbGwgcG9pbnRzIG91dCBpcyB0aGF0IHRoZXNlIHR3byBmaWx0ZXJpbmcgb3B0aW9ucyBhcmUgcHJvYmFibHkgYmVzdCB0byBpbXBsZW1lbnQKZm9yIGEgZmlyc3QgcGFzcyB0byBjYXRjaCBhbnkgb2YgdGhlICJTTlBQSVQga2lsbGVycyI6IG9mZnNwcmluZyB0aGF0IGNhdXNlIHByb2JsZW1zIGJlY2F1c2UgdGhleSBoYXZlIHNvCm1hbnkgcGFyZW50IHBhaXJzIHRoYXQgYXJlIG5vdCBmaWx0ZXJlZCBvdXQgb24gdGhlIGJhc2lzIG9mIE1lbmRlbGlhbiBpbmNvbXBhdGliaWxpdHkuICBUaGVzZSBpbmRpdmlkdWFscwpjb3VsZCBiZSByZW1vdmVkIGFuZCB0aGVuIFNOUFBJVCByZS1ydW4gd2l0aCBubyBMb2dsIG9yIG1heC1wYXItcGFpciBmaWx0ZXJpbmcuCgojIEEgc21hbGwgY2hhbmdlIHRvIHRoZSBvdXRwdXQgZmlsZQoKSSd2ZSBhZGRlZCB0aHJlZSBjb2x1bW5zIHRvIHRoZSBgc25wcGl0X291dHB1dF9QYXJlbnRhZ2VBc3NpZ25tZW50cy50eHRgIGZpbGU6CgoqIGBUb3RQYWlyc01lbmRDb21wYXRgOiB0b3RhbCBudW1iZXIgb2YgcGFyZW50IHBhaXJzIHRoYXQgd2VyZSByZXRhaW5lZCBhZnRlciB0aGUgTWVuZGVsaWFuIGluY29tcGF0aWJpbGl0eSBmaWx0ZXJpbmcuCiogYFRvdFBhaXJzTWVuZEFuZExvZ0xgOiBudW1iZXIgb2YgcGFyZW50IHBhaXJzIHJlbWFpbmluZyBhZnRlciB0aGUgTWVuZGVsaWFuIGluY29tcGF0aWJpbGl0eSBfYW5kXyB0aGUgYG1pbi1sb2dsYCBmaWx0ZXIuCiogYFRvdFBhcnNNZW5kTG9nbEFuZFJhbmtgOiBudW1iZXIgb2YgcGFyZW50IHBhaXJzIHJlbWFpbmluZyBhZnRlciBNZW5kZWxpYW4gaW5jb21wYXRpYmlsaXR5LCBgbWluLWxvZ2xgLCBhbmQgYG1heC1wYXItcGFpcmAgCmZpbHRlcnMuICAKClRoZSB0b3RhbCBudW1iZXIgb2YgcGFyZW50IHBhaXJzIHVzZWQgaW50IHRoZSBiYWNrd2FyZCBzdGVwIGlzLCBhcyBiZWZvcmUsIGluIHRoZSBjb2x1bW4gYFRvdFBhaXJzTm9uRXhjYC4KCk5vdGUgdGhhdCB0aGVzZSBleHRyYSBjb2x1bW5zIHN0aWxsIGFwcGVhciBldmVuIGlmIHRoZSBgbWluLWxvZ2xgIGFuZCBgbWF4LXBhci1wYWlyYCBvcHRpb25zIGFyZSBub3QgdXNlZC4gCklmIHVzZXJzIGhhdmUgc2NyaXB0cyB0aGF0IHBpY2sgb3V0IHRoZSB2YWx1ZXMgaW4gY29sdW1ucyBvZiBgc25wcGl0X291dHB1dF9QYXJlbnRhZ2VBc3NpZ25tZW50cy50eHRgIGJ5IHBvc2l0aW9uLApyYXRoZXIgdGhhbiBjb2x1bW4gbmFtZSwgdGhlbiB0aG9zZSBzY3JpcHRzIHdpbGwgaGF2ZSB0byBiZSB1cGRhdGVkLgoKCg==