Warning<\/span>: this post is a bit wonkish. Math ahead!\u00a0<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Controlling the Covid-19 pandemic requires testing and contact tracing, but the usefulness of tests declines if getting results takes too long: a week<\/a> is no longer unusual, and there have been reports of three week waits<\/a>. Given that the standard PCR tests, which try to identify viral RNA in the sample, involve sophisticated biochemistry that requires ingredients that are in limited supply, one wonders: aren\u2019t there simpler, faster tests? There are: antigen tests, which instead look for viral proteins. But the problem is that they are not as good<\/a>: the false-positive or the false-negative rate is substantially higher than for a good PCR test. To see why that matters, and how to overcome this problem, we need to discuss what a positive test result even means, which is best done within the framework of Bayes\u2019 theorem from probability theory. I will not<\/em> discuss or even prove it here, but you can find an excellent intuitive discussion by 3blue1brown<\/a>, who made an awesome video<\/a> about this. I can\u2019t possibly compete with his clarity, so I\u2019ll just urge you to look it up. If you don\u2019t already know what Bayes' theorem is, these are 15 minutes and 45 seconds that are incredibly well spent.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> OK, here we go.\u00a0<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Assume we have a test that can identify Covid-19. Of course, at times the test gets it wrong, and there are two different ways for how this could happen: either a person who is sick is erroneously told that they are healthy (a false negative), or a healthy person is erroneously told that they are sick (a false positive). Ideally, these rates are small, but they are never zero. How do they affect how we interpret the outcome of a test?<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Let\u2019s say we have a test where both the false positive and the false negative rate are just 1%. Looks pretty good, so far. Let\u2019s say you take the test, and it comes back positive. Are you sick? Well, not with certainty<\/em>, but with what probability<\/em> are you sick? Most people would say that the probability of being sick is something like 99%. That sounds extremely intuitive, but it is entirely wrong<\/em>. And it is crucial to understand why.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> The key problem is that we have not accounted for the prevalence of the disease. If the disease is rare, then most people who are tested are in fact healthy. And even if the sick ones are found out pretty reliably, among the large number of healthy people we will accumulate a sizable number of false positives. The problem is that we can\u2019t tell the false positives from the true positives. They are just positive test results. What we want to know is: if you tested positive, what are the odds that you are actually sick? Meaning, that in the group of true and false positives, you are a true positive?<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> That\u2019s where the theorem of Bayes comes in. It permits us to answer the question \u201cwhat\u2019s the probability someone is sick, given that they tested positive?\u201d Notice that this question is something like the \u201cinverse\u201d version of what we know about the test quality, namely, \u201cwhat\u2019s the probability that someone tests positive, given that they are sick?\u201d The key thing is that these two probabilities are not the same<\/em>, but that Bayes' theorem helps us to get one from the other. It says:<\/p>\n $$P(C|+) = \\frac{P(+|C)\\;P(C)}{P(+)} \\ .$$<\/p>\n Let's parse this: $P(C)$ is the probability of being sick (the \u201c$C$\u201d reminds us of \u201cCovid-19\u201d), and $P(+)$ is the probability of ending up with a positive test result. The expression $P(C|+)$ is the so-called conditional probability<\/em> of being sick, given that<\/em> one tested positive, and the expression $P(+|C)$ is the inverse<\/em> conditional probability of testing positive, given that<\/em> one is sick. Just remember that the vertical bar is pronounced \u201cgiven that<\/em>\u201d, and it is easy to read.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Sidenote<\/strong>: If you're still uncomfortable about the two conditional probailities not being equal, imagine rolling a dice and being interested in the two outcomes \"I got a 6\", versus \"I got an even (\"E\") number\". I'm sure you can very quickly convince yourself that $P(6|\\textrm{E})=1\/3$ and $P(\\textrm{E}|6)=1$, and so they are obviously not the same.<\/span><\/p>\n<\/blockquote><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> To answer our question, let us look at the following so-called probability tree:<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Here\u2019s how we read it. Start from the left, where you meet a first bifurcation: a person could either have Covid-19 ($C$), or not ($\\overline{C}$) (the bar over the letter is a shorthand to denote negation: not<\/em> Covid-19.) The letters next to the arrows indicate the probability of that happening: $p$ being the probability of the person being sick, and $1-p$ the probability for being healthy. Each of these two outcomes now has a second branch point: if we now test, will the test come back positive or not? This depends on whether the person has the disease or not. If the person is sick, a positive outcome happens with probability $t_+$, the true<\/em> positive rate, while the probability of accidentally getting a negative result is $1-t_+$; if instead the person is healthy, a positive outcome can still happen, but with the (hopefully much smaller) false<\/em> positive rate $f_+$, while the probability for a negative outcome, $1-f_+$, is ideally much higher.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Sidenote<\/strong>: there's another bit of lingo possibly worth knowing about. In medical diagnostics, the true positive rate is called the \"sensitivity<\/em>\" of the test: the ability to pick up the disease if it's there. And the complement of the false positive rate ($1-f_+=t_-$, the true negative rate!) is called the \"specificity<\/em>\" of the test: it's ability to correctly identify those who don't have the disease, or alternatively, to not be fooled into a positive result by something that looks similar (like a similar looking version of a different virus).<\/span><\/p>\n<\/blockquote><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> We're now ready for our first big question: what is the probability that a person who enters the test center gets a positive test? There are two ways for how this could happen: the person could be sick, and the test says so; or the person could be healthy, but the test gets it wrong. The probability for each of these scenarios is calculated by multiplying the probabilities \u201calong the way\u201d, and the total probability is merely the sum of those two probabilities. We hence find:<\/p>\n $$\\begin{align}P(+) &= P(C) P(+|C) + P(\\overline{C}) P(+|\\overline{C}) \\\\[1em] &= p t_+ + (1-p) f_+ \\ . \\end{align}$$<\/p>\n We now have everything we need to calculate the odds of a person being sick if the test comes back positive. Using Bayes\u2019 theorem, we get<\/p>\n $$\\begin{align}P(C|+) &= \\frac{P(+|C)P(C)}{P(+)}\\\\[0.5em] &= \\frac{p t_+}{p t_+ + (1-p) f_+}\\\\[0.5em] &= \\frac{1}{1+\\displaystyle\\frac{1-p}{p}\\times\\displaystyle\\frac{f_+}{t_+}} \\ .\\end{align}$$<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Ideally, we want this probability to be large. This is the case when the second term in the denominator is small, or, equivalently, if $(1-p) f_+ \\ll pt_+$. Now, if the disease is rare, $p$ is small, and then $1-p$ is close to one. Also, if the test is good, the true positive rate is close to one. Making these two simplifying assumptions, this inequality becomes $f_+ \\ll p$, which says that for a good test the false positive rate has to be small compared to the prevalence of the disease<\/em>. It's not enough to be \u201csmall compared to 1\u201d. It must be small compares to $p$, and this could be a difficult thing to achieve if the disease is rare.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Let us now apply this to the Covid-19 case. Assume we have a pretty good test, with 1% false positive and false negative rate, and let us consider the case that 5% of the people entering the testing site actually have Covid-19. If one of them gets a positive test result, what are the odds they are actually sick? From what we\u2019ve seen above in the general case, we simply need to put in numbers:<\/p>\n $$\\begin{align}P(C|+) &= \\frac{1}{1+\\displaystyle\\frac{(1-p) f_+}{p t_+}}\\\\[0.5em] &= \\frac{1}{1+\\displaystyle\\frac{0.95\\cdot0.01}{0.05\\cdot0.99}}\\\\[1em] &\\approx 84\\% \\ .\\end{align}$$<\/p>\n Does that strike you as surprising? The test seems really good, but the odds of having Covid-19 after testing positive are significantly smaller than the 99% one might initially have expected. But it gets worse: what if the test isn\u2019t quite so good? Specifically, antigen tests might have false positive and false negative rates in the 10% range (or worse, but let\u2019s stick with 10%). If we use these numbers instead, what are now<\/em> the odds that a positive Covid-19 tests actually correctly identifies the disease? Let\u2019s calculate:<\/p>\n $$\\begin{align}P(C|+) &= \\frac{1}{1+\\displaystyle\\frac{(1-p) f_+}{p t_+}}\\\\[0.5em] &= \\frac{1}{1+\\displaystyle\\frac{0.95\\cdot0.10}{0.05\\cdot0.90}}\\\\[1em] &\\approx 32\\% \\ .\\end{align}$$<\/p>\n That\u2019s pretty terrible. Yes, you might get your answer in minutes, but if the odds of being sick is actually only about 1 in 3, we\u2019d be quarantining a lot of people for a long time (and scare them). And of course when we now contact-trace, we\u2019ll run into a lot more people who will be falsely identified as sick. The speed of less reliable tests comes at a huge price.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> However, there is a way to use cheap tests that offers a true benefit, and that is to use them as a pre-test<\/em>, before we decide to actually run a more reliable but much slower test. Let\u2019s see how that works. A simple way to do that is again with a probability diagram:<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> We again have the Covid-19 versus no Covid-19 branch; but before we do a good<\/em> test, we do a quick and cheap pre-test<\/em>. That test does not yet have a positive or negative outcome, just a \u201csuspicious\u201d ($S$) and a \u201cnot suspicious\u201d ($\\overline{S}$) one. The true and false suspicious rates, $t_S$ and $f_S$, are like the true and false positive rates of a poor test, but they only measure \"suspicion\". Now, only if a person is identified as \u201csuspicious\u201d\u2014and we know that within minutes, which is the benefit of a cheap and quick test\u2014do we follow up with a full test, that strives to go for a \u201cdefinitive\u201d positive or negative. If the person is not suspicious, they can go.\u00a0<\/p><\/div>\n\t\t\t<\/div> There\u2019s a small catch, though, worth pointing out: The true and false positive rates we write at the last set of branches are not what we previously meant by the true and false positive rates for the better test. The reason is that we're not testing everyone. We're only testing the people who are already marked as \"suspicious\", and so what we rather have is the conditional probabilities<\/em> for true or false positives, given that the person was suspicious. That's why we added the additional \"S\" to the symbol.<\/p>\n Now, if whatever gave rise to this \"suspicious label\" is entirely independent of the subsequent good test, then the conditional rates are identical to our previous rates (i.e.<\/em>, the ones without the extra \"S\" label). But it is more likely that there is a connection: if the pre-test declares a person as \"suspicious\", it's quite possible that the subsequent test will have an easier time to correctly identify the disease, or avoid an incorrect branding of a healthy individual that has been declared \"not suspicious\". But it is also possible that the conditional probabilities are smaller than the non-conditional ones. For instance, if the \"suspicious\" test is particularly sensitive to some aspect of the disease that the subsequent good test is particularly likely to be confused about, then that good test might be performing worse (compared to the case where we didn't actually pre-select for such a confusing situation).<\/p>\n For now, let us simply make sure we understand what these rates mean and hope that in a real-world scenario someone takes the time to determine them!\u00a0<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> We\u2019re now ready to calculate the probability that in the two-stage testing scenario a person comes back with a positive test:<\/p>\n $$\\begin{align}P(+) &= pt_S t_{S+} + (1-p)f_S f_{S+}\u00a0\\ .\\end{align}$$<\/p>\n And this is what we need to get to the final question: what is the probability that a person with a positive test result actually has Covid-19? Using Bayes\u2019 formula, we get<\/p>\n $$ Fun fact: observe that compared to the single-step test, which had the ratio $f_+\/t_+$ in the denominator, this two-step test instead features the ratio $f_Sf_{S+}\/t_St_{S+}$.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Time to put in numbers. Let us assume that the second test is the good one, and that the first test is mediocre, with false rates of 10%. The only thing we need to know is: what are the rates $t_{S+}$ and $f_{S+}$? The first is the true positive rate of a sick person who has been correctly identified as \u201csuspicious\u201d; the second is the false positive rate of someone who has incorrectly been identified as \"suspicious\". As we had mentioned before, these need not<\/em>\u00a0coincide with the true and false positive rates, $t_+$ and $f_+$, from our initial scenario (i.e.<\/em>, the one without a pre-test); but in order to not complicate matters even further, let us assume that they do, and let us simply again use 99% for the true positive rate and 1% for the false positive rate.<\/p>\n We can now put in numbers:<\/p>\n $$\\begin{align}P(C|+) &= \\frac{1}{ 1+\\displaystyle\\frac{0.95\\times 0.1\\times 0.01}{0.05\\times 0.9\\times 0.99}}\\\\[1em] &= 97.9\\% \\ . \\end{align}$$<\/p>\n We see that the probability has quite significantly increased, compared to just doing the good test alone. That is maybe not surprising, given that we have two test results to rely on, and even though the poor test is not very good, it adds a bit of extra confidence.<\/p>\n But is this always true?<\/p>\n One can indeed show: yes. Provided only that the pre-test satisfies one quite minimal sanity check: the true suspicious rate must be larger than the false suspicious rate<\/em>, $t_S>f_S$. If that is not the case, then we deliberately send healthy people to get a full test and send sick people home without being tested\u2014not a smart condition for getting a good overall test!<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Interestingly, while the pre-test always improves matters, the extent<\/em> of improvement depends on the situation, for instance the probability of the disease. The following plot illustrates this:<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> This graph shows $P(C|+)$ as a function of the disease probability $p$ if one only does the good test (black curve), if one does the good test after a pre-test (blue curve), and the difference between these two probabilities (red curve). For illustration purposes we assume that the \"good\" test is actually quite poor, with both false rates being 5%, and the pre-test having false rates of 10%. If the prevalence of the disease is also 5%, then the good test alone gives\u00a0$P(C|+)=50\\%$, which is pretty lousy. But with an extra pre-test this is bumped to an\u00a0amazing<\/em>\u00a0$P(C|+)=90\\%$, which corresponds to a single good test with false rates as low as 0.58%! A poor and a lousy test in series<\/em> make an impressive test!<\/p>\n The biggest improvement would be seen if the disease probability were given by $p=1.7\\%$, where we get a shift by 50 percentage points. For $p=5\\%$ we \"only\" get a shift of 40 percentage points. Obviously, one could now try to adjust the quality of the tests such that the maximum is hit at the actual prevalence of the disease!<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Getting a more accurate overall test is very pleasant. But there is also a second<\/em> advantage of this two-tier procedure:\u00a0we can cut down on the number of slow and expensive full tests. To see how this works, let us check how many good tests are even done. What is the probability that a person entering the testing center will also be subjected to a good (and hence slow and expensive) test? Evidently, this is the same as the probability that that person will be identified as \u201csuspicious\u201d:<\/p>\n $$\\begin{align}P(S) &= pt_S + (1-p)f_S\\\\[1em] &= 0.05\\times 0.9+0.95\\times 0.1\\\\[1em] &= 14\\%\\ . \\end{align}$$<\/p>\n This means that only about 1 in 7 people are given the slow test (with our example numbers from above), and so much fewer slow tests need to be done. That frees up resources, and that could help to make the test overall faster (say, because labs are no longer overwhelmed with tests). Hence, here we have a way to make the system more efficient by including a pre-test that filters out those that likely don\u2019t need a full test.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Unfortunately, this system also has a downside: Since the poor test has a higher false negative rate as the good test, we will release more people back into the community who are actually sick. How many? If we apply the good test to everyone, then in this case the probability of sending somebody back who is sick is<\/p>\n $$P(-|C) = 1-t_+ = 1-0.99 = 1\\%\\ .$$<\/p>\n In the two-tier test we must add all those sick people who were erroneously identified as not suspicious, and therefore never received the full test. This gives<\/p>\n $$ <\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> This is not good. The likelihood of sending sick people back into the community, where they might infect others, is ten times as high<\/em> as if we had done a full test. Is this acceptable? This might depend on circumstances. But recall that if we had<\/em> actually done a full test, we\u2019d also<\/em> have sent these people back into the community, and then they might only have learned a week later that they really must properly quarantine. Absent that conclusive answer, will they preemptively quarantine? Maybe not. Given that, it might be better to send these people back after a cheap test, but then simply check people with this improved protocol more regularly<\/em>. After all, what looks like a 2-stage test is often really a 3-stage test: the people who enter the testing center often already have a suspicion<\/em>, and that\u2019s why they come in the first place. Say, they have a cough or a fever. Those coughing and feverish people who were deemed \u201cnot suspicious\u201d, but who are actually sick, and who we nevertheless sent back into the community, might continue to have these symptoms and feel that something is off. And, truth be told, we haven't told them they are negative; we just told them that a cheap pre-test<\/em> has not found them \"suspicious\". Now that we have removed a lot of strain on the system, we might actually be able to test them again,\u00a0 just a few days later (much before the current timeline of a week for a full result), and get another good chance at catching the disease.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Evidently, models of this type can be extended a lot. We could add levels for symptoms, or we could think harder about how the two tests interact. The upshot is that there\u2019s a lot<\/em> of clever procedures we can do that exploit the mechanics of testing and probability. We should not be shy to think hard about this. Or at least listen<\/em>, when epidemiologists have found a better way to test and contact trace. After all, it is in everybody\u2019s interest to beat this virus as soon as possible.<\/p><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> Markus Deserno is a professor in the Department of Physics at Carnegie Mellon University. His field of study is theoretical and computational biophysics, with a focus on lipid membranes.<\/em><\/p><\/div>\n\t\t\t<\/div>\n
![\"\"](\"http:\/\/research.phys.cmu.edu\/biophysics\/wp-content\/uploads\/sites\/2\/2020\/08\/poor-test-good-pretest-figure-1.jpg\" "\\"poor-test-good-pretest-figure-1\\"")
<\/span>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div>\n
![\"\"](\"http:\/\/research.phys.cmu.edu\/biophysics\/wp-content\/uploads\/sites\/2\/2020\/08\/poor-test-good-pretest-figure-with-pretest.jpg\" "\\"poor-test-good-pretest-figure-with-pretest\\"")
<\/span>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div>
\\begin{align}
P(C|+) &= \\frac{P(+|C)P(C)}{P(+)}\\\\[0.5em]
&= \\frac{p t_S t_{S+}}{p t_S t_{S+}+(1-p) f_S f_{S+}}\\\\[0.5em] &= \\frac{1}{ 1+\\displaystyle\\frac{1-p}{p}\\times\\frac{f_S f_{S+}}{t_S t_{S+}}} \\ .
\\end{align}
$$<\/p>\n![\"\"](\"http:\/\/research.phys.cmu.edu\/biophysics\/wp-content\/uploads\/sites\/2\/2020\/08\/impact-of-pretest.jpeg\" "\\"impact-of-pretest\\"")
<\/span>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div>
\\begin{align}
P(-|C)
&=
t_S(1-t_{S+}) + (1-t_S)\\\\[1em]
&=
0.9\\times(1-0.99)+(1-0.9)\\\\[1em]&= 10.9\\% \\ .
\\end{align}
$$<\/p>\n![\"\"](\"http:\/\/research.phys.cmu.edu\/biophysics\/wp-content\/uploads\/sites\/2\/2020\/07\/me-Pittsburgh-Airport-300x300-1.jpg\" "\\"me-Pittsburgh-Airport-300x300\\"")
<\/span>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t