Bayes’ Theorem and Bias

I wonder how many times I’ll say this. This will be my last Bayes’ theorem post. At this point a careful reader should be able to extract most of the following post from the past few, but it is definitely worth spelling out in detail here. We’ve been covering how academics have used Bayes’ theorem in their work. It is also important to see how Bayes’ theorem could be useful to you in your own life.

For this post I’m going to take as my starting point that it is better in the long run for you to believe true things rather than false things. Since this is an academic blog and most academics are seeking truth in general, they hold to some sort of rational or skeptic philosophy. Whole books have been written defending why this position will improve society, but wanting to believe true things shouldn’t be that controversial of a position.

Honestly, to people who haven’t spent a lot of time learning about bias, it is probably impossible to overestimate how important a role it plays in making decisions. Lots of well-educated, logical, careful people can look at both sides of the evidence of something and honestly think they are making an objective decision about what is true based on the evidence, but in reality they are just reconfirming a belief they made for totally irrational reasons.

We’ve all seen this type of picture before:

Even though you know the following things:

1. What the optical illusion (i.e. bias) is.
2. How and why it works.
3. The truth, which is easily verified using a ruler, that the lines are the same length.

This knowledge does not give you the power to overcome the illusion/bias. You will still continue to see the lines as different lengths. If bias can do this for a sense as objective as sight, think about how easily tricked you can be if you go off of intuition or feelings.

This exercise makes us confront a startling conclusion. In order to form a true belief, we must use the conclusion that looks and feels wrong. We must trust the fact we came to through the verifiably more objective means. This is true of your opinions/beliefs as well. You probably have false beliefs that will still look and feel true to you even once you’ve changed your mind about them. You need to trust the evidence and arguments.

A Bayesian analysis of this example might run as follows. You have the belief that the lines are different lengths from looking at it. In fact, you could reasonably set the prior probability that this belief is true pretty high because although your eyesight has been wrong in the past, you estimate that around 99% it wouldn’t make such an obvious and large error. The key piece of evidence you acquired is when you measured this with a ruler. You find they are the same length. This evidence is so strong in comparison with your confidence in your eyesight that it vastly outweighs the prior probability and you confidently conclude your first belief was false.

You probably came to many of the beliefs you have early on in life. Maybe your parents held them. Maybe your childhood friends influenced you. Maybe you saw some program on TV that got you thinking a certain way. In any case, all of these are bad reasons to believe something. Now you’ve grown up, and you think that you’ve re-evaluated these beliefs and can justify them. In reality, you’ve probably just reconfirmed them through bias.

Once you’ve taken a position on something, your brain has an almost incomprehensible number of tricks it can do in order to prevent you from changing your mind. This is called bias and you will be totally unaware of it happening. The rational position is to recognize this happens and try to remove it as much as possible in order to change an untrue belief to a true belief. Trust me. If done right, this will be a painful process. But if you want true beliefs, it must be done and you must be willing to give up your most cherished beliefs since childhood even if it means temporary social ostracization (spell check tells me this isn’t a real word, but it feels right).

What this tells us is that if we really want true beliefs we need to periodically revisit our beliefs and do a thorough sifting of the evidence in as objective a way as possible to see if our beliefs have a chance at being true.

Since there are literally thousands of cognitive biases we know about, I can’t go through all the ones you might encounter, but here are a few. One is confirmation bias. When you look at evidence for and against a belief you will tend to remember only the evidence that confirmed your already held belief (even if the evidence against is exponentially bigger!). It is difficult to reiterate this enough, but you will not consciously throw the evidence out. You will not be aware that this happened to you. You will feel as if you evenly weighed both sides of the evidence.

One of my favorite biases that seems to receive less attention is what I call the many-against-one bias (I’m not sure if it has a real name). Suppose you have three solid pieces of evidence for your belief. Suppose the counter-evidence is much better and there are seven solid pieces of it. When you look through this, what you will tend to do is look at the first piece of evidence and think, “Well, my side has these three pieces of evidence and so although that looks good it isn’t as strong as my side.” Then you move on to piece of counter-evidence two and do the same thing.

All of a sudden you’ve dismissed tons of good evidence that when taken together would convince you to change your mind, but since it was evaluated separately in a many-against-one (hence the name!) fashion you’ve kept your old opinion. Since you can’t read all the counter-evidence simultaneously, and you probably have your own personal evidence well-reinforced, it is extremely difficult to avoid this fallacy.

And on and on and on and on and on … it goes. Seriously. This should not be thought of as “bad” or something. Just a fact. It will happen, and you will not be aware of it. If you just simply look at both sides of the argument you will 99.99% of the time just come out believing the same thing. You need to take very careful precautions to avoid this.

Enter Bayes’ theorem. Do not misconstrue what I’m saying here as this being a totally objective way to come to the truth. This is just one way that you could try as a starting point. Here’s how it works. You take a claim/belief which we call B. Now you look at the best arguments and evidence for the claim you can find. You write each one down, clearly numbered, with lots of space between. Now you go find all the best counterarguments and evidence you can find to those claims and write those down next to the original ones. Now do the exact same thing with the best arguments/evidence you can find against the claim/belief.

One at a time you totally bracket off all your feelings and thoughts about the total question at hand. Just look at evidence 1 together with its counter-evidence. Supposing the claim is true, what are the chances that you would have this evidence? This is part of your P(E|B) calculation. Don’t think about how it will affect the total P(B|E) calculation. Stay detached. Find people who have the opposite opinion as you and try to convince them of your number just on this one tiny thing. If you can’t, maybe you aren’t weighting it right.

Go through every piece of evidence this way weighing it totally on its own merits and not in relation to the whole argument. Having everything written down ahead of time will help you overcome confirmation bias. Evaluating the probabilities in this way one at a time will help you overcome the many-against-one bias (you’ll probably physically feel this bias when you do it this way as you start to think, “But it isn’t that good in relation to this.”) This will also overcome numerous other biases, especially ones involving poor intuitions about probability. But do not think you’ve somehow overcome them all, because you won’t.

One of the hardest steps is then to combine your calculations into Bayes’ theorem. You should think about whether or not pieces of evidence are truly independent if you want a proper calculation. But overall you’ll get the probability that your belief is true given the evidence, and it will probably be pretty shocking. Maybe you were super confident (99.99% or something) that there was no real reason to doubt it, but you find out it is more like 55%.

Maybe something you believe only has a 5% chance of being true and you’ve just never weighed the evidence in this objective a way. You need to either update what you think is true, or at very least if it still seems to be able to go either way, be much more honest about how sure you are. I hope more people start doing this as I am one of those people that think the world would be a much better place if people stopped confidently clinging to their beliefs taught to them from childhood.

Changing your mind should not have the cultural stigma it does. Currently people who change their minds are perceived as weak and not knowing what they are talking about. At very least, they give the impression that since their opinion changes it shouldn’t be taken seriously as it might change again soon. What needs to happen is that we come to recognize the ability to change ones beliefs as an honest endeavor, having academic integrity, and is something that someone who really seeks to hold true beliefs does frequently. These people should be held up as models and not the other way around.

Bayesian vs Frequentist Statistics

I was tempted for Easter to do an analysis of the Resurrection narratives in some of the Gospels as this is possibly even more fascinating (re: the differences are starker) than our analysis of the Passion narratives. But we’ll return to the Bayesian stuff. I’m not sure what more to add after this discussion, so this topic might end. I feel like continually presenting endless examples of Bayesian methods will get boring.

Essentially everything in today’s post will be from Chapter 8 of Nate Silver’s book The Signal and the Noise (again from memory so hopefully I don’t make any major mistakes and if so don’t think they are in the book or anything). I should say this book is pretty good, but a large part of it is just examples of models which might be cool if you haven’t been thinking about this for awhile, but feels repetitive. I still recommend it if you have an interest in how Bayesian models are used in the real world.

Today’s topic is an explanation of essentially the only rival theory out there to Bayesianism. It is a method called “frequentism.” One might refer to this as “classical statistics.” It is what you would learn in a first undergraduate course in statistics, and although it still seems to be the default method in most fields of study, recently Bayesian methods have been surging and may soon replace frequentist methods.

It turns out that frequentist methods are newer and in some sense an attempt to replace some of the wishy-washy guess-work of Bayesianism. Recall that Bayesianism requires us to form a prior probability. To apply Bayes’ theorem we need to assign a probability based on … well … prior knowledge? In some fields like history this isn’t so weird. You look at similar cases that have been examined already to get the number. It is a little more awkward in science because when calculating P(C|E) the probability a conjecture is true given the evidence, you need to calculate P(C) which is your best guess at the probability your conjecture is true. It feels circular or like you can rig it so that you assume your experiment into a certain conclusion.

The frequentist will argue that assigning this probability involves all sorts of bias and subjectivity on the part of the person doing the analysis. Now this argument has been going in circles for years, but we’ve already addressed this. The Bayesian can just use probabilities that have a solid rationale that even opponents of the conclusion will agree to, or could make a whole interval of possible probabilities. It is true that the frequentist has a point, though. The bias/subjectivity does exist and an honest Bayesian admits this and takes precaution against it.

The frequentist method involves a rather simple idea (that gets complicated fast as anyone who has taken such a course knows). The idea is that we shouldn’t stack the odds for a conclusion by subjectively assigning some prior. We should just take measurements. Then, only after objective statistical analysis, should we make any such judgments. The problem is that when we take measurements, we only have a small sample of everything. We need a way to take this into account.

To illustrate using an example, we could do a poll to see who people will vote for in an election. We’re only going to poll a small number of people compared to everyone in the country. But the idea is that if we use a large enough sample size we can assume that it will roughly match the whole population. In other words, we can assume (if it was truly random) that we haven’t accidentally gotten a patch of the population that will vote significantly differently than the rest of the country. If we take a larger sample size, then our margin of error will decrease.

But built into this assumption we already have several problems. First is that hidden behind the scenes is that we must assume the voting population falls into some nice distribution for our model (for example a normal distribution). This is actually a major problem, because depending on what you are modelling there are different standards for what type of distribution to use. Moreover, we assume the sampling was random and falls into this distribution. These are two assumption that usually can’t be well-justified (at least until well after the fact when we see if its predictive value was correct).

After that, we can figure out what our expected margin of error will be. This is exactly what we see in real political polling. They give us the results and some margin of error. If you’ve taken statistics you’ve probably spent lots of time calculating these so-called “confidence intervals.” There are lots of numerics such as p-values to tell you how significant or trust-worthy the statistics and interval are.

Richard Carrier seems to argue in Proving History that there isn’t really a big difference between these two viewpoints. Bayesianism is just epistemic frequentism. They are just sort of hiding the bias and subjectivity in different places. I’d argue that Bayesian methods are superior for some simple reasons. First, the subjectivity can be quantified and put on the table for everyone to see and make their own judgments about. Second, Bayesian methods allow you to consistently update based on new evidence and takes into account that more extraordinary claims require more extraordinary evidence. Lastly, you are less likely to make standard fallacies such as the correlation implies causation fallacy.

For a funny (and fairly accurate in my opinion) summary that is clearly advocating for Bayesian methods see this:

Good Friday: A Thematic Analysis of the Passion Narrative as Presented in Mark and Luke

Since I already cracked open this door and was not punished by a flood of angry comments, I guess I’ll throw the door wide open and see what happens. This is one of my favorite non-math topics to read about, but I’ve always had a policy of keeping my posts as non-controversial as possible. I guess violating this rule once won’t be that bad. In honor of Easter, I’ll do a literary analysis of the passion narrative as presented in two of the Gospels.

The type of thing I’m going to do gets me in trouble with both the literalists and with fellow atheists. Most atheists believe that everyone should have a fairly good acquaintance with the Bible, and many think you should actually read the thing once. When I express that I think a lot can be gained by studying it as a work of literature, many scoff at that as nonsense. I aim to point out that by doing proper textual analysis you can learn a lot of interesting things.

Everything I write here should be attributed to either Bart Ehrman or Richard Carrier (and their many sources), unless you find some error. I’ve read several books by both of them and seen many debates and lectures they’ve given. I’m going to go almost entirely off of memory (and the text in the Bible itself) and hence will not be able to cite where I learned which interpretations. Both people mentioned are absolutely great Biblical scholars and if you find this post interesting you should look up their books which are excellent. I think essentially everything here should be in Jesus, Interrupted by Ehrman.

If you grew up in some Christian denomination, then you probably have this idea of what “the” passion narrative is. What I mean by this is that even if you’ve read all four accounts as given in the four Gospels, your brain has probably meshed them into one coherent piece. If asked to name any key difference between the four accounts you may have trouble coming up with any or might think they are all basically the same. This is not your fault, because it is probably what you were taught.

The reality is if you closely read any of the four accounts of any given particular aspect of Jesus’ life, you’ll find that there are not only discrepancies, but almost every single detail is in direct contradiction with some other detail of a different account. Ah. But the Christian retorts that even though the details are different the overall main point and content of the story is the same. The purpose of this post is to show you that not only is this false, but it is false for very good reasons.

The reason there are discrepancies even in the main overall points of the stories is that the authors want you to get different things out of it. It may be possible to shove interpretations together in an attempt to make some coherent overall story, but to do so is just wrong-headed. You’ll miss what the authors are trying to tell you by doing this.

Of course, Christians don’t like this idea because implicit in what I’m saying is that these are not meant to be read as historical documents describing what actually happened, but as long-form parables (or as I’ve heard Carrier say, “meta-parables”) with Jesus as the main character whose details are literary devices meant to have theological interpretation. Although this isn’t a well-known idea, it is in fact what the majority of historians (including Christians) believe (see The Homeric Epics and the Gospel of Mark by Dennis MacDonald for this idea fleshed out to a whole book).

Alright, on to the details (I can already tell this is going to be like 2000 words or something, but what else am I going to do while I sit in an airport for two hours). Let’s compare the death of Jesus as portrayed by Mark and Luke (I use the term “by” colloquially to mean “as written in the book of Mark and Luke respectively” because the authors were anonymous). The facts: In Mark the scene is solemn and tragic. Jesus says nothing and is mocked by everyone as he travels to his execution. The only words spoken by Jesus appear right before his death when he says “My God, my God, why have you forsaken me?” He dies. Then the veil at the temple rips. Then the centurion says that he must have been the son of God.

The two literary details to pay attention to are the following. First, Jesus seems to feel totally betrayed through the whole scene. He even asks why God has forsaken him. He doesn’t seem to realize what is happening. Right after his death the veil rips and the centurion tells you how to interpret this detail. The ripping of the veil (or curtain according to some translations) symbolizes that it was a Jewish ritual atonement. It seems clear that the whole scene is a literary construction to make a theological point. Jesus was an innocent lamb being lead unknowingly to a sacrificial atonement ritual.

Mark emphasizes the suffering and the feeling that God has left you. This was written to put the early persecution of Christians into context. It is those who suffer and are meek who will inherit the Earth. This theme is consistent throughout Mark. It is re-emphasized by the fact that he has women find the empty tomb later on as the last thing to happen in the story. It seems deliberate to show that women (who had less status) are the ones to learn of Jesus’ resurrection, and that this is the last event of the book (assuming that Mark actually ends at 16:8 which we won’t get into since books have been written on this debate, e.g. Perspectives on the Ending of Mark: Four Views).

In Luke we get a much different picture. Jesus is not silent while traveling to his execution. He comforts women on the way (tells them not to cry). He asks God to forgive the soldiers nailing him to the cross. He is not excessively mocked in the same way. Jesus tells one of the people being crucified with him that he will see him in heaven shortly. In other words, Jesus seems perfectly aware of his innocence and what is going on and why. He is not concerned that God has left him and seems to be in direct communion with God the whole time.

He even says “Into your hands I commend my spirit.” I repeat, there is no cry of being forsaken in this account at all. I was probably 20 before I realized this. I grew up believing (like most other Christians) that both are said. In reality one is said in one account for a very specific reason and the other is said in another account for a different reason. To combine these into one “historical” account is to completely miss the point of both authors.

In Luke, the temple curtain rips while Jesus is still alive. The centurion does not proclaim him the son of God in this version. Thus our earlier interpretation of this symbol is absolutely impossible here. A common interpretation of this is that the ripped curtain symbolizes God’s rejection of the Jewish system of worship. Sometimes said that the Old Covenant was repealed and Jesus brought the New Covenant.

Corroboration for this evidence is that Luke seems to be referencing Hebrews “By this the Holy Spirit indicates that the way into the holy places is not yet opened as long as the first section is still standing.” Or Jesus’ prophecy earlier in Luke that as long as the temple stands it signifies the continuation of the Old Covenant. Or even later in Acts (written by the same author as Luke) that God left the temple.

If again we take Luke’s account as a long-form parable, the moral does not seem to be that God is there even during great suffering and persecution and even when you feel forsaken by God. The moral in Luke seems to be that during times of suffering you should stay calm and confident of your future in heaven.

Of course the three hours of darkness occur in both, and the symbolism of that is pretty obvious. But remember, we can say with great certainty that this event never happened in history, so we are forced to think of it as a purely symbolic detail. A much more realistic interpretation of what the Gospels are, given this information, is that they are merely literary constructions and not historical accounts to teach the particular author’s theological point-of-view of this new emerging religion.

If you are a Christian and are persuaded of some level of historicity of the Gospels, the takeaway of this post should be the following. At very least, when you hear something in church you should go see what each of the four Gospels say about it separately by carefully comparing the details. Try to figure out the author’s intent rather than blindly taking some mixed interpretation that was presented to you. Question what they tell you and look it up for yourself. You’ll probably be surprised at what you find, and at very least you’ll find much richer theological interpretations of the authors.

Now to tie this back to the beginning. When I point out that almost every detail provided in the New Testament is contradicted somewhere else, the most common pushback I get is that “these are minor details, but the Bible is consistent on its main points and overall story it tells.” I guess you can decide that for yourself, but I’ll leave you with the this question. You could probably extract three major theological questions from the above analysis that get answered differently, but it seems to me that the most important theological question in all of Christianity is “Why did Jesus die?” If you read Mark you get one answer. If you read Luke you get a different answer. Should this be consider a minor detail?

Bayesianism in the Philosophy of Math

Today I’ll sketch an idea that I fist learned about from David Corfield’s excellent book Towards a Philosophy of Real Mathematics. I read it about six years ago while doing my undergraduate honors thesis and my copy is filled with notes in the margins. It has been interesting to revisit this book. What I’m going to talk about is done in much greater detail and thoroughness with tons of examples in that book. So check it out if this is interesting to you.

There are lots of ways we could use Bayesian analysis in the philosophy of math. I’ll just use a single example to show how we can use it to describe how confident we are in certain conjectures. In other words, we’ll come up with a probability for how plausible a conjecture is given the known evidence. As usual we’ll denote this P(C|E). Before doing this, let’s address the question of why would we want to do this.

To me, there are two main answers to this question. The first is that mathematicians already do this colloquially. When someone proposes something in an informal setting, you hear phrases like, “I don’t believe that at all,” or “How could that be true considering …” or “I buy that, it seems plausible.” If you think that the subject of philosophy of mathematics has any legitimacy, then certainly one of its main goals would be to take such statements and try to figure out what is meant by them and whether or not they seem justified. This is exactly what our analysis will do.

The second answer is much more practical in nature. Suppose you conjecture something as part of your research program. As we’ve been doing in these posts, you could use Baye’s theorem to give two estimates on the plausibility of your conjecture being true. One is giving the most generous probabilities given the evidence, and the other is giving the least generous. You’ll get some sort of Bayesian confidence interval of the probability of the conjecture being true. If the entire interval is low (say below 60% or something), then before spending several months trying prove it your time might be better spent gathering more evidence for or against it.

Again, mathematicians already do this at some subconscious level, so being aware of one way to analyze what it is you are actually doing could be very useful. Humans have tons of cognitive biases, so maybe you have greatly overestimated how likely something is and doing a quick Bayes’ theorem calculation can set you straight before wasting a ton of time. Or you could write all this off as nonsense. Whatever. It’s up to you.

If you’ve followed the posts up to now, you’ll probably find this calculation quite repetitive. You can probably guess what we’ll do. We want to figure out P(C|E), the probability that a conjecture is true given the evidence you’ve accumulated. What goes into Bayes’ theorem? Well, P(E|C) the probability that we would see the evidence we have supposing the conjecture is true; P(C) the prior probability that the conjecture is true; P(E|-C) the probability we would see the evidence we have supposing the conjecture is not true; and P(-C) the prior probability that the conjecture is not true.

Clearly the problem of assigning some exact probability to any of these is insanely subjective. But also, as before, it should be possible to find the most optimistic person about a conjecture to overestimate the probability and the most skeptical person to underestimate the probability. This type of interval forming should be a lot less subjective and fairly consistent. One should even have strong arguments to support the estimates which will convince someone who questions them.

Let’s use the Riemann hypothesis as an example. In our modern age, we have massive numerical evidence that the Riemann hypothesis is true. Recall that it just says that all the zeroes of the Riemann zeta function in the critical strip lie on the line with real part 1/2. Something like the first 10,000,000,000,000 zeroes have been checked by computer plus lots (billions?) have been checked in random other places after this.

Interestingly enough, if this were our “evidence” our estimation of P(E|C) may as well be 1, but P(E|-C) would have to contribute a significant non-trivial factor in the denominator of Bayes’ theorem. This is because we estimate this probability based on what we’ve seen in the past in similar situations. It turns out that in analytic number theory we have several prior instances of the phenomenon of a conjecture looking true for exceedingly large numbers before getting a counterexample. In fact, Merten’s Conjecture is explicitly connected to the Riemann hypothesis and the first counterexample could be around (no explicit counterexample is known, just that one exists, but we know by checking that it is exceedingly large).

It probably isn’t unreasonable to say that most mathematicians believe the Riemann hypothesis. Even giving generous prior probabilities, the above analysis would give a not too high level of confidence. So where does the confidence come from? Remember, that in Bayesian analysis it is often easy to accidentally not use all available evidence (subconscious bias may play a role in this process).

I could do an entire series on the analogies and relations between the Riemann hypothesis for curves over finite fields and the standard Riemann hypothesis, so I won’t explain it here. The curves over finite fields case has been proven and provides quite good evidence in terms of making P(E|-C) small.

The Bayesian calculation becomes much, much more complicated in terms of modern mathematics because of all the analogies and more concretely the ways in which the RH is interrelated with theorems about number fields and Galois representations and cohomological techniques. We have conjectures equivalent to (or implying or implied by) the RH which allows us to transfer evidence for and against these other conjectures.

In some sense, essentially all this complication will only increase the Bayesian estimate, so we could simplify our lives and make some baseline estimate taking into account the clearest of these and then just say that our confidence is at least that much. That is one explanation of why many mathematicians beleive the RH even if they’ve never explicitly thought of it that way. Well, this has gone on too long, but I hope the idea has been elucidated.

Bayes’ Theorem 3: Arguments from Absence of Evidence (Historical Edition)

If you move in the same circles that I do then you’ve probably heard the following phrase many times, “Absence of evidence is not evidence of absence.” In and of itself this is totally true. In fact, it is just a special case of a well-known logical fallacy called an argument from ignorance.

One of the really cool things about using Bayesian methods when analyzing historical events (actually you could adapt the following to the example of the scientific method as well) is that you can quantify how improbable a certain absence of evidence is to make a sound argument. This allows you to conclude that a historical event actually did not take place based on the absence of evidence.

Now I could try to do this in some extremely abstract fashion, but it is so much clear to just show you in an example. There’s some good news and some bad news. The good news is that this post is being made near Easter, so the example is timely. The bad news is that some people might find the example highly offensive, because we will show using Bayesion inference that a certain event from the Gospels was entirely made up (or at least we can say with better certainty than we could ever hope for in our wildest imagination that this is the case).

This example, including the numbers, is entirely lifted from Richard Carrier’s book Proving History. This is not intended as plagiarism, but as I am not an expert in history I feel like randomly making up probabilities about how likely certain historical events are would just not make as convincing an example. Here’s the example: In the Synoptic Gospels (Matthew, Mark, and Luke) it is said that up to the death of Jesus the entire Earth was covered in darkness for three hours.

We want to figure out the probability that this was a historical event using the fact that there are no extra-Biblical accounts of this event happening. One thing to note is that there were civilizations all across the Earth in the first century who were already keeping copious records of bizarre astronomical phenomena that have survived to this time.

One very important thing to keep in mind when doing this example is the following. One might be tempted to make an argument about history vs supernatural events and so on. But the cool thing about this is that we don’t need to make any assumption about the occurrence of supernatural events to do this analysis. In fact, we could assume that supernatural events happen all the time and we will still come to the conclusion that this story was fabricated.

Let A be the statement that the Earth was covered in darkness for three hours. Let B be the event that we have no extra-Biblical accounts of this fact (I use the term “extra-Biblical” loosely to mean no sources that don’t admit they are referencing the Bible). We want to calculate P(A|B) the probability that the Earth was actually covered in darkness for three hours given the fact that we have no evidence for it.

The quantities that come up in Bayes’ theorem are the following: P(B|A), the probability we have no evidence of the event occurring supposing that it actually did occur. If we are exceedingly generous we can assign this probability at 1%. Note how high this percentage is though. Given our knowledge of surviving records of the time it is so mind-bogglingly unlikely that every civilization on the planet just accidentally missed something that would have scared them all out of their minds.

We also have P(A). This is slightly subtle, because in this case it represents not merely the probability that the event occurred, but is really the probability that the author of Mark (or one of the other Gospels which were probably just copying this detail) is telling the truth about the event occurring. More precisely, considering all the times that we know of (our “prior knowledge” as it was called in the previous post) of people telling us that the sun was blotted out how frequently did it actually happen (or less confusingly, when doing P(-A) how frequently did it turn out the story was made up). Being exceedingly generous again we’ll call this 1%.

Note we are not dismissing this on grounds of being a supernatural event (we’ve assumed for the purposes of this calculation that they happen all the time). The low number of 1% comes from the fact that we know of tons of examples in history where people tell stories like this one, but where we later find out they were made up. Lastly, we need P(B|-A) which is the probability of finding no external evidence for the event assuming the event was made up. This is so close to 100% that we may as well assign it a probability of 1.

Plugging everything in tells us that with at least (remember we were quite generous with the numbers) 99.99% certainty (re: there is a 99.99% chance that) the event never happened in history and was just made up by the authors of the Synoptic Gospels. And that is how Bayesian inference can lead to a sound argument from absence of evidence.

Of course, this should be an entirely non-controversial example because outside of a tiny few fundamentalist “scholars” who are clearly pushing an agenda, the fact that this even never happened in history has essentially unanimous consensus among all historians and Biblical scholars. So our result shouldn’t actually be surprising.

Bayes’ Theorem 2: Bayesian Epistemology

Last time I ended by saying we’d look at an example from the philosophy of math. We’ll get to that later, but I realized that even though we did an example of applying Bayes’ theorem I gave no feel for what it might mean to “be a Bayesian.”

The word Bayesian has been stuck in front of basically any branch of study you can think of (just look at the wikipedia disambiguation page on Bayesian). The term basically does the same modification to any field of study and it just means that you recast your arguments in a way that allows you to use Bayes’ theorem to make inferences.

Today’s post will attempt to show what this means by recasting the scientific method in terms of Bayesian inference. I’ve been told that the philosopher of science Ian Hacking was the first to do this, but I don’t have a reference and haven’t read his stuff to know if this post will match how he uses the term.

Let’s just recap what the scientific method is briefly. Well, this will depend on who you ask, but for our purposes let’s just say it is the following. You form a hypothesis. This hypothesis allows you to make predictions. You design a carefully controlled experiment to see whether or not those predictions are valid. The experiment gives you evidence for or against your hypothesis. Based on this evidence you decide to accept or reject the hypothesis.

If we want to apply Bayesian inference to the scientific method, then we should re-interpret the example from last time in terms of “prior knowledge” and “evidence.” Recall that we had a test for a disease that was 99% accurate, but we also knew that only 1% of the population had the disease. You got tested and came up positive for the disease, and then Bayesian probability told us that there was only a 50% chance that you actually had the disease.

Again, since I only mean to suggest a rough idea by what is meant by this term “Bayesian” I’ll ignore some of the subtleties with whether proper scientific method requires a null hypothesis and whether you test for or against the predictions, etc and just focus on framing this as easily as possible.

In the example last time we’ll say that our hypothesis is that you have the disease and our experiment is to do this test. We are going into the experiment with some prior scientific knowledge. Namely, how often our experiment gives us the wrong answer and how many people have the disease. In other words, before running any test we have 99% confidence that our hypothesis is wrong.

Now we run the experiment to gather evidence. The evidence is a single instance of the test telling us that you have the disease. Bayes’ theorem tells us that we can only be 50% confident that the hypothesis is correct. Using Bayesian methods, I would hope that any scientist would say that the experiment was inconclusive.

Let’s consider a modified experiment. It consists of doing the disease test twice. After that first positive, all of a sudden all you get a negative. Bayes’ theorem gives us 99% confidence that you don’t have the disease and we could with scientific certainty (above 95% is the typical scientific cut-off) reject the hypothesis. This happens because the chance of you having the disease is so low and the chance of that negative result being wrong is so low it totally outweighs that positive result. Bayes’ theorem tells us that there is a 99% chance that the positive was a false positive (N.B. this is because of using two pieces of evidence from our experiment and not because false positives only happen 1% of the time).

Let’s consider getting a different result from our experiment. If we got both tests to come up positive, then Bayes’ theorem tells us that the probability of actually having the disease is 99%. So we can say that the hypothesis is true. There is just no way that the test came up with 2 false positives when there is such a small chance of a false positive.

Here’s the moral of all of this. Bayesian inference gives a mathematically precise way to make sense of the following phrase which is central to the scientific method: The more extraordinary the hypothesis (re: hypotheses that are counter to prior scientific knowledge) the more extraordinary the evidence must be.

Do you see how Bayesian inference does this? We started with the hugely extraordinary hypothesis that you had the disease despite the fact that we could go into the experiment with 99% certainty that this was incorrect. So we needed extremely good evidence in order to affirm the hypothesis. In the first experiment our evidence was testing positive for the disease. This might seem like good evidence considering the 99% confidence we can have in such a test, but our evidence had to overcome the huge obstacle of an extraordinary hypothesis.

Bayes’ theorem then told us that that evidence just wasn’t good enough for that hypothesis. So the Bayesian interpretation of the scientific method says we should look at how confident we are in the various pieces of prior scientific knowledge that confirm and reject our hypothesis as is. Then we do an experiment and get some evidence. We plug all that into Bayes’ theorem and see whether or not that evidence was good enough to have a high level of confidence in either accepting or rejecting the hypothesis.

Bayes’ Theorem 1: The Idea

I was going to do one more music theory post, but it seemed way more effort than it was worth. I’ll definitely come back to this topic in the future. I really want to look at the crazy huge book The Topos of Music and try to distill out what the main idea is. So someday you can look forward to that.

We’ll move on to a topic that use to fascinate me a lot, and then I sort of forgot about it. I started reading Richard Carrier’s book on using Bayes’ theorem in the historical method, so it has come up again. I just started the book, so I might not talk about this in particular, but over the years I’ve come across some very fascinating applications of Bayes’ theorem to surprising situations.

What does it say? Well, the simplest form of it is just a formula for calculating a probability when you have some information (technically I’m referring to a conditional probability). Suppose A and B are two events such as “it is raining in Seattle” and “I am carrying an umbrella.” The negation, , would then be “it is not raining in Seattle.”

We will use the notation to denote the probability that happens. We will use the notation to mean “the probability that happens given that has happened.” Now in a simple two event situation like this Bayes’ theorem says we can calculate the probability as follows:

There are tons of equivalent ways to express this, but this is the one we’ll find most useful for now. Before reading the example below it is important to remember that this really is a “theorem” with rigorous proof. We can have all sorts of philosophical debates about what it means to actually know the probability of an event happening with varying levels of certainty, but what can not be debated is that if you accept that we know somehow the probabilities , , and (of course the last one is redundant) then we can know using the formula with the same level of certainty.

I first saw this as an undergrad in a introduction to statistics and probability class. I then went on to tutor nursing majors in a similar class for several years, so maybe my proto-typical application of this is skewed by my experience. Still, it gives a really good idea of why this theorem is useful in tons and tons of everyday situations.

Let’s say a new disease has just been discovered: Hilbert’s disease (I’m pretty sure this isn’t real). Doctors develop a highly accurate way to test for the disease. It turns out (through testing a huge sample of the population) that 99% of the time you test positive for the disease you actually have it (in the language of conditional probability we could say “the probability that you test positive given that you actually have the disease is 99%) and 99% of the time that you test negative for the disease you don’t actually have it. Alternatively, false positives and false negatives only occur one percent of the time.

Now this is a newly discovered disease, so it turns out that very few people have it. Specifically only 1% of the population has it. There is also no known cause or early symptoms (I throw this in so that when I say “you” in the next sentence you are truly a random choice from the population). You decide to go get tested. Oops. You test positive. What is the Bayesian probability that you actually have the disease?

If you haven’t seen this before, then you might be tempted to say that since the test has 99% accuracy, then it must be the case that there is a 99% chance you have the disease. But this is your human intuition at work, and if there is one thing we know about the human brain it is notoriously bad at intuiting probabilities (just think of the infamous Monty Hall controversy).

Well, we can just plug all the numbers into Bayes’ theorem. If A is the event of testing positive for the disease and B is the event of actually having the disease, then we want to calculate P(B|A) the probability that you have the disease given the information of testing positive.

Bayes theorem says

What?! This says there is only a 50% chance that you have the disease even though the test is 99% accurate and you tested positive for it. If you find this surprising it is because you are ignoring a huge piece of information. Bayes’ theorem is accounting for the fact that we know that only one percent of the population actually has the disease. If you really are a random member of the population, then there is a huge chance you don’t have the disease. So if you test positive it is very likely that you fall into the one percent of cases that give a false positive.

This is pretty cool right? It gives you a radically different perspective on these numbers when you see these statistics like pregnancy tests are whatever percent accurate or drug tests are whatever percent accurate and so on. Anyway, that’s the gist of Bayes’ theorem. Next time we’ll see how Bayesian ideas can actually be applied to philosophy of mathematics and proof theory.