Last time I ended by saying we’d look at an example from the philosophy of math. We’ll get to that later, but I realized that even though we did an example of applying Bayes’ theorem I gave no feel for what it might mean to “be a Bayesian.”
The word Bayesian has been stuck in front of basically any branch of study you can think of (just look at the wikipedia disambiguation page on Bayesian). The term basically does the same modification to any field of study and it just means that you recast your arguments in a way that allows you to use Bayes’ theorem to make inferences.
Today’s post will attempt to show what this means by recasting the scientific method in terms of Bayesian inference. I’ve been told that the philosopher of science Ian Hacking was the first to do this, but I don’t have a reference and haven’t read his stuff to know if this post will match how he uses the term.
Let’s just recap what the scientific method is briefly. Well, this will depend on who you ask, but for our purposes let’s just say it is the following. You form a hypothesis. This hypothesis allows you to make predictions. You design a carefully controlled experiment to see whether or not those predictions are valid. The experiment gives you evidence for or against your hypothesis. Based on this evidence you decide to accept or reject the hypothesis.
If we want to apply Bayesian inference to the scientific method, then we should re-interpret the example from last time in terms of “prior knowledge” and “evidence.” Recall that we had a test for a disease that was 99% accurate, but we also knew that only 1% of the population had the disease. You got tested and came up positive for the disease, and then Bayesian probability told us that there was only a 50% chance that you actually had the disease.
Again, since I only mean to suggest a rough idea by what is meant by this term “Bayesian” I’ll ignore some of the subtleties with whether proper scientific method requires a null hypothesis and whether you test for or against the predictions, etc and just focus on framing this as easily as possible.
In the example last time we’ll say that our hypothesis is that you have the disease and our experiment is to do this test. We are going into the experiment with some prior scientific knowledge. Namely, how often our experiment gives us the wrong answer and how many people have the disease. In other words, before running any test we have 99% confidence that our hypothesis is wrong.
Now we run the experiment to gather evidence. The evidence is a single instance of the test telling us that you have the disease. Bayes’ theorem tells us that we can only be 50% confident that the hypothesis is correct. Using Bayesian methods, I would hope that any scientist would say that the experiment was inconclusive.
Let’s consider a modified experiment. It consists of doing the disease test twice. After that first positive, all of a sudden all you get a negative. Bayes’ theorem gives us 99% confidence that you don’t have the disease and we could with scientific certainty (above 95% is the typical scientific cut-off) reject the hypothesis. This happens because the chance of you having the disease is so low and the chance of that negative result being wrong is so low it totally outweighs that positive result. Bayes’ theorem tells us that there is a 99% chance that the positive was a false positive (N.B. this is because of using two pieces of evidence from our experiment and not because false positives only happen 1% of the time).
Let’s consider getting a different result from our experiment. If we got both tests to come up positive, then Bayes’ theorem tells us that the probability of actually having the disease is 99%. So we can say that the hypothesis is true. There is just no way that the test came up with 2 false positives when there is such a small chance of a false positive.
Here’s the moral of all of this. Bayesian inference gives a mathematically precise way to make sense of the following phrase which is central to the scientific method: The more extraordinary the hypothesis (re: hypotheses that are counter to prior scientific knowledge) the more extraordinary the evidence must be.
Do you see how Bayesian inference does this? We started with the hugely extraordinary hypothesis that you had the disease despite the fact that we could go into the experiment with 99% certainty that this was incorrect. So we needed extremely good evidence in order to affirm the hypothesis. In the first experiment our evidence was testing positive for the disease. This might seem like good evidence considering the 99% confidence we can have in such a test, but our evidence had to overcome the huge obstacle of an extraordinary hypothesis.
Bayes’ theorem then told us that that evidence just wasn’t good enough for that hypothesis. So the Bayesian interpretation of the scientific method says we should look at how confident we are in the various pieces of prior scientific knowledge that confirm and reject our hypothesis as is. Then we do an experiment and get some evidence. We plug all that into Bayes’ theorem and see whether or not that evidence was good enough to have a high level of confidence in either accepting or rejecting the hypothesis.