(Pesky?) Priors

When I tell people I am learning Bayesian statistics, I tend to get one of two responses: either people look at me blankly—“What’s Bayesian statistics?”—or I get scorned for using such “loose” methods—“Bayesian analysis is too subjective!”¹. This latter “concern” arises due to (what I believe to be a misunderstanding of) the prior: Bayesian analysis requires one state what one’s prior belief is about a certain effect, and then combine this with the data observed (i.e., the likelihood) to update one’s belief (the posterior).

On the face of it, it might seem odd for a scientific method to include “subjectivity” in its analysis. I certainly had this doubt when I first started learning it. (And, in order to be honest with myself, I still struggle with it sometimes.) But, the more I read, the more I think this concern is not warranted, as the prior is not really “subjectivity” in the strictest sense of the word at all: it is based on our current understanding of the effect we are interested in, which in turn is (often) based on data we have seen before. Yes, sometimes the prior can be a guess if we² have no other information to go on, but we would express the uncertainty of a belief in the prior itself.

The more I understand Bayesian statistics, the more I appreciate the prior is essential. One under-stated side-effect of having priors is that it can protect you from dubious findings. For example, I have a very strong prior against UFO predictions; therefore, you are going to have to present me with a lot more evidence than some shaky video footage to convince me otherwise. You would not have to provide me with much evidence, however, if you claimed to have roast beef last night. Extraordinary claims require extraordinary evidence.

But, during my more sceptical hours, I often succumbed to the the-prior-is-nothing-but-subjectivity-poisoning-your-analysis story. However, I now believe that even if one is sceptical of the use of a prior, there are a few things to note:

• If you are concerned your prior is wrong and is influencing your inferences, just collect more data: A poorly-specified prior will be washed away with sufficient data.

• The prior isn’t (really) subjective because it would have to be justified to a sceptical audience. This requires (I suggest) plotting what the prior looks like so readers can familiarise themselves with your prior. Is it really subjective if I show you what my prior looks like and I can justify it?

• Related to the above, the effect of the prior can be investigated using robustness checks, where one plots the posterior distribution based on a range of (plausible) prior values. If your conclusions don’t depend upon the exact prior used, what’s the problem?

• Priors are not fixed. Once you have collected some data and have a posterior belief, if you wish to examine the effect further you can (and should) use the posterior from the previous study as your prior for the next study.

These are the points I mention to anti-Bayesians I encounter. In this blog I just wanted to skip over some of these with examples. This is selfish; it’s not really for your education (there really are better educators out there: My recommendation is Alex Etz’s excellent “Understanding Bayes” series, from where this blog post takes much inspiration!). I just want somewhere with all of this written down so next time someone criticises my interest in Bayesian analysis I can just reply: “Read my blog!”. (Please do inform me of any errors/misconceptions by leaving a comment!)

As some readers might not be massively familiar with these issues, I try to highlight some of the characteristics of the prior below. In all of these examples, I will use the standard Bayesian “introductory tool” of assessing the degree of bias in a coin by observing a series of flips.

A Fair Coin

If a coin is unbiased, it should produce roughly equal heads and tails. However, often we don’t know whether a coin is biased or not. We wish to estimate the bias in the coin (denoted theta) by collecting some data (i.e., by flipping the coin); a fair coin has a theta = 0.5. Based on this data, we can calculate the likelihood of various theta values. Below is the likelihood function for a fair coin.

In this example, we flipped the coin 100 times, and observed 50 heads and 50 tails. Note how the peak of the likelihood is centered on theta = 0.5. A biased coin would have a true theta not equal to 0.5; theta closer to zero would reflect a bias towards tails, and a theta closer to 1 would reflect a bias towards heads. The animation below demonstrates how the likelihood changes as the number of observed heads (out of 100 flips) increases:

So, the likelihood contains the information provided by our sample about the true value for theta.

The Prior

Before collecting data, Bayesian analysts would specify what their prior belief was about theta. Below I present various priors a Bayesian may have using the beta distribution (which has two parameters: a and b):

The upper left plot reflects a prior belief that the coin is fair (i.e., the peak of the distribution is centered over theta = 0.5); however, there is some uncertainty in this prior as the distribution has some spread. The upper right plot reflects total uncertainty in a prior belief: that is, the prior holds that any value of theta is likely. The lower two plots reflect prior beliefs that the coin is biased. Maybe the researcher had obtained the coin from a known con-artist. The lower left plot reflects a prior for a biased coin, but uncertainty about which side the coin is biased towards (that is, it could be biased heads or tails); the lower right plot reflects a prior that the coin is biased towards heads.

The effect of the prior

I stated above that one of the benefits of the prior is that it allows protection (somewhat) from spurious findings. If I have a really strong prior belief that the coin is fair, 9/10 heads isn’t going to be all that convincing evidence that it is not fair. However, if I have a weak prior that the coin is fair, then I will be quite convinced by the data.

This is illustrated below. Both priors below reflect the belief that the coin is fair; what differs between the two is the strength in this belief. The prior on the left is quite a weak belief, as the distribution (although peaked at 0.5) is quite spread out. The prior on the right is a stronger belief that the coin is fair.

In both cases, the likelihood is the result of observing 9/10 heads.

You can see that when the prior is a weak belief, the posterior is very similar to the likelihood; that is, the posterior belief is almost entirely dictated by the data. However, when we have a strong prior belief, our beliefs are not altered much by observing just 9/10 heads.

Now, I imagine that this is the anti-Bayesian’s point: “Even with clear data you haven’t changed your mind.” True. Is this a negative? Well, imagine instead this study was assessing the existence of UFOs rather than simple coin flips. If I showed you 9 YouTube videos of UFO “evidence”, and 1 video showing little (if any) evidence, would you be convinced of UFOs? I doubt it. You were the right-hand plot in this case. (I know, I know, the theta distribution doesn’t make sense in this case, but ignore that!)

What if the prior is wrong?

Worried that your prior is wrong³, or that you cannot justify it completely? Throw more data at it. (When is this ever a bad idea?) Below are the same priors, but now we flip the coin 1,000 times and observe 900 heads. (Note, the proportion heads is the same in the previous example.) Now, even our strong prior belief has to be updated considerably based on this data. With more data, even mis-specified priors do not affect inference.

To get an idea of how sample size influences the effect of the prior on the posterior, I created the below gif animation. In it, we have a relatively strong (although not insanely so) prior belief that the coin is biased “heads”. Then, we start flipping the coin, and update the posterior after each flip. In fact, this coin is fair, so our prior is not in accord with (unobservable) “reality”. As flips increases, though, our posterior starts to match the likelihood in the data. So, “wrong” priors aren’t really a problem. Just throw more data at it.

“Today’s posterior is tomorrow’s prior” — Lindley (1970)

After collecting some data and updating your prior, you now have a posterior belief of something. If you wish to collect more data, you do not use your original prior (because it no longer reflects your belief), but you instead use the posterior from your previous study as the prior for your current one. Then, you collect some data, update your priors into your posteriors…and so on.

In this sense, Bayesian analysis is ultimately “self-correcting”: as you collect more and more data, even horrendously-specified priors won’t matter.

In the example below, we have a fairly-loose idea that the coin is fair—i.e., theta = 0.5. We flip a coin 20 times, and observe 18 heads. Then we update to our posterior, which suggests the true value for theta is about 0.7 ish. But then we wish to run a second “study”; we use the posterior from study 1 as our prior for study 2. We again observe 18 heads out of 20 flips, and update accordingly.

Conclusion

One of the nicest things about Bayesian analysis is that the way our beliefs should be updated in the face of incoming data is clearly (and logically) specified. Many peoples’ concerns surround the prior. I hope I have shed some light on why I do not consider this to be a problem. Even if the prior isn’t something that should be “overcome” with lots of data, it is reassuring to know for the anti–Bayesian that with sufficient data, it doesn’t really matter much.

So, stop whining about Bayesian analysis, and go collect more data. Always, more data.

Click here for the R code for this post

---

1. Occasionally (althought his is happening more and more) I get a slow, wise, agreeing nod. I like those. ↩
2. I really wanted to avoid the term “we””, as it implies I am part of the “in-group”: those experts of Bayesian analysis who truly appreciate all of its beauty, and are able to apply it to all of their experimental data. I most certainly do not fall into this camp; but I am trying. ↩
3. Technically, it cannot be “wrong” because it is your belief. If that belief is justifiable, then it’s all-game. You may have to update your prior though if considerable data contradict it. But, bear with me. ↩

Advertisements

Surviving a conference!

I am currently in London at my first conference of the year, speaking at the Experimental Psychology Society (EPS) conference at University College London. The EPS meets three times per year, but the London meeting is often the best-attended. (This might be in no small part due to people in early January being fed up of being at home stuffing their faces with turkey & trimmings.) Conferences are an essential part of an academic’s life, but I have not always found conferences so easy. In fact, I still find them difficult on many levels.

This post is not really for anyone other than myself; it’s a post to my old-self, as if I have travelled back in time and have passed-on wisdom I have gleaned from my five years of conferencing. You see, I am sure I could have got much more out of these five years of being surrounded by fellow researchers that I feel I have missed out. Were I to go back in time and try all over again, I would probably try to follow this advice.

For those new to conferencing, I hope you might find something of interest here, too.

Speak to People

I start off being guilty of hypocrisy here. “Do as I say, not as I do!” You see, I find it very difficult to talk to people at conferences. I am told I am not as socially-awkward as I fear I am in my own head, but I just am never comfortable talking to new people. But, this is (or so I am told) one of the main benefits of going to conferences. You are surrounded by fellow researchers, some of whom may be working on similar questions, but all of whom are interested in research. Your next successful research collaboration could be born over coffee.

The difficulty is (for me) that speaking to people is very difficult. Indeed, this was one of the main attractions of academia: being locked away in your office in isolation thinking about what interests you, and only communicating via papers.

How wrong I was.

So, when speaking to my old-self, I would say “Speak to people!”. This doesn’t make it easier, so maybe we need a trick. If you attended a talk you found interesting, try to formulate an interesting question. Then, instead of approaching the speaker at coffee time with mundane small-talk, tell them you found their talk interesting and ask them the question you had.

Attend the Talks

I think a lot of people do this anyway, but I often when I speak to people about their last conference visit overseas I tend to hear more of the local sight-seeing opportunities rather than the science presented. I am in danger of sounding like an incredible bore now, but attend the talks! By all means see the sights, but you are here for the science, so do science.

Ask a Question

Linked to how to make sensible conversations with conference delegates, I always like to ask a question during talks I attend. Now, there is a fine line here, because there is nothing worse than the person who always asks irritating questions just to make themselves look clever. I have been guilty of this in the past, so I am not one to cast stones, but it is not the reason to ask a question.

Listening to a talk with the intention of asking a question forces you to pay more attention to the talk. It forces you to think critically about the science being presented. How easy is it to switch off during the talks and think about your talk (or where you will go for dinner)? If you know you are going to ask a question, you will get more out of the talk. (As a side note, pay attention to the session chair during the talk; they will likely be making notes of questions because it is their job to ask a question if no one in the audience does, to avoid embarrassing silences.)

I tend to start off my question by introducing myself. This helps the speaker track you down later if they wanted to follow up your question, but I tend to do it just out of politeness: I like to know who I am speaking to, so I make sure people know who they are speaking to. I tend to say “Hi, I am Jim Grange from Keele University. Thank you for an interesting talk. I was wondering…[insert question]”.

Sometimes your formulated question should only be an exercise, and shouldn’t be asked. (This reminds me of a delightful quip by Christopher Hitchens: “It’s true that everyone has a book in them. In most cases, that is where it should stay.”) There are times when asking your question is not recommended. Want a nice guide as to whether you should ask the question you have formulated? See this (note that this is not my graph; I don’t know the original source, so if you do know it please let me know so I can reference accordingly!):

Think Actively During Talks

I guess this is related to the advice above, but there is one thing I always do during talks. For every experimental talk I attend, I like to think of one experiment I would like to do to extend the work presented. I think of how I would design it, what I would expect to find, etc. This is just good practice for thinking about how to design experiments.

I find this very useful because I get so used to thinking about experimental design in task switching contexts because this is the research I do. There is the danger, then, that when I have a real reseach question outside of task switching, my design ends up looking rather like a task switching experiment. (“If all you have is a hammer, everything you look at will look like a nail.”)

Want some bonus points? Collect these hypothetical experiments and actually run one! Side-projects are fun.

Look for New Research Programmes

The past two years I have been attending talks hoping to listen to a talk that introduces me to a new area of research that excites me that I can then go back to my lab and start work in. You see, I am getting a bit bored with my research area. I have done task switching research almost continuously since my undergraduate thesis (about 8 years, now!). I am looking for something new to excite me for the next 10 years as much as task switching has. I always live in hope that my eyes will be opened during a talk about a new research programme that I can get my teeth stuck in to. This is one major motivator for attending conferences at the moment.

Attend the Poster Sessions

The poster sessions at most conferences I attend are populated by PhD students presenting their work, so this is an excellent opportunity to speak to “up-and-coming” scientists. Be kind to them; for many, this will be their first step into academic presentations, and will probably be nervous. Compliment them on their work (but don’t bullshit).

Give a Talk!

This may seem obvious, but don’t be a fly on the wall at conferences. Get stuck in. Giving a talk is the best way. People will be exposed to your research, you will (likely) get critical feedback on your ideas (which is great), and people get to know you (linked with some of the topics above). Talks also allow you to present “work-in-progress”, which will allow you to test your ideas before your project has fully developed. This is important.

Publish your Slides

Many people are now publishing their research papers online so anyone can access them. Why aren’t people (generally) doing the same with their presentation slides? Likely the answer is related to the fact that conferences tend to present unpublished research, so people don’t wish to be scooped. I can sympathise with this, but I think it is short-sighted. You have already “released” your ideas when you gave the talk. So, publish your slides, too.

From 2016, I will be publishing all of my slides online. For those interested, here are my slides for the talk I am giving this Friday:

Conclusion

In sum, enjoy the conference. It’s a time to listen to great ideas and share yours. After all, isn’t this what science is all about? Just don’t be as shy as me. Try and speak to people; they (probably) won’t bite!