Confidence intervals

You can’t put an impact factor on that!

I’m very proud of my most recent paper (Schuch & Grange, in press), soon to be published in the Journal of Experimental Psychology: Learning, Memory, & Cognition. The pride  comes not just from being accepted in a prestigious cognitive journal—but wow, I’m certainly thrilled by it—but from aspects of the research that have no metrics. Forget your impact factors, they don’t give me the same buzz as what I experienced during this project: 

  1. It reports an effect I discovered rather serendipitously one summer whilst going through some old (published) data sets. The lay-person’s perception is that science moves forward from “Eureka” moments, whereas the truth is it occurs more along the “Hmm, that’s odd…” paths. It’s actually quite a difficult realisation sometimes because you don’t feel in control (and I’m quite the control freak!), but it’s very exciting indeed; you never know what finding is just around the corner! 
  2. It involved my first collaboration with a member of a lab in Germany I have had great respect for since starting my work in this area (the great respect is for the individual AND the lab). I remember reading a paper by Stefanie Schuch (my co-author) when I was doing my undergraduate thesis (Schuch & Koch, 2003; JEP:HPP), and that I now have a paper with her blows my mind every time I think about it. If I went back in time and told my younger self that I would soon have a paper with these authors, I really wouldn’t have believed myself. Yet here it is! It’s funny how things turn out. It’s like the academic equivalent of a guitar enthusiast getting the chance to jam with Slash.
  3. It involved exploratory analysis and confirmatory analysis, and as such is probably my most “robust” paper to date. E-J Wagenmakers has a nice paper extolling the virtues of confirmatory studies, and I am pleased to have followed much of this advice (although, the confirmatory studies were not pre-registered). As I found the effect during exploratory analysis, it would have been hasty to publish this data without seeing if it replicates in a confirmatory study, yet this is something I would have tried to do even as recently as last year. Instead, I decided to contact the German lab to see whether they find the effect in their data, too (they did). Then, we decided to run a confirmatory study. It was very nice (and re-assuring) to see it replicated in a different lab. Plus, I felt a little more “grown up” in scientific terms doing something the right (and rigorous) way. 
  4. I got to use “mini meta-analyses” for the first time. I love data analysis. Sometimes I joke with my colleagues that I would be just as happy in my job if there were no psychology involved, just so long as I was doing some form of data analysis. I just love it. Seriously, I could be measuring & analysing rainfall and I would be just as happy. I just love the process of testing ideas and getting data. So, to be able to try out a new analysis method for the first time was great fun. Meta-analyses are typically used in large-scale reviews where the meta-analysis might contain tens (or more) of data points. However, in a recent book, Geoff Cumming is very persuasive in asking the reader to think of every data point they are publishing as a point in a future meta analysis. Such “meta-analytical” thinking is important, as one experimental result in isolation means next to nothing. So, focus on the bigger picture. In the book, he recommends doing “mini meta-analyses”, where you perform a meta-analysis across only a few studies or experiments. Most papers reporting a new effect in cognitive psychology tend to have 3+ experiments reporting the effect and testing for boundary conditions etc. Cumming suggests you should do a mini meta-analysis across these studies to get an estimate of the true effect size of the new effect. This is what I did in my study. My initial exploratory data analysis was using data from a 2010 paper of mine that had 3 experiments, and a total of 5 conditions that showed the “standard” effect. So, in the current paper, I conducted a mini meta-analysis across these 5 conditions showing the magnitude (and consistency) of the “new” effect. The Figure reporting this meta analysis is below. It was really neat trying out new techniques, and I think it really added weight to the paper. I shall certainly be using this again in the future!

To sum up this short post, this project was very exciting and satisfying from a scientific perspective. I would have almost retained the same joy from this project had it never seen the light of day.

You can’t put an impact factor on that! 


A paper without p-values

All I seem to be writing about recently is rejection. So, today I thought I would write about a paper I have recently had accepted (they DO happen!). The paper is in press at the Quarterly Journal of Experimental Psychology, and I am very proud of it. (You can read it here; yay for Open Science.)

My pride is a consequence of several factors: it is a product of two years’ (hard) work and thought; it contains my first published attempts at computational modelling (see Appendices); it touches upon a very topical question (do memories for tasks/goals decay, or are they lost due to interference); Ellen Cross was an incredibly intelligent Year 2 (at the time) student at Keele Psychology, so it was a pleasure to help her career along (she is now doing a PhD in Neuroscience); and it doesn’t contain a single p-value.

Yes, that’s right. There isn’t a single p-value in the paper. No null-hypothesis significance test.  I must admit, I submitted the paper with some trepidation, and was  expecting the reviewers and editor to ask for p-values to be included (or some other form of NHST), but to my delight there was no mention of it! 

I am quite convinced by Geoff Cummings’ assertion that researchers should focus on estimation of effect sizes and meta-analytic thinking rather than NHST. This conviction began by reading his book (see here), and was cemented by his paper in Psychological Science 

So, instead of using NHST and p-values, I used confidence intervals. The paper is littered with plots like the one below; the panel on the left (A) shows the main data (mean response times [RTs] to task-switches and task-repetitions as a function of the interval between tasks, known as the response–cue interval [RCI]) together with their 95% confidence intervals. In this literature, one key effect is whether the difference between switch RT and repetition RT (the “switch cost”) decreases as RCI increases.




To keep with the “estimation” philosophy of Cumming, I was more interested in the magnitude of the switch cost as RCI increased. So, in the right panel (B), I calculated the switch cost (switch RT minus repetition RT) for each RCI, and plotted them with their 95% confidence intervals. There are two types of confidence intervals; the thin one is the standard 95% CI, and the dark one is the Loftus & Masson CI for within-subject designs (see paper here).  (If you want to know how to do two error bars in Excel, I produced a short video here.)

I believe NHST is not essential for my purpose here (or throughout the paper). The panel on the left shows the general pattern of data, together with an estimate of the precision of these estimates. The panel on the right shows the magnitude (and precision of the estimate) of the switch cost. It is clear to see that the switch cost is reducing as RCI increases. The advantage of the CI approach—in my opinion—is that it forces us to focus on the magnitude & precision of effects, rather than their mere presence/absence. 

I’m not sure that I will never use NHST again. This paper was unique in that I was practically the sole-author on the paper, and thus can take full responsibility for it. EJ-Wagenmakers and colleagues also make a compelling argument for why hypothesis testing is essential for psychological science (see paper here). 

Either way, this paper was fun to write. It was incredibly refreshing to not discuss p-values and “significance”. It is odd, but it felt more scientific. Why not give it a try in your next paper?