General

Reproducibility Article in “The Conversation”

I was asked to write 200-300 words on my views on whether there is a reproducibility crisis in the sciences for an article that was appearing in The Conversation. I was so passionate about what I was writing that I ended up writing over 1,200 words. The final article was, of course, edited down by their team to meet the 300 word guide. Below I have posted my full piece.

That there is a reproducibility crisis in psychological science—and arguably across all sciences—is, to me, beyond doubt. Murmurings of low reproducibility began in 2011— the so-called “year of horrors” for psychological science (Wagenmakers, 2012), with the infamous fraud case of Diedrik Stapel being its low-light. But murmurings now have empirical evidence. In 2015, the Open Science Collaboration published the findings of our large-scale effort to closely-replicate 100 studies in psychology (Open Science Collaboration, 2015). And the news was not good: Only 36% of studies were replicated.

Whilst low reproducibility is not unique to psychological science—indeed, cancer biology is currently reviewing its own reproducibility rate, and things are not looking great (see Baker & Dolgin, 2017)—psychology is leading the way in getting its house in order. Several pioneering initiatives have been introduced which, if embraced by the community, will leave psychological science in a strong position moving forward. Here I focus on three I believe are the most important.

Study Pre-Registration & Registered Reports

In a delightfully concerning study, Simonsohn et al. (2013) demonstrated that, in the absence of any true effect, researchers can find statistically significant effects in their studies by engaging in questionable research practices (QRPs), such as selectively reporting outcome measures that produced significant effects and dropping experimental conditions that produced no effect. Another QRP could include analysing your data in a variety of ways (for example, maybe a couple of participants didn’t show the effect you were looking for, so why not remove them from the analysis and see whether that “clears things up”?). What was concerning about this study is that many of these QRPs were not really considered “questionable” at the time. Indeed, many researchers have admitted to engaging in such QRPs (John et al., 2013).

As such, I do not believe that the presence of QRPs reflect explicit attempts at fraud. Rather, they likely stem from a blurred distinction between exploratory and confirmatory research. In exploratory research, many measures might be taken, many experimental conditions administered, and the data scrutinised using a variety of approaches looking for interesting patterns. Confirmatory research tests explicit hypotheses using pre-planned methods and analytical strategies. Both approaches are valid—exploratory research can generate interesting questions, and confirmatory research can address these questions—but what is not valid is to report an exploratory study as though it were confirmatory (Wagenmakers et al., 2012); that is, to find an effect in exploratory research and to publish the finding together with a narrative that this effect was expected all along.

Many researchers have started to pre-register their studies detailing their predictions, experimental protocols, and planned analytical strategy before data collection begins. When the study is submitted for publication, researchers can demonstrate that no QRPs have occurred because they can point to a time-stamped document verifying their plans before data collection commenced, leading to an increase in confidence in the claims reported. This is confirmatory research at its finest.

Some journals have taken this one stage further, by introducing Registered Reports, where papers containing details of a study’s rationale and detailed proposed methods are reviewed and accepted (or rejected!) for publication before the experiment has been conducted. The neuroscience journal Cortex—with their Registered Reports Editor Professor Chris Chambers of Cardiff University—has led the way with this format. Many other journals have now started to offer such reports.

This is an important contribution to the academic publishing structure because it incentivises best research practice. Here research is judged on the soundness of the methods and the importance of the question being addressed, and not the particular results of the study. Current incentive structures in our universities—together with general pressure for increased publications (the so-called “publish or perish” attitude)—leads researchers to prioritise “getting it published” over “getting it right” (Nosek et al., 2012), potentially leading to implicit or explicit use of QRPs to ensure a publishable finding. With the advent of Registered Reports, researchers can finally do both: prioritise “getting it right” by submitting a strong and well-evidenced research proposal, and it will be published regardless of what the data say.

Open Data, Open Materials

Science works by independent verification, not by appeal to authority. As noted by Wicherts and colleagues (2011), independent verification of data analysis is important because “…analyses of research data are quite error prone, accounts of statistical results may be inaccurate, and decisions that researchers make during the analytical phase of a study may lean towards the goal of achieving a preferred (significant) result” (p. 1). Given this importance, most journal policies ask for researchers to make available their data. Yet, when asked for their data, Wicherts and colleagues (2006) found just 73% of researchers provided their data when asked. Some researchers have begun to refuse to review journal article submissions unless the authors provide their data (or provide a convincing reason for why this is not possible) as part of the Peer-Reviewers’ Openness Initiative (see Morey et al., 2015); after all, if a reviewer cannot access the data a paper is based upon, how can a full review be completed?

The flagship psychology journal Psychological Science since 2014 has incentivised researchers to share their experimental material and data by providing badges to studies that comply to open practices by publishing data and materials together with their papers in the journal. (The journal offers a third badge if the study is pre-registered.) This intervention has been remarkably effective: Kidwell et al. (2016) reported that 23% of studies in Psychological Science provided open data, a rise from lower than 3% before the badges were in use. More journals are now encouraging authors to make their data open as a consequence.

Registered Replication Reports

I tell my students all the time that “replication is the most important statistic” (source of quote unknown). To me, an empirical finding in isolation doesn’t mean all that much until it has been replicated. In my own lab, I make an effort to replicate an effect before trying to publish it. As my scientific hero Richard Feynman is famous for saying “Science is a way of trying not to fool yourself… …and you are the easiest person to fool”. As scientists, we have a professional responsibility to ensure the findings we are reporting are robust and reproducible.

But we must also not allow others’ findings to fool us, either. That is why replication of other people’s findings should become a core component of any working lab (a course of action we have facilitated by publishing a “Replication Recipe”: a guide to performing convincing replications; Brandt et al., 2014).

You’d be forgiven for thinking that reports of replications must be common place in the academic literature. This is not the case. Many journals seek novel theories and/or findings, and view replications as treading over old ground. As such, there is little incentive for career-minded academics to conduct replications. However, if the results of the Open Science Collaboration (2015) tell us nothing else, it is that old ground needs to be re-trodden.

The Registered Replication Report format in the high-impact journal Perspectives on Psychological Science seeks to change this. In this format, many teams of researchers each independently perform a close replication of an important finding in the literature, all following an identical and shared protocol of study procedures. The final report—a single paper with all contributing researchers gaining authorship—collates the findings across all teams in a meta-analysis to firmly establish the size and reproducibility of an effect. Such large-scale replication attempts in a high-profile journal such as Perspectives can only help to encourage psychological scientists to view replication as a valid area of  their research programme.

Conclusion

2011 was described as a year of horrors for psychological science. Whilst certainly improvements can be made, our discipline has made impressive strides to improve our science. In just 6 years psychological has moved from a discipline in crisis to a discipline leading the way in how to conduct strong, rigorous, reproducible research.

References

Baker, M, & Dolgin, E. (2017). Cancer reproducibility project releases first results. Nature, 541, 7637, 269.

Brandt, M.J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R., Grange, J.A., Perugini, M., Spies, J., & van ‘t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 214-224.

John L. K., Loewenstein G., Prelec D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532

Kidwell, M.C., Lazarević, L.B., Baranski, E., Hardwicke, T.E., Piechowski, S., Falkenberg, L-S., et al. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456.

Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., . . . Zwaan, R. A. (2016). The peer reviewers’ openness initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3(1), 150547.

Nosek, B.A., Spies, J.R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 6, 615-631.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349, 943.

Simons, D.J., Holcolmbe, A.O., & Spellman, B.A. (2014). An Introduction to Registered Replication Reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9, 552–555

Wagenmakers, E.-J. (2012). A year of horrorsDe Psychonoom, 27, 12-13.

Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L.J., & Kievit, R. (2012). An agenda for purely confirmatory research. Persepctives on Psychological Science, 7, 632-638.

Wicherts, J.M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE 6(11), e26828.

Wicherts, J.M., Borsboom, D., Kats, J., & Molenaar, D. (2006) The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728.

Advertisements

Low Power & Effect Sizes

Yesterday I posted the following tweet which has since turned out to be my most popular tweet EVER with hundreds of retweets and “likes” in 24 hours:

My motivation for the tweet was quite straightforward. I have recently been emailing academics in my department every week with different topics in an attempt to raise awareness of topics associated with increasing the information value of the research we are conducting as a department. This week’s topic was “Power”. In my email—in which I included a copy of Button et al.’s (2013) excellent paper on low-power in the neurosciences—I mentioned in passing that power is not just an issue for statistical significance. I have heard from people before that low power is only an issue when interpreting null results, and that if a study produces a significant outcome, then power is not an issue.

To pre-empt this response to my encouragement to increase the power of our studies, I said in my email: “Studies with low power and significant effects have been shown to over-estimate effect sizes, meaning your low-powered study—although significant—is not giving you precision.”

As soon as I sent the email, I realised that I couldn’t recall ever reading a study that had demonstrated this. Now, I knew that such a study (or studies) would have been conducted, but I realised that I had never actually read it myself. It turns out that such studies have indeed been conducted before, as people helpfully pointed out to me on Twitter in response to my tweet:

As I was unaware of these studies—plus it was a Sunday, and I was a little bored—I thought instead of doing a literature search I would code a simulation demonstrating the inflation of effect sizes with low-powered, significant, studies the results of which I emailed to my department to demonstrate that what I had said was indeed the case. Then I thought, “Well, I haven’t tweeted much this year, so why not put it on Twitter, too.”

The incredible engagement I have had with this tweet—I propose—is due to this being a rather under-appreciated fact. Indeed, I “knew” that low-powered studies over-estimate effect sizes, but I didn’t KNOW it in the sense that I had seen hard evidence for it.

 

Details of Simulation

Because my tweet was made in passing, I didn’t explain in much detail about the stimulation implementation. I discuss this here in case others want to extend the simulation in some way.

The effect size of interest is a measure of correlation between two measures. I arbitrarily chose IQ (mean = 100, SD  = 20) and response time (mean = 600ms, SD = 80ms). I fixed the “true” effect size to be r = 0.3. It turns out that to obtain 80% power for an r=0.3 requires 85 subjects. In my simulation, I wanted to explore a wide range of sample sizes, so chose the set 10, 20, 30, 50, 85, 170, 500, and 1000.

For each sample size—N—I simulated 1,000 “studies”. For each simulated study, the following procedure occurred:

  • Sample N draws from a multivariate normal distribution with the means and SD for IQ and RT as above and a population correlation coefficient of 0.3
  • Conduct a Pearson’s correlation between the two samples
  • If the correlation was significant, store the observed correlation coefficient in a new data frame
  • If the correlation was not significant, move on without storing anything
  • After 1,000 studies are completed, plot a boxplot of the observed effect sizes for N

The result was the image in the tweet.

Limitations

Many limitations exist to this simulation, and I point interested readers to the material cited above in others’ tweets for a more formal solution. I didn’t intend for this to be a rigorous test, so it shouldn’t be taken too seriously; it was more for my own curiosity and also to provide a graphical image I could send to my colleagues at Keele to show the imprecision of effect sizes with low power. The particular outcomes are likely sensitive to my choice of means, SDs, r, etc. So, don’t generalise the specifics of this simulation, but maybe code your own tailored to your study of interest.

For me this was a bit of fun. Ten minutes of coding was time well spent on Sunday!

10 Recommendations from the Reproducibility Crisis in Psychological Science

This week I gave an internal seminar at my institution (Keele University, UK) entitled “Ten Recommendations from the Reproducibility Crisis in Psychological Science”. The audience was to be faculty members and psychology graduate students. My aim was to collate some of the “best-practices” that have emerged over the past few years and provide direct advice for how researchers and institutions can adapt their research practice. It was hard to come up with just 10 recommendations, but I finally decided on the following:

  1. Replicate, replicate, replicate
  2. Statistics (i): Beware p-hacking
  3. Statistics (ii): Know your p-values
  4. Statistics (iii): Boost your power
  5. Open data, open materials, open analysis
  6. Conduct pre-registered confirmatory studies
  7. Incorporate open science practices in teaching
  8. Insist on open science practices as reviewers
  9. Reward open science practices (Institutions)
  10. Incorporate open science into hiring decisions (Institutions)

The link to the slides are below. I might expand upon this in a fuller blog post in time, if there is interest.

 

 

My Voluntary Commitment to Research Transparency & Open Science

The past few weeks have been quite an exciting time for psychological science, with the much-anticipated publication of the Open Science Collaboration’s reproducibility project results in the journal Science. As should be well-known to you by now, the findings were bleak. (For what I believe to be the best summary of this paper, see Ed Yong’s blog post.)

As a contributor to this project, I have been thinking about issues surrounding reproducibility & robustness of methods in psychology for some time. Clearly, something is not right in the way we conduct psychological research, and I am convinced that things need to change. Now.

Felix Schönbrodt recently posted a “Voluntary commitment to research transparency and open science” on his blog, where signatories can commit to the statements contained within. Reading over the statements in the commitment, I believed whole-heartedly that were all researchers to conduct their science according to the commitment, psychology would be in a better place.

Today, I signed this voluntary commitment. Below I re-post this commitment. The wording is identical to that of Felix and his colleagues, but I have added one point (point number 4, below). Although Felix has initiated this commitment, each commitment is personal, and he encouraged personalisation of future commitments. What matters is that researchers publicly declare their commitment. This is a very inspiring initiative. and I would like to thank Felix and his colleagues for leading this. 

Voluntary Commitment to Research Transparency & Open Science

We embrace the values of openness and transparency in science. We believe that such research practices increase the informational value and impact of our research, as the data can be reanalyzed and synthesized in future studies. Furthermore, they increase the credibility of the results, as an independent verification is possible.

Here, we express a voluntary commitment about how we will conduct our research. Please note that to every guideline there can be justified exceptions. But whenever we deviate from one of the guidelines, we give an explicit justification for why we do so (e.g., in the manuscript, or in the README file of the project repository).

As signatories, we warrant to follow these guidelines from the day of signature on:

Own Research

  1. Open Data: For every first authored publication we publish all raw data necessary to reproduce the reported results on a reliable repository with high data persistence standards (such as the Open Science Framework).
  2. Reproducible scripts: For every first authored publication we publish reproducible data analysis scripts.
  3. We provide (and follow) the “21-word solution” in every publication: “We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”1 If necessary, this statement is adjusted to ensure that it is accurate.
  4. For every first authored publication, we submit a pre-print of the manuscript to a dedicated archive (such as the PeerJ or the Social Science Research Network), to ensure accessibility of the material contained within to all researchers in a timely fashion.
  5. As co-authors we try to convince the respective first authors to act accordingly.

Reviewers

  1. As reviewers, we add the “standard reviewer disclosure request”, if necessary (https://osf.io/hadz3/). It asks the authors to add a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes.
  2. As reviewers, we ask for Open Data (or a justification why it is not possible).2

Supervision of Dissertations

  1. As PhD supervisors we put particular emphasis on the propagation of methods that enhance the informational value and the replicability of studies. From the very beginning of a supervisor-PhD student relationship we discuss these requirements explicitly.
  2. From PhD students, we expect that they provide Open Data, Open Materials and reproducible scripts to the supervisor (they do not have to be public yet).
  3. If PhD projects result in publications, we expect that they follow points I. to III.
  4. In the case of a series of experiments with a confirmatory orientation, it is expected that at least one pre-registered study is conducted with a justifiable a priori power analysis (in the frequentist case), or a strong evidence threshold (e.g., if a sequential Bayes factor design is implemented). A pre-registration consists of the hypotheses, design, data collection stopping rule, and planned analyses.
  5. The grading of the final PhD thesis is independent of the studies’ statistical significance. Publications are aspired; however, a successful publication is not a criterion for passing or grading.

Service to the Field

  1. As members of committees (e.g., tenure track, appointment committees, teaching, professional societies) or editorial boards, we will promote the values of open science.

References

1Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012). A 21 word solution. Retrieved from: http://dx.doi.org/10.2139/ssrn.2160588

2See also Peer Reviewers’ Openness Initiative: http://opennessinitiative.org/

How many times is “Science” mentioned in each party’s manifesto?

The U.K. General Election is fast approaching, and each political party is clamouring for the public’s attention. All main parties have now published their manifestos: the document outlining plans and policies the party will implement if in office. In them you will find promises of reducing unemployment, cutting the nation’s deficit, and improving the National Health Service. 

But what about science? I was interested in whether the parties had mentioned science at all in their manifestos. After all, each manifesto page is prime real-estate for publicly-popular policies (hey, how else can the party get nominated?). So, I downloaded each party’s manifesto, and did a search for the occurrence of the word “science”. The results are plotted below, with some caveats:

  1. it was merely a count of the word “science” appearing in each manifesto—I made no judgement about what was being discussed (i.e., it could have said “We will never invest in science”, and would still receive a count of one
  2. The number of pages (i.e., the scope of the real-estate) varied between parties, so there is a second plot which shows the count as a proportion of total page numbers in the manifesto.

R Code for Reproducing the Plots

#------------------------------------------------------------------------------
# clear all
rm(list = ls())

# you need the ggplot2 package for this
library(ggplot2)

# declare parties
Party <- c("Conservative", "Labour", "Lib. Dems.", "UKIP", "SNP",
 "Plaid Cymru", "Green")
Party <- factor(Party, levels = Party)

# count for the word "science"
Count <- c(16, 3, 7, 6, 1, 5, 10)

# pass all to a data frame
allData <- data.frame(Party, Count)
#------------------------------------------------------------------------------

#------------------------------------------------------------------------------
## do the plotting for raw counts of the word "Science"

# define party colours
# (see http://www.stat.columbia.edu/~tzheng/files/Rcolor.pdf)
cols <- c("deepskyblue", "firebrick1", "gold", "purple", "goldenrod", "green4",
 "limegreen")

# now plot
p <- ggplot(data = allData, aes(x = Party, y = Count, fill = Party))
p <- p + geom_bar(stat = "identity")
p <- p + scale_fill_manual(values = cols)
# p <- p + coord_polar(theta = "y") # un-comment if you want a pie-chart
p <- p + guides(fill = FALSE)
p
#------------------------------------------------------------------------------

#------------------------------------------------------------------------------
## do the plotting of the proportion of the occurence "Science" to the number
# of pages in the manifesto

# how many pages in each party's manifesto?
nPages <- c(84, 86, 158, 76, 56, 64, 84)

# what is the proportion?
Proportion <- Count/nPages

# remove NAs
Proportion[is.nan(Proportion)] <- 0

# now plot
p <- ggplot(data = allData, aes(x = Party, y = Proportion, fill = Party))
p <- p + geom_bar(stat = "identity")
p <- p + scale_fill_manual(values = cols)
# p <- p + coord_polar(theta = "y") # un-comment if you want a pie-chart
p <- p + guides(fill = FALSE)
p
#------------------------------------------------------------------------------

 

Grant-Givers: Embrace (and Fund!) Research Without Impact

impact / n.  the benefit or contribution to society of research

wordle 2

Researchers in the U.K.—and likely elsewhere—will be no strangers to the term “impact”. Our research is supposed to have it. We are supposed to evidence it. Our institutions in their Research Excellence Framework—the “system for assessing the quality of research in UK higher education institutions”—have to provide case studies of it. But what is “impact”? Why is it deemed important? More importantly, is there room for research without impact? I argue that yes, there is room, and grant-givers should be embracing (and funding!) it.

What is Impact, & Why is it Important?

As the definition at the head of this post states, impact is generally considered to be the benefit or contribution to society of research, a definition echoed by the Economic and Social Research Council (ESRC)—Britain’s premier funder of social-science research. Importantly in the context of this post, impact is generally considered to be non-academic; just because your recent research was published in Nature, it doesn’t automatically qualify as impact.

The ESRC state—on their excellent online resources for researchers wishing to submit a grant to them—that evidence is essential to qualify something as impact, for example “…that it has been taken up and used by policy-makers, and practitioners, has led to improvement in services or business”. Impact can be generated via many avenues: via public engagement & knowledge exchange programmes, networking with stakeholders and participatory groups, direct engagement with end-users and/or practitioners etc. 

But why is impact important? It might be abundantly clear why impact might be important to university managers, as research conducted at their institutions that has led to serious impact can only boost the university’s prestige (which in turn leads to student numbers, which in turn leads to more income, which in turn….). It is also clear why the government—who typically fund a lot of research—want impact; they paid for it with taxpayers’ money, and lord knows the politicians don’t want to be seen wasting taxpayers’ money.

If research is being conducted to contribute meaningfully to society and/or the economy, then society itself should also value impact. This can ensure that decisions being made by policy-makers are based on research evidence, which can only be a good thing. Researchers also should value impact; who among us doesn’t want to contribute to society in some small way? We all want our research to be meaningful.

What’s the Problem with Impact?

Given all of the above, you would be forgiven for thinking what my problem with impact is. It boils down to the distinction between basic and applied  research. Basic research (also known as pure/fundamental research) is research conducted without a practical outcome in mind; that is, it is the pursuit of knowledge for its own sake. Applied research applies knowledge and the scientific method to a particular problem or a particular purpose. 

I am a basic psychological scientist. I do no applied research. I do not have applied questions in mind when conducting my research. I aim to understand a small portion of cognition: cognitive control processes. That is my aim, and only this. To understand. My problem with impact is that it is heavily biased towards applied research. This wouldn’t be problematic, necessarily, were it not for the fact that many grant-giving agencies in the U.K. require a comprehensive plan for how the research currently being proposed will lead to demonstrable—non-academic—impact. As a consequence, an application geared solely to a basic research question potentially lends itself to less impact than an application addressing an applied research question. There seems to be an imbalance. Should we prioritise research that is applied in nature? I argue NO. 

I am not the first to argue that shunning basic research because it has no immediate application is short-sighted. As Nobel prize-winning chemist Sir George Porter wrote: “To feed applied science by starving basic science is like economising on the foundations of a building so that it may be built higher. It is only a matter of time before the whole edifice crumbles.” Basic research is the foundation upon which applied research can be built. The former is important in its own right, but the latter is difficult (impossible?) without the former. 

Basic research develops human understanding, a noble pursuit in its own right. An added benefit is that with this increased understanding, applications can arise naturally. Many technologies in use today arose from basic research, without its current use in mind. My favourite example is the origin of all of our digital world: binary coding. Wikipedia states that binary code was developed by Indian scholar Pingala in the 2nd century BC. Surely this chap didn’t have digital communication via iPhones in mind when developing this method!

A problem with pursuing only applied research is that our understanding—which can only be developed via basic research—stagnates. We need the basic research to push the applied research. My favourite quote regarding this point comes from another Nobel laureate—George Smoot—who stated: “People cannot foresee the future well enough to predict what’s going to develop from basic research. If we only did applied research, we would still be making better spears“. 

Grant-Givers: Embrace (and Fund!) Research Without Impact

There exists a problem in psychological science: careers can be built on the grant income a researcher can bring to her department. A budding career means more time to devote to the research. But this raises an immediate problem: If impact is the game, and basic research has little/no impact, basic research won’t ever be a priority for funding. Therefore, those pursuing basic research will likely jump ship to the applied-research game in order to advance their careers. But as I’ve discussed, this means that psychological science will be “economising its foundations”. 

Therefore, grant-giving agencies should develop programs purely directed to basic research in psychological science. Let’s first build a strong foundation of understanding before building the skyscraper of application.

You can’t put an impact factor on that!

I’m very proud of my most recent paper (Schuch & Grange, in press), soon to be published in the Journal of Experimental Psychology: Learning, Memory, & Cognition. The pride  comes not just from being accepted in a prestigious cognitive journal—but wow, I’m certainly thrilled by it—but from aspects of the research that have no metrics. Forget your impact factors, they don’t give me the same buzz as what I experienced during this project: 

  1. It reports an effect I discovered rather serendipitously one summer whilst going through some old (published) data sets. The lay-person’s perception is that science moves forward from “Eureka” moments, whereas the truth is it occurs more along the “Hmm, that’s odd…” paths. It’s actually quite a difficult realisation sometimes because you don’t feel in control (and I’m quite the control freak!), but it’s very exciting indeed; you never know what finding is just around the corner! 
  2. It involved my first collaboration with a member of a lab in Germany I have had great respect for since starting my work in this area (the great respect is for the individual AND the lab). I remember reading a paper by Stefanie Schuch (my co-author) when I was doing my undergraduate thesis (Schuch & Koch, 2003; JEP:HPP), and that I now have a paper with her blows my mind every time I think about it. If I went back in time and told my younger self that I would soon have a paper with these authors, I really wouldn’t have believed myself. Yet here it is! It’s funny how things turn out. It’s like the academic equivalent of a guitar enthusiast getting the chance to jam with Slash.
  3. It involved exploratory analysis and confirmatory analysis, and as such is probably my most “robust” paper to date. E-J Wagenmakers has a nice paper extolling the virtues of confirmatory studies, and I am pleased to have followed much of this advice (although, the confirmatory studies were not pre-registered). As I found the effect during exploratory analysis, it would have been hasty to publish this data without seeing if it replicates in a confirmatory study, yet this is something I would have tried to do even as recently as last year. Instead, I decided to contact the German lab to see whether they find the effect in their data, too (they did). Then, we decided to run a confirmatory study. It was very nice (and re-assuring) to see it replicated in a different lab. Plus, I felt a little more “grown up” in scientific terms doing something the right (and rigorous) way. 
  4. I got to use “mini meta-analyses” for the first time. I love data analysis. Sometimes I joke with my colleagues that I would be just as happy in my job if there were no psychology involved, just so long as I was doing some form of data analysis. I just love it. Seriously, I could be measuring & analysing rainfall and I would be just as happy. I just love the process of testing ideas and getting data. So, to be able to try out a new analysis method for the first time was great fun. Meta-analyses are typically used in large-scale reviews where the meta-analysis might contain tens (or more) of data points. However, in a recent book, Geoff Cumming is very persuasive in asking the reader to think of every data point they are publishing as a point in a future meta analysis. Such “meta-analytical” thinking is important, as one experimental result in isolation means next to nothing. So, focus on the bigger picture. In the book, he recommends doing “mini meta-analyses”, where you perform a meta-analysis across only a few studies or experiments. Most papers reporting a new effect in cognitive psychology tend to have 3+ experiments reporting the effect and testing for boundary conditions etc. Cumming suggests you should do a mini meta-analysis across these studies to get an estimate of the true effect size of the new effect. This is what I did in my study. My initial exploratory data analysis was using data from a 2010 paper of mine that had 3 experiments, and a total of 5 conditions that showed the “standard” effect. So, in the current paper, I conducted a mini meta-analysis across these 5 conditions showing the magnitude (and consistency) of the “new” effect. The Figure reporting this meta analysis is below. It was really neat trying out new techniques, and I think it really added weight to the paper. I shall certainly be using this again in the future!
mini-meta

To sum up this short post, this project was very exciting and satisfying from a scientific perspective. I would have almost retained the same joy from this project had it never seen the light of day.

You can’t put an impact factor on that!