Month: June 2014

Dealing with Rejection

If there is one thing academics need to get used to very quickly it’s rejection. I’ve had a multitude of papers rejected, I’ve had several grants (large and small) rejected, and I even once had a conference submission rejected (yep – it CAN happen). There’s no other way to say it: rejection hurts.  It’s difficult—sometimes it feels impossible—but I try to see the positive side of rejections where I can.

Rejected

 

I clearly remember finding my first rejection less difficult than I had anticipated. It was within the first three months of my PhD, and I had the opportunity to submit the first experiment of my project as a short commentary paper. When the reviews came back, I was disappointed to see it had been rejected; in fact, this is likely an understatement, as I had high hopes it would be lauded and immediately put on the front page of the journal (OK, this is an exaggeration, but I thought it was a nice little paper). Despite my disappointment, I remember the next day reading the reviews again through fresh eyes, and smiling to myself. It was an unexpected smile; one of those that take you by surprise when you suddenly realise you’re doing it, and you’re not sure why. I had the sudden realisation that my work—yes, MINE!—had been looked at by three experts in my field. These researchers—whom I respected greatly—had taken the time to look at my work and provide me with valuable feedback. Was it all positive? No, of course not, but there were positive aspects. Was it all negative? No, of course not, but there were negative aspects. The point is, I had received critical feedback on my work from three experts in my field. What a learning experience!

That’s not to say I’ve always found rejection plain-sailing. Despite the positives to be gained from the reviews of a rejected paper, I’ve recently had a bad run of rejections which I found very difficult: seven consecutive paper rejections (not all the same paper). After a while (maybe the fourth?), it was difficult to not start questioning myself: Am I up to this job? Imposter syndrome had kicked in royally. This rejection-fest has recently ended, and I had a paper accepted the other day. This paper contains work I am most proud of to date, but imposter syndrome is still here.

It never helps my cause that I also have a very bad habit of looking at people at similar stages of their career to me and looking at their extensive CVs. I have an even worse habit of looking at professors’ CVs and trying to work out how I shape up in comparison to them when they were at my stage of career. Some of these comparisons give me hope; other comparisons leave me feeling even more incapable. (Does anyone else do this, too? If you don’t, DO NOT start doing it; what a complete waste of time and energy!)

These negative feelings are common, and the more academics I speak to the more I realise I’m not the only one who feels this way. My greatest discovery was finding out that other people have rejections, too; you can’t help but feel sometimes that you are the only one! This realisation came from when I started to review other people’s work, as reviewers get cc’d in to the decision letter (“Bloody hell, even Professor XXX gets rejections!”). I’m not the only one, it seems, and neither are you.

How do I deal with rejection?

I feel that I have a pretty broad back when it comes to rejections. Yes, I feel crap about it for a little while. But, I try not to let it dominate my thoughts. I have a pretty good routine for dealing with rejections, which I want to briefly outline below. My process is not novel, and I remember reading something similar from someone else, but for the life of me I can’t recall where I saw it.

  • When I receive the decision letter, I read the editor’s comments first (obviously). This tends to be a panic-stricken scan for the word “unfortunately” rather than a comprehensive read, but I can quickly assess the damage.
  • I read through the reviewers comments once. I aim to get the broad “feel” for the issues that have been raised, but at this stage I don’t focus on the details so much.
  • I put the reviews away in an email folder.
  • I do not look at the reviews again for at least a couple of days. This is my “licking my wounds” phase. I try to fill it with as many positive things as I can. If I’m fortunate enough to have another paper I am working on I continue working on it during this phase. For me, this is very important and serves two purposes: to take my mind off the rejection, and to make me feel like I am still progressing (doing nothing during this time has the danger of making one feel like you’ve been rejected and there’s nothing you can do about it).
  • After a few days, I return to the reviews. I often realise at this stage that the reviewers actually raised some very insightful and important points. I note all of these down.
  • In a revision—either back to the same journal or in revision to submit the paper elsewhere—I make sure I deal with ALL comments. This doesn’t mean I do everything a reviewer asks for, but I do make sure I can defend why I haven’t done a particular thing. 
  • Craft my revision letter. This often turns into a very lengthy document. I’ve had revision letters that have been just as long as the manuscript I’m revising. I outline in detail every point that every reviewer raised, and point to where in the manuscript the change is, or—if I didn’t agree with a point—I elaborate why I haven’t included it. I want to leave no doubt in the editor’s mind that I’ve thought deeply about the issues raised, and I’ve acted in an open and responsive manner. I don’t do it for brownie-points; I do it because I take the reviewers’ comments very seriously and I want to ensure I provide a comprehensive response.
  • I acknowledge the work of the reviewers in my revision letter. They’ve taken the time out of their busy research schedule to assess my work and I am always grateful for that, whether they recommend acceptance or not.
  • I also try to remember some sage advice: you’re only being rejected because you’ve been productive enough to submit something.

Ironically enough, as I was writing these last points I had another rejection through (a small teaching-related grant). Time to go lick my wounds and start the cycle all over again…

 

Advertisements

The benefit of a lab book

I always though that use of a lab book—a dedicated space to note down experimental methods, results etc.—was largely restricted to sciences like chemistry or physics, where one might conduct several experiments per week, and thus tracking progress is essential lest you lose your train of thought. What role do they have in psychology, where the pace of experimentation—both of the experiment itself and the time between each experiment—is arguably much more sedate?

Since January this year I’ve been keeping my own lab book, and it has boosted organisation of my thoughts and progress considerably. It’s just a simple word document, organised by themes. I have chapters titled “Experiments”,  “Models”, and “Research Ideas”. It has a table of contents, list of figures, and list of tables. It also has a references section. Its layout is very much like a thesis.

Now, in one place, I log all of my experiments in as much detail as I would in a paper submission (minus the protracted introduction & discussion). Usually I would write up only those experiments which “worked”, in that I would be preparing them for rejection submission, but it has been a revelation writing up all of my experiments. All of them were important enough to me to run; all were sufficiently powered and—in my opinion—sufficiently designed to address a question I have, so why not write them up somewhere? It’s better than letting the data rot in an electronic file drawer.

Keeping everything in one place has made me feel much more in control of my work. The enhanced feeling of organisation having one lab book is liberating. I carry a dictaphone around with me in case I have thoughts whilst in a situation I can’t write in; now, I put the audio file in dropbox and have an entry to my lab book with a link to this file. I also carry a small note book around with me; instead of keeping notes contained in this book, I can take a snapshot with my smartphone and enter the photo as a figure in my lab book. 

It also gives me a feeling of accomplishment. Academia is famed for delayed rewards (if they come at all), so it is nice to be able to look back and see that I have made some progress, even if most of it will never see the light of day. Science is a cumulative process, and the steps that make this progress are small, and often made behind closed doors; published work doesn’t always reflect these small steps, so I like having a permanent record of mine.  

The greatest benefit has come from organising the modelling work which I try to do. Modelling requires a lot of tedious steps, most of which get scrubbed from final reports: what did you try first that didn’t work? What tests did you do to check the model code was bug-free? Did you do parameter-recovery simulations? What about testing for model mimicry and model-recovery simulations? Each of these steps require new scripts of code, new parameters, and new data. Before now, I would continuously update one script to cover all of these stages, and the final script would be long and complex with little acknowledgement of its heritage. I would know that I’ve conducted these stages, but I wouldn’t log the results anywhere. I would just “know” that they are complete. Now, I log each of these stages—and their results—in my lab book, together with links to archived code for each stage. This makes the process much cleaner, and I feel more confident about the final product.

Open Lab Book?

One step I have not yet had the confidence to take is to go open; open lab books are those which are kept “live” on the internet, so others can see. Wikipedia has a great section on open lab books with links to examples from other sciences. 

I see many advantages to this: notably, science is—or at least, should be—an open dialogue, so why not let others see what I’m doing? Perhaps I would get some comments/ideas that aid my research. Of course, there is the fear of being “scooped”, but I don’t think this is a fear worth entertaining (at least, not for the work I do). What puts me off doing it is it would change the way I write in my lab book. By writing just for me, I can be more economical with explanations. I can also be free to be more informal with my thoughts (“Why didn’t that bloody experiment work?”). And yes, I can also hide some of the awful ideas I have.

I prefer to use my lab book primarily as a way to organise my thoughts. With increasing demands on my time, the organisational boost it has brought has been worth its weight in gold. I’m also looking forward to looking back in 5 years’ time at all the work I have done. 

Try it! 

On the diversity of response time trimming methods

Below I outline an argument for moving towards a clearer, more objective, way to trim response times. I first discuss the importance of response time trimming, and then outline various methods commonly used by researchers. I then quantify the diversity of these methods by reviewing 3 years of articles from two prominent cognitive psychology journals, and catalogue usage of each method. I then suggest that a technique introduced by Van Selst and Jolicoeur (1994) might—and here I stress might—be a solution to the lack of objectivity in choosing which method to use. To aid its use, I provide R scripts for the trimming method by Van Selst and Jolicoeur. 

I don’t usually intend to write posts as long as this, but the text below represents a small comment paper I have been trying—unsuccessfully—to publish. Rather than it sit in my drawer, I thought I would share it here. Comments welcomed. 

Overview

Response times (RT) are an incredibly popular dependent variable in cognitive psychology, whereby researchers typically take a measure of central tendency of the distribution of total RTs for a given condition (often the mean, but sometimes the median) to infer the time-course of discrete psychological processes. The challenge facing researchers is how to best deal with so-called outliers: A small proportion of RTs that lie at the extremes of the RT distribution and thought to arise from processes not under investigation. These outliers can occur at the slow end of the distribution (e.g. due to distraction, lack of alertness etc.) or at the faster end (e.g. a rapid anticipatory response). As these outliers can influence the estimate of central tendency—and hence contaminate the estimate of the psychological process under investigation—researchers typically remove outliers (“trimming”) before conducting inferential analysis. 

But what method should be used to identify outliers? This question turns out to be very challenging to answer (with no necessarily correct answer); as such, there exists a vast and diverse range of methods typically employed. Some statisticians in fact recommend not trimming RT data at all (see e.g. Ullrich & Miller, 1994). Alternatives include taking the median (which is less affected by extreme scores than the arithmetic mean), fitting a model to the entire RT distribution (Heathcote, Popiel, & Mewhort, 1991), analysing cumulative distribution frequencies (Ratcliff, 1979; see Houghton & Grange, 2011), or applying one of a class of process models of response time (e.g. Wagenmakers, 2009).

However, if RT trimming is to be used for calculation of mean RT, it is desirable that the method employed is as objective as possible.  The inconsistency of possible methods—at best—leaves researchers in a quandary how best to process their data; at worst, it increases researcher degrees of freedom (Simmons, Nelson, & Simonsohn, 2011): The increased flexibility in choosing which RT trimming method to use might increase false-positive rates. 

The purpose of this post is to highlight the diversity in methods so that researchers are cognizant of the issue; I also propose that researchers consider establishing a standardised method of response time trimming using objective criteria. My issue is not with the trimming methods per se, but rather the potential for a lack of objectivity in selecting which method to use. In this day of increasing concern of replicability in psychological science, it is imperative to start the discussion regarding a uniform method of RT trimming. One candidate method was introduced by Van Selst and Jolicoeur (1994), but it is relatively complicated to implement. To facilitate its use, researchers have provided routines in proprietary software (e.g. SPSS; Thompson, 2006); in addition, I provide routines in the statistical package R (R Development Core Team, 2012) that implements this method.

Quantifying the Diversity

In an attempt to informally quantify the diversity of trimming methods employed, Table 1 catalogues the frequency of a number of diverse trimming methods reported in the 2010–2012 volumes of Journal of Experimental Psychology: Learning, Memory, & Cognition, and Quarterly Journal of Experimental Psychology. What strikes me is the sheer number of trimming-options researchers have to choose from. 

No trimming is where no clear report of a trimming method could be found in the article. This, of course, does not necessarily mean that trimming was not employed, so is likely an over-estimate of the true number of studies that did not employ trimming. Absolute cut-off involves identifying an absolute upper- and lower-limit on RTs to include in the final analysis (e.g. “RTs faster than 200ms and slower than 2,000ms were excluded from data analysis”). Standard deviation (SD) trimming comes in different guises: Global SD trim removes any RTs that fall outside of a certain number of SDs from the global mean (i.e. across all participants and conditions; e.g. “RTs slower than 2.5 SD of the mean were excluded”); per cell SD trimming removes RTs outside of a certain number of SDs from the global mean of each experimental cell (“RTs slower than 2.5 SD of the mean of each experimental condition were excluded”); per participant trims RTs outside of certain number of SDs from the mean of each participant’s overall RT (“RTs slower than 2.5SD of the mean of each participant were excluded”); per cell, per participant is arguably more fair, as it trims RTs from all participants for all conditions, and hence will certainly trim from all experimental conditions (e.g. “RTs slower than 2.5 SD of the mean of each participant for each condition were excluded”). 

 

Table 1

Lack of Objectivity?

The main issue with all of the above trimming methods is their potential lack of objectivity; for example, when using an absolute cut-off, what criteria should one use for deciding on the upper limit? The choice from the articles reviewed showed that the choice ranged from 800 milliseconds (ms) to 10,000ms. Obviously, the choice will be influenced by the difficulty of the task, but even with a relatively simple task, how does one choose whether to use 2,000ms or 2,500ms as the upper limit? The lower limit is potentially simpler, as it defines the value below which responses were likely anticipatory (i.e. unrealistically fast); but even in this simpler case, there was a wide range of limits used, ranging from 50ms to 400ms. As such, a popular alternative to the absolute cut-off is to allow the data itself to identify outliers, by removing RTs above a certain number of standard deviations (SDs) above the mean. However, this process too might suffer from a lack of objectivity, as how does one decide on the SD value to use (2.5SDs or 3SDs?). In the articles reviewed, the SD chosen for the trimming ranged from 2 to 4.

 

A Potential Solution?

A strong candidate for an objective response time trimming method was introduced by Van Selst and Joliceur (1994). They noted that the outcome of many trimming methods is influenced by the sample size (i.e. number of trials) being considered, thus potentially producing bias. For example, even if RTs are drawn from identical positively-skewed distributions, a “per cell per participant” SD procedure would result in a higher mean estimate for a small sample size “condition” than a large sample size condition. This bias was shown to be removed when a “moving criterion” (MC) was used; this is where the SD used for trimming is dynamically adapted to the sample size being considered. This meets the criteria for objectivity in a trimming method as the SD used for calculating cut-off values is not determined by the researcher, but by the sample size under investigation. Thus, this method is an excellent candidate for a standardised trimming procedure.

Van Selst and Jolicoeur (1994) introduced two MC methods that reduced the bias with sample size: The non-recursive (MC) method removes any RT that falls outside a certain number of SDs from the mean (of the whole sample) being considered, with the value of SD being determined by the sample size of the distribution, with a lower SD value being used for smaller sample sizes (see Table 4 in Van Selst & Jolicoeur). The modified recursive (MC) procedure performs trimming in cycles. It first temporarily removes the slowest RT from the distribution; then, the mean of the sample is calculated, and the cut-off value is again calculated using a certain number of SDs around the mean, with the value for SD being determined by sample size (in this procedure, required SD decreases with increased sample size; see Van Selst & Jolicoeur for justification). The temporarily removed RT is then returned to the sample, and the fastest and slowest RTs are then compared to the cut-off, and removed if they fall outside. This process is then repeated until no outliers remain, or until the sample size drops below four. The SD used for the cut-off is thus dynamically altered based on the sample size of each cycle of the procedure. Van Selst and Jolicoeur reported slight opposing trends of these two methods, suggesting a “hybrid moving criterion” method (see their footnote 2, page 648) which simply takes the average of the non-recursive (MC) and modified recursive (MC) procedures. 

Although the non-recursive (MC) procedure is relatively simple to implement with equal sample sizes between conditions and participants in standard software such as Excel, the modified recursive (MC) procedure and the hybrid present some technical challenges. Specifically, the modified recursive procedure requires many cycles of removing individual RTs, calculating means, establishing a dynamic SD criterion based on the current sample size on the current cycle, replacement of RTs, and trimming; the procedure must also be aware of the stopping rule when the sample drops below four. 

Of course, the Van Selst and Jolicoeur (1994) method is just one possible approach, and the field might not reach a consensus as to which method could become the standard (or might not even want a consensus). As such, the field might continue (quite understandably) to use any one of a number of methods; but at the very least, I recommend that researchers should justify explicitly why they chose the method of RT trimming they did, and potentially demonstrate whether the pattern of results changes depending on the method employed. Such disclosure will allow readers to assess to what degree the results presented might be reliant on the trimming method chosen.

 

R Script for Implementing Van Selst & Jolicoeur (1994)

To facilitate implementation of this method for researchers, I provide routines in the statistical package R; this set of scripts is capable of executing all three of the methods recommended by Van Selst and Joliceour (1994), and also includes a “quick start guide” for users unfamiliar with R. The scripts can be downloaded from Github.

 

References

Heathcote, A., Popiel, S.J., & Mewhort, D.J.K. (1991). Analysis of response time distributions—An example using the Stroop task. Psychological Bulletin, 109, 340-347.

Houghton, G. & Grange, J.A. (2011). CDF-XL: Computing cumulative distribution frequencies of reaction time data in Excel. Behavior Research Methods, 43, 1023-1032.

Ratcliff, R. (1979). Group reaction time distributions and an analysis of distribution statistics. Psychological Bulletin, 86, 446-461.

R Development Core Team. (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from http://www.R-project.org/.

Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.

Thompson, G.L. (2006). An SPSS implementation of the non-recursive outlier detection procedure with shifting z-score criterion (Van Selst & Jolicoeur, 1994). Behavior Research Methods, 38, 344-352.

Ulrich, R. & Miller, J. (1994). Effects of truncation on reaction time analysis. Journal of Experimental Psychology: General, 123, 34-80.

Van Selst, M. & Jolicoeur, P. (1994). A solution to the effect of sample size on outlier elimination. Quarterly Journal of Experimental Psychology, 47 (A), 631-650.

Wagenmalers, E.-J. (2009). Methodological and empirical developments for the Ratcliff diffusion model of response times and accuracy. European Journal of Cognitive Psychology, 21,641-671.