Neuroscience
- Dramatic Study Shows Participants Are Affected By Psychological Phenomena From The Future
Perhaps there's something in the drinking water at Cornell University. A new study involving hundreds of Cornell undergrads has provided a dramatic demonstration of numerous 'retroactive' psi effects - that is, phenomena that are inexplicable...
- Psychology's Real Replication Problem: Our Methods Sections
Replication has been a big topic in psychology recently as a) we've suddenly realised we need more of it and b) because there have recently been several high profile replication efforts published (e.g the Many Labs effort; see Ed Yong's summary)....
- Connecting The Conceptual Dots In Embodied Cognition
UPDATE 9 June 2015: We published this critique. Around about the time we published our embodied cognition paper, we also reviewed a paper for that research topic, which happened to be an example of the conceptualisation hypothesis ...
- Replication Will Not Save Psychology
"Replication is our only hope." "No. There is another"Psychology is big into replication these days. A lot of people think that a major problem with the field is that many important results have not been replicated, and that this is in part because journals...
- "moving Through Time" And Embodied Cognition
In which I am a bit rude about a rubbish paper and worry about how to kill papers like it. The term 'embodied cognition' has been checked out of the library by a lot of different people, all of whom use it to mean something different. My least...
Neuroscience
Brief Note: Daryl Bem and Precognition
In case you missed it, the Journal of Personality and Social Psychology, a flagship APA journal, published a study by Daryl Bem containing evidence for psi (precognition). I didn't really want to post in too much detail about this study (which has been doing the rounds online all week) because I'm not that interested in being a science journalist. But I did feel it was worth posting a few links and some brief comments to this provocative paper, because it raises a lot of interesting questions about the business of doing psychological science.Should a paper on precognition be published in a major social psychology journal? How did Bem get his results? How seriously should we take what seems to be evidence for something that a lot of other science suggests is impossible?
Bem's paper contains 9 studies, each of which takes a classic effect (e.g. priming) and reverses the order in which things were presented, so that the stimulus we know affects behaviour came after the response had been made. Bem's logic was that if you could reliably see the known effect under these conditions, this would count as evidence that people's behaviour was being affected by precognitive knowledge of the future event. He got small but statistically significant effects in this direction in each of the 9 studies.
The paper is available here, and is summarised here and here. My basic take:
- I don't believe a word of it because a) let's face it, it's about precognition and b) there's simply no effort to propose a mechanism that might support such an outrageous claim (and no, 'quantum mechanics' is not a mechanism). Bem explicitly states that coming up with a mechanism isn't his job and he's just 'reporting the data'. But this is precisely the problem with psychology right now - not enough theory - and the links below that talk about the analysis problems with this paper (and all statistical testing in the social sciences) make good points about the fact that statistical testing in the absence of a clear theory which includes mechanisms is effectively a fishing trip. There's also a nice discussion here about why it's healthy to be immediately sceptical of a study that claims to have found something inconsistent with the rest of science.
- That said, it is a rigourous paper; there are 9 carefully run studies with a lot of detail, and the goal (to provide a framework for replication and testing of psi) is admirable. It might be wrong, but it's at least replicable and falisfiable, and I can respect that. I'm also not grumpy with JPSP for publishing it, because the paper has done what it was intended to do and sparked a lot of discussion, analysis and replication. The risk, of course, is that people will only remember the original, positive study and never hear about the failures to replicate. I hope the various science bloggers talking about this study remember to follow up.
Speaking of following up: there are already three failures to replicate that I know of, here (and they have more coming), here and here; if you are planning a replication, register it on Richard Wiseman's site here. There has been one detailed analysis of the statistical problems that are probably leading to the significant results, and this post led me to this book preview by two authors describing in great detail the problem with the kind of significance testing we do in the social sciences. Maggie Koerth-Baker has been keeping tabs on all this on her Twitter feed.
References
Bem, D (in press). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology. Download.
The real lesson? This is the level of methodological scrutiny every paper should receive, and not just the ones you think are crazy: the ones you like and rely on for your own work should get a good working over like this too (especially these ones; and I'm as guilty on this as everyone else). In the social sciences we depend entirely on statistics to make our points; this dependence on distributions makes replication an absolute necessity. I just hope this is the lesson we learn, because it's not rewarded much right now.
Update, 2010.11.18: Richard Wiseman thinks he has a methodological flaw.
Update, 2010.11.18: Richard Wiseman thinks he has a methodological flaw.
References
Bem, D (in press). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology. Download.
- Dramatic Study Shows Participants Are Affected By Psychological Phenomena From The Future
Perhaps there's something in the drinking water at Cornell University. A new study involving hundreds of Cornell undergrads has provided a dramatic demonstration of numerous 'retroactive' psi effects - that is, phenomena that are inexplicable...
- Psychology's Real Replication Problem: Our Methods Sections
Replication has been a big topic in psychology recently as a) we've suddenly realised we need more of it and b) because there have recently been several high profile replication efforts published (e.g the Many Labs effort; see Ed Yong's summary)....
- Connecting The Conceptual Dots In Embodied Cognition
UPDATE 9 June 2015: We published this critique. Around about the time we published our embodied cognition paper, we also reviewed a paper for that research topic, which happened to be an example of the conceptualisation hypothesis ...
- Replication Will Not Save Psychology
"Replication is our only hope." "No. There is another"Psychology is big into replication these days. A lot of people think that a major problem with the field is that many important results have not been replicated, and that this is in part because journals...
- "moving Through Time" And Embodied Cognition
In which I am a bit rude about a rubbish paper and worry about how to kill papers like it. The term 'embodied cognition' has been checked out of the library by a lot of different people, all of whom use it to mean something different. My least...