Neuroscience
That's the opening sentence in an abstract by Joshua Carp, who will be presenting tomorrow in Slide Session 3 at the 2012 CNS Meeting in Chicago. The question caught my eye in light of the Psych Science paper on False-Positive Psychology ["undisclosed flexibility in data collection and analysis allows presenting anything as significant"] and the recent blog post by Dr Daniel Bor on The dilemma of weak neuroimaging papers.
Reference
Simmons JP, Nelson LD, Simonsohn U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 22:1359-66.
- How Much Of The Neuroimaging Literature Should We Discard?
Guilty article in PNAS (2003). Just a quick pointer to a pair of posts on how sub-optimally designed and analyzed fMRI studies can continue to influence the field. Professor Dorothy Bishop of Oxford University posted a scathing analysis of one such paper,...
- Quotidian Virtual Violence
The title of this article (below) and its abstract certainly caught my eye (they actually did use the word "quotidian" and did state that playing violent video games is a natural, real-life behavior). Human Brain Mapping Early View (Articles online in...
- Neuropsychology Abstract Of The Day: Stretch!
Kashdan TB, & Rottenberg J. Psychological flexibility as a fundamental aspect of health. Clinical Psychology Review. 2010 Mar 12. Traditionally, positive emotions and thoughts, strengths, and the satisfaction of basic psychological needs for belonging,...
- Neuropsychology Abstract Of The Day: Adhd Prevalence
Polanczyk G, de Lima MS, Horta BL, Biederman J, & Rohde LA. The Worldwide Prevalence of ADHD: A Systematic Review and Metaregression Analysis. American Journal of Psychiatry. 2007 Jun; 164(6): 942-948. Child and Adolescent Psychiatric Division, Hospital...
- Who Watches The Watchmen? Bias In Studying Bias
The coins are often very old by the time they reach the jeweller With his hands and ashes he will try the best he can He knows that he can only shine them Cannot repair the scratches The Jeweller (by Pearls Before Swine) Publication...
Neuroscience
How vulnerable is the field of cognitive neuroscience to bias?
That's the opening sentence in an abstract by Joshua Carp, who will be presenting tomorrow in Slide Session 3 at the 2012 CNS Meeting in Chicago. The question caught my eye in light of the Psych Science paper on False-Positive Psychology ["undisclosed flexibility in data collection and analysis allows presenting anything as significant"] and the recent blog post by Dr Daniel Bor on The dilemma of weak neuroimaging papers.
Slide Session 3
Sunday, April 1, 10:00 am - 12:00 pm, Red Lacquer Room
Estimating the analytic flexibility of functional neuroimaging: Implications for uncertainty and bias in cognitive neuroscience Joshua Carp; University of MichiganHow vulnerable is the field of cognitive neuroscience to bias? According to a recent mathematical model, the potential for scientific bias increases with the flexibility of analytic modes. In other words, the greater the range of acceptable analysis strategies, the greater the likelihood that published research findings are false. Thus, the present study sought to empirically estimate the analytic flexibility of fMRI research. We identified five pre-processing decisions and five modeling decisions for which two or more analysis strategies are commonly used in the research literature. By crossing each of these strategies and decisions, we identified 4,608 unique analysis pipelines. Next, we applied each of these pipelines to a previously published fMRI study of novelty detection in an auditory oddball task. We found that activation estimates were highly dependent on methodological decisions: contrasts that yielded significant positive activation under one pipeline were associated with non-significant positive activation or even with negative activation under other pipelines. Some analysis decisions contributed more to this variability more than others, and each decision exerted a unique pattern of variability across the brain. The effects of a given decision also varied across contrasts, subjects, and other analysis parameters. In sum, we found considerable quantitative and qualitative variability across analysis pipelines, suggesting that the results of cognitive neuroimaging experiments may be more uncertain than they seem. Indeed, given a supercomputer, a sufficiently motivated analyst might observe almost any imaginable pattern of results.Reference
Simmons JP, Nelson LD, Simonsohn U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 22:1359-66.
- How Much Of The Neuroimaging Literature Should We Discard?
Guilty article in PNAS (2003). Just a quick pointer to a pair of posts on how sub-optimally designed and analyzed fMRI studies can continue to influence the field. Professor Dorothy Bishop of Oxford University posted a scathing analysis of one such paper,...
- Quotidian Virtual Violence
The title of this article (below) and its abstract certainly caught my eye (they actually did use the word "quotidian" and did state that playing violent video games is a natural, real-life behavior). Human Brain Mapping Early View (Articles online in...
- Neuropsychology Abstract Of The Day: Stretch!
Kashdan TB, & Rottenberg J. Psychological flexibility as a fundamental aspect of health. Clinical Psychology Review. 2010 Mar 12. Traditionally, positive emotions and thoughts, strengths, and the satisfaction of basic psychological needs for belonging,...
- Neuropsychology Abstract Of The Day: Adhd Prevalence
Polanczyk G, de Lima MS, Horta BL, Biederman J, & Rohde LA. The Worldwide Prevalence of ADHD: A Systematic Review and Metaregression Analysis. American Journal of Psychiatry. 2007 Jun; 164(6): 942-948. Child and Adolescent Psychiatric Division, Hospital...
- Who Watches The Watchmen? Bias In Studying Bias
The coins are often very old by the time they reach the jeweller With his hands and ashes he will try the best he can He knows that he can only shine them Cannot repair the scratches The Jeweller (by Pearls Before Swine) Publication...