Implausibility and Common Sense

One of Ben Goldacre’s criticisms of the pharmaceutical industry in his new book Bad Pharma (which I’d recommend), is that publication bias is rife – trials with positive results are much more likely to be published than are trials that find no effect – when this process is controlled by people (pharmaceutical companies), with an incentive to show their drug is effective, this is incredibly dangerous but completely unsurprising.

A piece from the Atlantic, suggests that there is something else we should be looking out for – a kind of ‘novelty’ bias. The piece deals with social psychology and the efforts of Uri Simonsohn to uncover research fraud in that field, but the lessons are clearly more widely applicable (that the discipline of social psychology is reliant on large number of small-scale lab experiments makes them particularly vulnerable, however – it allows for easier manipulation of the data by unscrupulous researchers). 

Even when out-and-out research fraud isn’t occurring (please remember that these cases represent an incredibly minority), there is an incentive for researchers to build their theories around their most novel finding.

Researchers are (in my limited experience) human, and so will err, just like everyone else, and become excited by the new and novel. Some burden lies on the readers of research to work out whether they buy the finding – particularly the size of the effect (it stands to reason that bias would be strongest among results which are both novel and large in their effect). We shouldn’t reject the results of properly robust research out of hand because it doesn’t “feel right” – but where a result is perhaps too good to be true, we should look for replications that confirm the effect (publication bias may make this difficult) and exercise some common sense. For example, I’ve been reading a lot of papers evaluating strategies for crime prevention, and at least 5 that I’ve come across this week report 60% or more reductions in crime. If these results are correct and even remotely additive, it is a wonder there is any crime left.  

December 19, 2012 | Categories: Uncategorized | Tags: crime, evidence, experiment, rct, socialpsych | Leave a comment