Evidence based policies are often built on sand

Last week we saw yet another major reversal of opinion by experts. For years we have all been lectured severely on the need to finish every single course of prescription drugs. But the latest wisdom is that this is not necessary.

The announcement that petrol and diesel cars will be banned by 2040 only serves to remind the millions of diesel car owners that they were told only a few years ago that diesel was a Good Thing.

These stories have been very prominent in the media.  But they are by no means isolated examples.  Such reversals of opinion are all too common in the softer social and medical sciences.  The “evidence base”, a phrase beloved of metropolitan liberal experts, is often built on sand.

This is neatly illustrated by psychology.  Science is probably the most prestigious scientific journal in the world.  At the end of 2015, a group of no fewer than 270 authors published a paper in it.  They were all part of the teams which had published 100 scientific articles in top psychology journals. 

In only 16 out of the 100 cases could the experimental results be replicated sufficiently closely to be confident that the original finding was valid. 

The papers had been published in top psychology journals, and the original authors were involved in the replication experiment.  So the replication rate should have been high.  Instead, it was so low that the lead author of the Science piece points out that they effectively knew nothing.  The original finding could be correct, the different result in the attempted replication could be.  Or neither of these could be true.   

There is no suggestion at all that any sort of fraud or misrepresentation was involved when the original results were submitted for publication.

But economic theory helps us understand how this situation came about.  The great insight of economics is that people react to incentives. 

Academics now face immense pressure to publish research papers.  If they do not, they get more burdensome teaching loads, don’t get promoted and might even get sacked.  Their incentive is to publish.

Academic journals will only very rarely accept a paper which contains negative results.  The whole culture is to find positive ones.  So experiments will be re-designed, run with different samples, until that sought-after positive finding is obtained.

More and more academics are now desperate to publish more and more papers.   To meet this increase in demand, there has been a massive increase in the supply of journals willing to publish.  Many of these are highly dubious, being prepared to accept papers on payment of a fee by the authors.

For all except a small elite of individuals and institutions, academic life has become increasingly proletarianised.  In the old Soviet Union, workers could get medals for exceeding the quota of, say, boot production.  It did not matter if all the boots were left footed.

Many universities are now similar, with useless articles being churned out to meet the demands of bureaucrats.  Time for a big purge, both of academics and their institutions.