Dá pra confiar nas pesquisas acadêmicas?

Publication bias in academic journals is nothing new. A finding of no correlation between sporting events and either violent crime or property crime may be analytically top class, but you couldn’t be blamed, frankly, for not giving a damn. But if journal editors are more interested in surprising or dramatic results, there is a danger that the final selection of published papers offer a distorted vision of reality.

This should skew the distribution of published results, towards more 'significant' findings. But a paper just published in the American Economic Journal finds evidence of a different sort of bias, closer to the source. Called "Star Wars, the empirics strike back", it analyses 50,000 papers published between 2005 and 2011 in three top American journals. It finds that the distribution of results (as measured by z-score, a measure of how far away a result is from the expected mean) has a funny double-humped shape (see chart). The dip between the humps represents "missing" results, which just happen to be in a range just outside the standard cut-off point for statistical significance (where significance is normally denoted with stars, though the name may also be something to do with a film recently released—file under 'economists trying to be funny'). Their results suggest that among the results that are only just significant, 10-20% have been fudged.

Continua aqui
Resumo:

Using 50,000 tests published in the AER, JPE, and QJE, we identify a residual in the distribution of tests that cannot be explained solely by journals favoring rejection of the null hypothesis. We observe a two-humped camel shape with missing p-values between 0.25 and 0.10 that can be retrieved just after the 0.05 threshold and represent 10-20 percent of marginally rejected tests. Our interpretation is that researchers inflate the value of just-rejected tests by choosing "significant" specifications. We propose a method to measure this residual and describe how it varies by article and author characteristics. (JEL A11, C13)

Brodeur, Abel, Mathias Lé, Marc Sangnier, and Yanos Zylberberg. 2016. "Star Wars: The Empirics Strike Back." American Economic Journal: Applied Economics, 8(1): 1-32.