Distribuição do p-valor

No artigo abaixo, o autor mostra que a distribução do p-valor é extremamente assimétrica e volátil. Assim, p-valor de 0.05 (o ideal é 0.005, segundo o autor) e poder do teste de 90% (ideal é 99%, segundo Taleb) não fazem sentido.

Resumo:

We present an exact probability distribution (meta-distribution) for p-values across ensembles of statistically identical phenomena, as well as the distribution of the minimum p-value among m independents tests. We derive the distribution for small samples 2<n≤n∗≈30 as well as the limiting one as the sample size n becomes large. We also look at the properties of the "power" of a test through the distribution of its inverse for a given p-value and parametrization.
P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude.
The formulas allow the investigation of the stability of the reproduction of results and "p-hacking" and other aspects of meta-analysis.
From a probabilistic standpoint, neither a p-value of .05 nor a "power" at .9 appear to make the slightest sense.

Fonte: Taleb, Nassim Nicholas, The Meta-Distribution of Standard P-Values (September 3, 2016). Available at http://dx.doi.org/10.2139/ssrn.2834266