This article aims to provide some of the derivations from Ioannidis' 2005 article: "Why Most Published Research Findings Are False" which exposed what has since been termed "The Replication Crisis."
The issue begins with the subject of -values which measure the probability of a study finding a positive result, assuming the presented hypothesis is false. A strong -values is considered to be , indicating a regrettable 5% of published findings are false.
Before diving into the derivations, some examples:
Suppose we represent all possible hypotheses that can be tested with a more manageable 100,000 hypotheses. Let's allow a generous 50:50 true:false split for this set as well as a statistical power of 80%.
|positive result ||40k||2.5k||42.5k|
|negative result ||10k||47.5k||57.5k|
Here, the -value where is a positive relationship, and is a flase result. The statistical power and Positive Predictive Value which is pretty satisfactory given our generous values.
Once again, we'll take 100,000 hypotheses, but now with a 10:90 true:false split for this set as well as a statistical power of 80%. Filling out the table we get:
Here, which is significantly worse than the assumed 95% if the study is positive without publicaiton bias, cheating etc. which is covered below.
Before getting much further, it will be useful to define a glossary
|probability of a study finding a positive result, given that the hypothesis is false|
|Positive Predictive Value|
|the pre-study odds of the hypothesis is tested|
|an alternate expression of probability, e.g. 10:90 odds: |
|Type I Error|
|Type II Error|
|bias factor influenced by -hacking, conflict of interest, competitive publication motivations, etc.|
Now we can recreate the general table for all such examples above and derive their values:
|Total|| the number of relationships tested|
Starting with the top left cell which represents:
Similarly, for the top-middle cell:
So, for all true positives, the top-right cell:
in terms of Type I, II error and pre-study odds.
When is a Study More Likely to be True than False?
Some fields of study have inherently small or values
What Happens if we Introduce Bias?
negative study results become positive with
This can alter our outcome by in two cases:
Note that these truths/falsehoods have to be independent of the decision making otherwise they would impair judgement, disallowing us from applying the Product Rule
The Issue of Incorrect pre-publication
Research efforts do not occur in isolation. Several teams may be independently, competitive working on the same hypotheses over and over and over again without adjusting their -values.
This means that statistical power decreases as the experiments are repeated:
which could be ... a lot of probabilities
which is the negative results of all studies
Meaning that for each subsequent, competing trial, the likelihood of your own -value genuinely being sufficiently small decreases.