The difference between 0.051 and 0.049 is 0.002 and that is the exact number which determines if the result of a scientific study is going to be accepted or rejected.To put it in context, p values of less than 5% are widely used as an ultimate decision making tool for any scientific research.But, how do we know there is statistically significant difference between p value of 0.049 and p value of 0.051? Let us try to understand p values better.
Goal of any scientific or statistical inferential study is to make a decision about a hypothesis (i.e. null hypothesis) which is either true or false.The decision is based on the observed values of a random sample to draw conclusions about an entire population.We design a test statistic based on the sample to make a decision about the hypothesis.Any test comes with a chance of error. The error associated with rejecting a hypothesis when it is actually true is known as type I error.Let’s assume, the probability of committing a type I error is some number, α , which is the level of significance any researcher sets for his study. An α of 0.05 indicates that we are willing to accept a 5% chance that we are wrong when we reject the null hypothesis.On the other hand, p-value tells us if the null hypothesis is correct, what is the probability of observing an effect at least as large as the one in our sample.
0.005 imply Now, let’s try to answer the following question – does a p value of 0.005 imply much more statistically significant result than a p value of 0.05? The answer is no.Null hypothesis can either be true or false.It cannot be a random variable.
When Fisher first declared the phrase ‘statistical significance’ using a p value in 1925, his original intention was to use p values merely as a tool and not as a tyrant to decide the fate of all scientific researches.The concept of ‘p values’ were conceived to be used as an indicator when a result warrants a further scrutiny.It doesn’t necessarily mean the effect is significant in a real world.Using such conclusive rules for justifying scientific claims can lead to erroneous decision making.
So what should be the right approach? Well, it is understandable that discarding this concept entirely is not desired but the standard practice should be reporting p values as absolute number and avoid making any conclusive remark based on that.In real world, data is quite noisy and one of the main cause behind uncertainty is data anomaly. So, we should also report p values along with an indication of the uncertainty in the statistical conclusion to interpret potential error.