What is statistical significance?

Statistical significance refers to how likely a difference or relationship is due to something other than chance.  A statistically significant difference or relationship *is* significantly different from chance, and in this case, the null hypothesis is rejected. 

What is the significance level and how is the significance level set? 

The significance level, or alpha level, is predetermined in advance before statistical tests are run. Generally, the alpha level is set at .05 (that is, there is 95% or greater probability that the results are not due to chance). However, in cases where making a Type I error would have more significant repercussions, a smaller alpha level is used.

What is power, and how is it related to the significance level? 

Power is the probability of rejecting the null hypothesis when it should be rejected. It is generally accepted that power should be .8 or greater. The greater the sample size, the greater the power, but the sample size should not be too large also. 

To calculate power, you need to know the type of test you will use, the alpha or significance level, the effect size, and the sample size you are planning on using. Statistical programs or online calculators are easy methods to calculate sample size. If the power is under .8, the sample size should be increased. 

What is effect size and how is it related to statistical significance?

If there is a statistically significant difference between samples, or a significant relationship between two variables, that does not necessarily mean that the difference or relationship is meaningful, but it only means you are confident there is a difference or relationship that is not due to chance. 

The calculation of the effect size varies by tests. Generally, you take the difference between the mean of the experimental group and the mean of the control group, and divide by the standard deviation for Cohen’s d. Pearson’s r is a relationship measure of effect size and is the mean cross-product of the two variables’ z-scores.  .  

The following table contains values that can help you determine effect size:

How do you determine whether a difference or relationship is significant?

Different readings incorporate different tests depending on the research question, variables used, and whether the data meet statistical assumptions. These therefore have different ways of showing whether the relationship or difference is statistically significant.

If the test has a p value, if the p value is lower than the alpha level, the test is statistically significant. In many current studies, the p value is given. 

Pearson’s r correlation coefficient is used to determine significance for correlations. If you have many variables, a correlation matrix is used. The correlation coefficients should be signaled as significant, but the values of Pearson’s r can also be used to determine significance.