The following is a guest post by Joseph Mudge, who published a paper with colleagues recently arguing for a different approach to setting α values. Joseph has written a summary of the argument below. Please chime in with your thoughts. Do you agree/disagree?
by Joseph F. Mudge,
Ph.D. candidate (biology), University of New Brunswick,
Saint John, N.B., Canada,
Ecologists are often charged with making binary decisions concerning ecological data. This causes problems because most patterns in nature exist as gradients. Null hypothesis significance tests have been a traditional tool for making binary decisions from gradient patterns in ecology, and they remain common in ecological research, despite many well known problems. One of the most obvious problems with null hypothesis significance testing in ecology is the dogmatic adherence to the arbitrary α level of 0.05 as the significance threshold for decision-making. Two consequences of always applying this arbitrary standard are (1) the decoupling of statistical significance (or lack thereof) and biological significance (or lack thereof) and (2) radical inconsistencies in statistical power to detect biologically relevant effects between studies (ranging from near 0% to near 100%). Although the α problem has been well discussed by ecologists over the last few decades, the use of arbitrary α levels has persisted in the ecological literature due to the continuing need for a statistical binary decision-making tool and because a more rational, yet still universally applicable, approach to setting α levels has not been available.
If a researcher can quantify 2 things that I believe should always be important considerations for any ecological research question; (1) the level of effect that would be considered biologically meaningful if it were to exist, and (2) the relative seriousness of Type I vs. Type II errors, it becomes possible to set an optimal, study-specific, α level for decision-making that minimizes either the combined probabilities or costs of both Type I errors and biologically relevant Type II errors. Although specifying a biologically meaningful critical effect size and the relative importance of Type I and Type II errors is not a trivial matter for many ecological questions, it should be noted that implicit and unexamined decisions about biologically meaningful effect sizes and the relative importance of Type I and II error are made when α is set at 0.05. You can’t avoid making these decisions, you can only ignore that you’ve made them. It seems ill-advised to set an arbitrary (albeit easier) decision-making threshold that fails to minimize either chances or costs of mistakes when consideration of important factors related to the research being undertaken can minimize the chances and/or costs of making mistakes.
Once biologically relevant effect sizes and relative importance of Type I and Type II errors have been identified for an ecological research question, determining the optimal α level is simple, as long as the researcher can calculate power for the null hypothesis significance test. The researcher only needs to calculate statistical power for the biologically relevant effect size at a variety of different α levels, and the optimal α level is that which converges on the smallest average of α and β (1-statistical power), with α and β weighted by their respective relative costs of Type I or Type II error. Weighting α and β equally with respect to their relative costs assumes that errors are equally undesirable regardless of whether they be Type I or Type II, and the result for this scenario is the optimal α level that minimizes the combined probabilities of Type I and Type II error.
The result of applying the optimal α approach in ecological research is that studies with high sample sizes end up with very small optimal α levels (while still having very high power to detect biologically relevant effects), reflecting the excellence of the study. In contrast, studies with low sample size end up with larger optimal α levels (in order to maintain some power to detect biologically relevant effect sizes), reflecting the lower quality of the study. Thus, optimal α can be an important and useful indicator of study quality.
I encourage ecologists to try using the optimal α approach to choose a statistical threshold for decision-making in their ecological research. I can think of no rationale for continuing to use an arbitrary threshold for decision-making in ecological research, other than that (1) it requires less thought, and (2) everybody else is using it.
Robinson, D.H., & Wainer, H. 2002. On the past and future of null hypothesis significance testing. Journal of Wildlife Management 66, 263-271. here
Mudge, J.F., Baker, L.F., Edge, C.B., & Houlahan, J.E. 2012. Setting an optimal α that minimizes errors in null hypothesis significance tests. PLoS ONE 7(2), e32734.doi:10.1371/journal.pone.0032734. here