Various Editorial Policies Regarding
Statistical Significance Tests and Effect Sizes

Return to Bruce's Home Page.
4/30/00

Educational and Psychological Measurement

 

We will go further [than mere encouragement]. Authors reporting statistical significance will be required to both report and interpret effect sizes. However, these effect sizes may be of various forms, including standardized differences, or uncorrected (e.g., r2, R2, eta2) or corrected (e.g., adjusted R2, omega2) variance-accounted-for statistics. (Thompson, 1994, p. 845, emphasis in original)

 

Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.

 

 

 

Journal of Agricultural Education

 

Authors should report effect sizes in the manuscript and tables when reporting statistical significance. (Kotrlick, 2000, inside cover)

 

Kotrlick, J.W. (2000). Guidelines for authors. Journal of Agricultural Education, 41(1), inside cover.

 

 

 


Journal of Applied Psychology (APA)

 

If an author decides not to present an effect size estimate along with the outcome of a significance test, I will ask the author to provide specific justification for why effect sizes are not reported. So far, I have not heard a good argument against presenting effect sizes. Therefore, unless there is a real impediment to doing so, you should routinely include effect size information in the papers you submit. (Murphy, 1997, p. 4)

 

Murphy, K.R. (1997). Editorial. Journal of Applied Psychology, 82, 3-5.

 

 

 

 

 

Journal of Consulting and Clinical Psychology 

 

Evaluations of the outcomes of psychological treatments are favorably enhanced when the published report includes not only statistical significance and the required effect size but also a consideration of clinical significance. (Kendall, 1997, p. 3, emphasis added)

 

Kendall, P.C. (1997). Editorial. Journal of Consulting and Clinical Psychology, 65, 3-5.

 

 

 

 

 

 

Journal of Experimental Education 

 

Authors are required to report and interpret magnitude-of-effect measures in conjunction with every p value that is reported. (Heldref Foundation, 1997, pp. 95-96, emphasis added)

 

Heldref Foundation. (1997). Guidelines for contributors. Journal of Experimental Education, 65, 95-96.

 

 

 

Journal of Learning Disabilities

 

The APA Publication Manual cites the need for including effect-size information in manuscripts utilizing quantitative data analysis techniques... If authors do not include this information in submitted manuscripts (and the manuscript is based on a quantitative research design), the author(s) will be asked to provide this information should the manuscript be recommended for publication or revision and publication. (Hresko, 2000, p. 214)

 

Hresko, W. (2000). Editorial policy. Journal of Learning Disabilities, 33, 214-215.

 

 

 

Language Learning

 

        Language Learning, like many journals that publish research using quantitative and statistical methods, is increasingly influenced by the advantages of the reporting of effect sizes. Submitting authors to this journal have to date been referred to the statement in the Publication Manual of the American Psychological Association (4th edition) which emphasizes that statistical significance p values are not acceptable indices of effect because they depend on sample size and that "you are [therefore] encouraged to provide effect size information." (APA, 1994, p. 18). Unfortunately, empirical studies of this and other journals (Wilkinson & the American Psychological Association Task Force on Statistical Inference, 1999) indicate that this encouragement has had negligible impact.
        The reporting of effect sizes is essential to good research. It enables readers to evaluate the stability of results across samples, operationalizations, designs, and analyses. It allows evaluation of the practical relevance of the research outcomes. It provides the basis of power analyses and meta-analyses needed in future research. This role of effect sizes in meta-analysis is clearly illustrated in the article by Norris and Ortega which follows this editorial statement. Submitting authors to Language Learning are therefore required henceforth to provide a measure of effect size, at least for the major statistical contrasts which they report.
        More information on effect sizes and their reporting is given in the guidelines of Wilkinson et al. (1999) which can also currently be accessed over the internet at Web address: http://www.apa.org/journals/amp/amp548594.html. This article cites several key reviews such as Rosenthal (1994), Cohen (1992), and Kirk (1996) which summarize various measures of effect sizes used in psychological and educational research. Consult these articles for information on computing them.
        Always present effect sizes and their confidence intervals for primary outcomes. These effect sizes might be of various forms. If the units of measurement are meaningful on a practical level (e.g., reading rate, normed proficiency test scores), then unstandardized measures (regression coefficient or mean difference) are appropriate. If not, standardized differences (d) or uncorrected (e.g., r, R2, eta2) or corrected (e.g., adjusted R2, omega2) variance-accounted-for-statistics should be reported.
        These effect sizes are required in addition to the usual inferential statistical tests of significance, they do not replace them. It is also appropriate in the textual argument of the results section to place these effect sizes in their practical and theoretical context. (Ellis, 2000)

 

Ellis, N. (2000). Editorial. Language Learning, 50(3).

 

 

 

The Professional Educator

 

In all quantitative studies, effect sizes must be reported and interpreted for significant and nonsignificant results. (volume 21(2), p. 54)

 

 

 


Research in the Schools

 

...The Publication Manual of the American Psychological Association (1994), the style guide required by Research in the Schools, provides little guidance. The Manual discusses both statistical significance (pp. 17-18) and effect size (p. 18), but only "encourages" (p. 18) authors to provide effect size information.
        Since the [RITS] Special Issue [on statistical significance], we have also "encouraged" authors to provide effect size information. In fact, we have required authors to provide effect size information to accompany statistical significance tests unless they could provide a compelling reason not to. Since that time, we have had no author make a compelling case to omit effect size information. Thus, we have decided to make this policy explicit and require that authors accompany the reporting of statistical significance tests with effect size information. Specifically, the following line has been added to the Research in the Schools "Information for Authors" section: "All reporting of statistical significance must include an estimate of effect size."
        We are hopeful that this change will encourage educational researchers to consider effect size and practical significance when evaluating the results of a study. We are also encouraging our Editorial Board members to consider effect size and the practical significance of a study in their recommendations. In the end, we hope this change supports the movement towards the reporting of more complete and accurate results of research studies. Since this change in policy merely formalizes what we have been practicing for at least two years, we do not expect that it will have an impact on the number of manuscripts we receive, but we do hope it will have an impact on the quality of the manuscripts... (McLean & Kaufman, 2000)

 

McLean, J.E., & Kaufman, A.S. (2000). Editorial: Statistical significance testing and Research in the Schools. Research in the Schools, 7(2).