| Various Editorial Policies
Regarding Statistical Significance Tests and Effect Sizes |
Return to Bruce's Home
Page. Educational and Psychological Measurement We will go further [than mere encouragement]. Authors reporting
statistical significance will be required to both report and interpret
effect sizes. However, these effect sizes may be of various forms, including
standardized differences, or uncorrected (e.g., r2,
R2, eta2) or corrected (e.g., adjusted
R2, omega2) variance-accounted-for statistics.
(Thompson, 1994, p. 845, emphasis in original) Thompson, B. (1994). Guidelines for authors. Educational and Psychological
Measurement, 54, 837-847. Journal of Agricultural Education Authors should report effect sizes in the manuscript and tables
when reporting statistical significance. (Kotrlick, 2000, inside cover) Kotrlick, J.W. (2000). Guidelines for authors. Journal of Agricultural
Education, 41(1), inside cover. Journal of Applied Psychology (APA) If an author decides not to present an effect size estimate
along with the outcome of a significance test, I will ask the author to provide
specific justification for why effect sizes are not reported. So far, I have not
heard a good argument against presenting effect sizes. Therefore, unless there
is a real impediment to doing so, you should routinely include effect size
information in the papers you submit. (Murphy, 1997, p. 4) Murphy, K.R. (1997). Editorial. Journal of Applied Psychology,
82, 3-5. Journal of Consulting and Clinical Psychology Evaluations of the outcomes of psychological treatments are
favorably enhanced when the published report includes not only statistical
significance and the required effect size but also a consideration of
clinical significance. (Kendall, 1997, p. 3, emphasis added) Kendall, P.C. (1997). Editorial. Journal of Consulting and Clinical
Psychology, 65, 3-5. Journal of Experimental Education Authors are required to report and interpret
magnitude-of-effect measures in conjunction with every p value that is
reported. (Heldref Foundation, 1997, pp. 95-96, emphasis added) Heldref Foundation. (1997). Guidelines for contributors. Journal of
Experimental Education, 65, 95-96. Journal of Learning Disabilities The APA Publication Manual cites the need for including
effect-size information in manuscripts utilizing quantitative data analysis
techniques... If authors do not include this information in submitted
manuscripts (and the manuscript is based on a quantitative research design), the
author(s) will be asked to provide this information should the manuscript be
recommended for publication or revision and publication. (Hresko, 2000, p. 214)
Hresko, W. (2000). Editorial policy. Journal of Learning Disabilities,
33, 214-215. Language Learning Language Learning, like many
journals that publish research using quantitative and statistical methods, is
increasingly influenced by the advantages of the reporting of effect sizes.
Submitting authors to this journal have to date been referred to the statement
in the Publication Manual of the American Psychological Association (4th
edition) which emphasizes that statistical significance p values are not
acceptable indices of effect because they depend on sample size and that "you
are [therefore] encouraged to provide effect size information." (APA, 1994, p.
18). Unfortunately, empirical studies of this and other journals (Wilkinson
& the American Psychological Association Task Force on Statistical
Inference, 1999) indicate that this encouragement has had negligible impact.
Ellis, N. (2000). Editorial. Language Learning,
50(3). The Professional Educator In all quantitative studies, effect sizes must be reported and
interpreted for significant and nonsignificant results. (volume 21(2), p.
54) Research in the Schools ...The Publication Manual of the American Psychological
Association (1994), the style guide required by Research in the Schools,
provides little guidance. The Manual discusses both statistical significance
(pp. 17-18) and effect size (p. 18), but only "encourages" (p. 18) authors to
provide effect size information. McLean, J.E., & Kaufman, A.S. (2000). Editorial:
Statistical significance testing and Research in the Schools. Research
in the Schools, 7(2).
4/30/00
The reporting of effect sizes is essential to
good research. It enables readers to evaluate the stability of results across
samples, operationalizations, designs, and analyses. It allows evaluation of the
practical relevance of the research outcomes. It provides the basis of power
analyses and meta-analyses needed in future research. This role of effect sizes
in meta-analysis is clearly illustrated in the article by Norris and Ortega
which follows this editorial statement. Submitting authors to Language Learning
are therefore required henceforth to provide a measure of effect size, at least
for the major statistical contrasts which they report.
More information on effect sizes and their reporting is given in the
guidelines of Wilkinson et al. (1999) which can also currently be accessed over
the internet at Web address: http://www.apa.org/journals/amp/amp548594.html.
This article cites several key reviews such as Rosenthal (1994), Cohen (1992),
and Kirk (1996) which summarize various measures of effect sizes used in
psychological and educational research. Consult these articles for information
on computing them.
Always present effect sizes
and their confidence intervals for primary outcomes. These effect sizes might be
of various forms. If the units of measurement are meaningful on a practical
level (e.g., reading rate, normed proficiency test scores), then unstandardized
measures (regression coefficient or mean difference) are appropriate. If not,
standardized differences (d) or uncorrected (e.g., r, R2, eta2) or corrected
(e.g., adjusted R2, omega2) variance-accounted-for-statistics should be
reported.
These effect sizes are required in
addition to the usual inferential statistical tests of significance, they do not
replace them. It is also appropriate in the textual argument of the results
section to place these effect sizes in their practical and theoretical context.
(Ellis, 2000)
Since the
[RITS] Special Issue [on statistical significance], we have also
"encouraged" authors to provide effect size information. In fact, we have
required authors to provide effect size information to accompany statistical
significance tests unless they could provide a compelling reason not to. Since
that time, we have had no author make a compelling case to omit effect size
information. Thus, we have decided to make this policy explicit and require that
authors accompany the reporting of statistical significance tests with effect
size information. Specifically, the following line has been added to the
Research in the Schools "Information for Authors" section: "All reporting of
statistical significance must include an estimate of effect size."
We are hopeful that this change will encourage
educational researchers to consider effect size and practical significance when
evaluating the results of a study. We are also encouraging our Editorial Board
members to consider effect size and the practical significance of a study in
their recommendations. In the end, we hope this change supports the movement
towards the reporting of more complete and accurate results of research studies.
Since this change in policy merely formalizes what we have been practicing for
at least two years, we do not expect that it will have an impact on the number
of manuscripts we receive, but we do hope it will have an impact on the quality
of the manuscripts... (McLean & Kaufman, 2000)