Teaching and Learning Data Analysis and Uncertainty Concepts

The Role of Design in Educational Research

The research design is often alluded to or discussed briefly without any regard to the implications for types and strengths of the assertions that can be made from any resultant statistical analysis of the data collected. Some refer to research design in terms of analyses such as randomized block, one-way, two-way, repeated measures ANOVA, or fully nested. Discussing the design in these terms creates a misconception that the analytic method governs the types and strengths of the assertions that can be made but this approach does provide a road map for making decisions about what a research design should have looked like. Regardless of how data are collected (Design) any of the previously mentioned analyses can be used although with spurious results. Just like reliability is not a characteristic of a test the analysis does not govern the generalizability of the findings.

So what is meant by Design in this course? When I refer to designs I will refer to either the bold-capitalized Design or the lower case often plural- -designs. Design is used to refer to participant selection, assignment method to control or treatment groups, and conditions for managing validity threats and designs to refer to analytic methods or just methods-often singular, that in my belief, result from your Design choices.

It is important to understand how the Design of a research study influences generalizability, strength of associations, and causal relationships. While an Experimental Design provides the most generalizability and allows causal inferences, it is often compromised in education because of validity threats. Quasi-experimental designs are much more common but unfortunately often grossly over or under-estimate the effects. So what can you do? There are many useful and practical suggestions for improvements to what I consider the lowest and most useless level of quasi-experimental Design, the convenience sample without treatment or control groups. This course will help you establish a personal hierarchy of quasi-experimental Designs from ones to be considered suspect, to ones that have shown merit for closely approximating experimental design effects.

The Research Methods are an Implication of Research Design

The research methods can be as simple a descriptives and effect size or as complicated as hierarchical linear modeling, interrupted time series, or a newer idea in education research- - regression discontinuity. These analytic choices should be matched to your research Design and your belief in either a univariate or multivariate reality. It is very important to remember that reliability or internal consistency attenuate analytic results. So it is important to examine the reliability of the data in hand before invoking an analytic method. Now non of this examines the importance of normality of the data in hand and how shades of normality can drastically influence the obtained effects.

Another category of analytic choices is graphical. IT is not uncommon to encounter scatter plots, line graphs, line plots, or box plots but what is emerging as more useful graphical technique is confidence intervals that convey more information in an easily understood and compact presentation.