Using Pat Hutchings’s taxonomy of SoTL questions can make it easier to think about SoTL project design, particularly the most common kinds: projects that ask “what is?” or “what works?”
“What is?” Designs: Descriptive
“What is?” projects are thick descriptions, snapshots of learning dynamics, or other detailed analyses of what student learning looks like, or what’s happening during the moments of learning. The size and scope can range from small (e.g., a case study of one student, a thin slice of learning in a single class period) to large (many students [a “large n”], multiple activities occurring over time).
“What Works?” Designs: Explanatory
“What works?” projects assess the effectiveness of or change caused by an activity, tool, or other intervention. According to the scientific method, a traditional experiment would have two randomly assigned groups: one who receives the intervention and one serving as a control. In SoTL, however, this isn’t always possible. It can be challenging to access two sections of a single course, students are not randomly assigned to courses, and there are so many variables affecting the students’ learning that it’s difficult to design a “clean” experiment.
Some SoTL theorists have pointed out that it may not even be particularly desirable to aim for such “cleanness” in design. For instance, Catherine Lewis, Rebecca Perry, and Aki Murata (2006) explain in “How Should Research Contribute to Instructional Improvement?” as follows:
“The very qualities that suit an innovation to controlled trial may handicap it at the later stage of broad dissemination” by … emphasizing “fidelity to the original design that stifles continuing improvement”; whereas “the very characteristics that make an innovation…difficult to study in a well-controlled fashion” make it more applicable and usable by others. Some scholars are drawn to the “general proof” of controlled experimental design and fail to recognize the “local proof” of local and seemingly messy findings that result from less controlled research…. “In part, this is because we often choose the wrong endpoint for research: causal proof of an innovation’s effectiveness under controlled circumstances at initial sites, rather than ‘legs’ and effectiveness at subsequent sites of spread…. In short, it should be incumbent on researchers to justify their choice of research methods based on the endpoint—instructional improvement at sites of spread [dissemination and broader use]—not just the premature stopping-point, proof of innovation effectiveness under controlled conditions. The seeds of dissemination success or failure may well be planted at the outset of innovation.” (p. 8)
As a result, SoTL researchers attempting comparisons try to use groups that are as similar as possible to measure the effect of an activity as precisely as possible. If the groups are similar, the differences in outcomes are more likely to have arisen from the intervention, rather than from innate differences across groups, but the researchers are always explicit about the specific groups, context, and potential limitations.
*Adapted from Vanderbilt University Center for Teaching
For more information on this stage of the SoTL project, please see Vanderbilt University Center for Teaching’s guide on “Project Design: Structuring the Learning Activities and Data Collection.”
Next: Identifying Evidence >