Suggestions on Setting Up an Efficacy Study

Are my students learning? It’s the question that every educator asks, wanting to know if their efforts have an impact on their students.

                                                                             

In recent years, educators have witnessed an instructional revolution focusing on using student data to provide targeted remediation and drive student results. But what does that actually look like? Pearson MyLabs and Mastering products keep detailed track of student results from homework and assessments, but what to do with this data?  Pearson’s Efficacy Research Team offers guidance on how to set up an efficacy study.

The first step requires setting a realistic, quantifiable goal, such as increasing student retention, boosting scores, or improving attendance.  One example might read, “I want to increase my final exam scores by ten percent.”  The teacher might try implementing a new learning intervention, like a MyLab or Mastering product, to achieve the increase in test scores.

Before classes start, create or redesign a course to optimize results and confirm that necessary resources are in place. Will the computer lab accommodate all the students? Consider the proper measurement tools to use in the evaluation process. For our example above one proper tool would be to administer last year’s final exam as it will be a like comparison to the current year. Another useful tool might be comparing placement tests to verify that students in prior years were at the same level as this year’s student body. If our study design wants to determine the MyLab or Mastering impact specifically, then all the other course elements should remain unchanged—same teacher, same book, etc.—allowing us to identify one causal element.

While working through the semester, it is a good idea to record breakthroughs with students.  Do students ask more intelligent questions in class than in previous years? Is there increased student engagement?  This type of qualitative evidence can help interpret the data at the end of the course.

Pearson can help analyze the data at the end of the course.  While Pearson can help determine if any quantifiable goals were reached, we cannot help with the more important educational interpretation of what that result means.  Ultimately, it is the teacher’s own evaluation of the data that is relevant in determining educational significance.  Looking at the analyzed data at the end of the semester along with qualitative data from course records allows one to highlight successes, reflect on areas of weakness, and redefine program goals. In this manner, course redesign is an ever continuous flow of improvement. 

How do you show the results of your redesign to colleagues? Do you collect hard data and share observations or do you prefer one type of information over the other?