Evidence of the effectiveness of an educational intervention is not a straight forward process The research needed can be expensive and time consuming. It can take quite a few years and many studies before convincing evidence is gathered and reported.
The first level of evidence is the case study. Basically, a single student (or a number of students) show improvement as a result of an intervention. Case studies often lead to testimonials where those who have experienced gains, make personal statements to this effect. However, case studies leave many questions unanswered. Maybe the improvement was a result of the personal qualities of the teacher and not the intervention itself. Maybe students could have made better gains with a different intervention. Maybe it was the additional teaching time that the students received rather than the intervention. However, this sort of evidence is relatively easy and cheap to obtain. They can be very influential, especially if the testimonial comes for a well respected teacher or are made by a large number of people across a variety of situations. Click here for case study and testimonial evidence for My Virtual Reading Coach.
The next level of evidence comes with the use of a control group. This is where gains made by the group receiving the intervention exceed those made by a similar group of students not receiving the intervention, the control group. While this is a stronger level of evidence than the case study, there are still a number of issues. Are the two groups really compatible? For example, one group could be a lot smarter than the other one or come from more supportive families so gains could be the result of other factors, not the intervention. Could gains be the result of the additional teacher time devoted to those receiving the intervention?
The next level of intervention is the use of randomly assigned groups. Students are randomly assigned to either the intervention or the control group , thus controlling for factors like intelligence or family background. The research design can also be strengthened by having the control group receive an intervention not related to the one being tested so both groups receive similar periods of additional teacher time. There is an ethical dilemma with this approach. If the intervention works, then those in the control group miss out even thought they have similar needs to those receiving the intervention. The solution is the swap the two groups after a set period of time so everyone receives the intervention. This increases the time (and cost) of the study. Click here for studies using control groups to evaluate the effectiveness of MVRC.
Then to be really effective, there has to be some long term benefit of the intervention and not just an immediate gain. Research may need to follow students for 3 or 4 years to establish this. Such longitudinal research is expensive and time consuming. Large numbers are needed as students often move and drop out of the study.
Finally, all research needs to be replicated. Much research on educational interventions is initially done by the developers of the intervention itself or by a company that has commercialised the intervention. Thus this research is open to criticism of bias – those involved had a vested interest in the success of the intervention.
Evidence for the effectiveness of MVRC comes from studies using all the approaches listed above, with the exception of longitudinal studies. Given the online version of this program was not published until 2016, this is not surprising.