Evaluating the Effect of Coaching on SAT Scores: A Meta-Analysis

DerSimonian and Laird present a quantitative analysis of published results on the effect of coaching programs on SAT scores. Their analysis differs from previous methods of combining evidence from different sources by separating out the within-study sampling error from the variation in coaching effectiveness. They explicitly model the methodology used in each study in order to analyze the variation in study results. They find that studies which compare the gains achieved by coached students to national norms yield estimated "coaching effects, " which are four to five times greater than corresponding effects estimated for matched or randomized evaluations. In addition, the matched or randomized evaluations show a much greater degree of consistency in their results than do less well-controlled evaluations. The authors conclude that the data do support a positive effect of coaching on SAT scores, but that the size of the coaching effect estimated from the matched or randomized studies (10 points) seems too small to be practically important.