One of the most important (and easiest) pieces of information you need to research when you are evaluating a program’s “research base” is how was the treatment implemented in the study? Did the participants use a supplemental program for 400 minutes a week? Did they have personal coaching? Was support provided from the program company in any way?
It’s pretty common for program’s to be evaluated under “ideal circumstances”. That is, the program is implemented exactly as its creator intended. I know a curriculum company that would cite growth for a curriculum that was studied in a school that used it as their core curriculum with additional teacher support to a school planning to use it as a supplement and not purchase any support. The differences in results were not a true “apples to apples” comparison.
It’s important to ask how the program was implemented when studied and decide if you are able to implement it that way. If you don’t follow the implementation exactly, it doesn’t mean your students won’t see success, but you should prepare yourself for the possibility they won’t be as large or pervasive as you expect or were told.
