What educators ask most often about effectiveness research
Over the past decade, we have had the opportunity to visit with literally thousands of educators across the country. Not surprisingly, we have found tremendous interest in effectiveness research and evidence-based educational practice. Among the many questions we are asked, some questions are by far the most common. Here are the most common questions we hear.
Why do I need a formal study; all of our customers tell us our product/service is effective?-
Positive comments and testimonials are useful, but this does not constitute "scientific evidence." A small set of client opinions is not considered "scientific evidence." Scientifically acceptable proof of effectiveness requires a sound research design accepted by the scientific community. School buyers and government funding agencies want to see a scientific study supporting the effectiveness of your product or service before buying or providing funds for purchase. Buyers of school products and services are savvier and are no longer willing to settle for products and services backed by unsubstantiated claims and testimonials. They are looking for hard evidence that will give them confidence in solutions before buying. Many communities, still strapped for cash based on the recent economy, are shopping "smarter," by limiting purchases to those with research-based efficacy support.
SEG can help you work through an effectiveness study design that makes sense for your product or service.
A strong efficacy study typically includes a scientific sample of users (treatment group) that are tested using reliable outcome measures both before using the product and after using the product. And, it is best if we compare the results of those users to a comparison group (control group) that is not using the product. Proper statistical techniques need to be applied to evaluate whether the results obtained are greater than would be expected by chance alone. Scientific effectiveness studies vary in scope (big-small, long-short, etc.) but tend to share these common features.
When should I begin planning for an efficacy study?-
Often we hear from educational providers that they are ready to conduct a study and wish to start as soon as possible. While we very much want to support everyone and admire their drive, careful planning and preparation is critical for effectiveness study success. We encourage product and service providers to begin planning about 6 months before executing a study, or at least 3 months before starting the study.
Planning and preparation include, first and foremost, designing the study. Second, we also encourage clients to begin preliminary identification and recruitment of the sample 3-6 months ahead of time. Third, those conducting an effectiveness study should leave plenty of time for the selection or identification of appropriate measures for the study. While planning involves much more than this, these three upfront activities will dramatically influence the success of the study.
Thinking about conducting an effectiveness study? SEG can help you begin planning to maximize your success.
What questions do you have? Let us know what questions you want answered. Contact hrickert@segmeasurement to share your feedback.
