Frequently Asked Questions

Assessment
What is PARCC and How is it Related to the Common Core Assessments?

 PARCC is the Assessment of Readiness for College and Careers consortium of US states collaborating on a unified set of K-12 assessments in English and math to gauge readiness for college and careers.  With a grant of $186 million from the Federal Government, the PARCC assessment system will measure the full range of Common Core State Standards  For the first time, states will share a common platform and standards.  PARCC assessments will be available to test students in the 2014 - 2015 school year.

SEG can help you align your program with the Common Core Assessments and ensure that they reflect the design of the new assessments.  Contact us today and let us help you improve your product with effective assessment.

What is the Standard Error of an Assessment?
Any student’s score on an assessment (“observed score”) is comprised of the student’s “true score” combined with measurement error.  The standard error of measurement (SEM) is the square root of the error variance of an assessment.  SEM is used to define the amount of error or “noise” around a score on the assessment.   This is closely related to the reliability of an assessment.  As reliability increases, the standard error of measurement decreases.  The lower the standard error of measurement, the closer the student’s observed score is to their true score on that assessment.
Does My Educational Product Need An Assessment Component?

Many districts and schools are looking for products that provide both instruction and assessment.  Tools need to be provided to assist in monitoring students' progress over time.  If your instructional program does not have an assessment component, we can work with you to determine an assessment design that would be most appropriate and effective for your product.  We can develop the assessments for you, help you field test (pilot) the assessments, finalize the items and forms, calibrate and scale the tests, and help you design reports, and help you plan for the integration and utilization of the assessment results into the program.

What Types Of Assessments Do You Provide?

We provide test development services for K-12, higher education, and adult learners in all subject areas.  We also provide test development services for certification and licensure products.  Whether you need a few new items written or a complete assessment program, we are able to help you.

General
What is an RTI Program?

 RTI is Response to Intervention.  RTI programs provide early, systematic assistance to children who are having difficulty learning. More and more schools are implementing Response to Intervention programs to better address the individualized needs of students.  In part, these programs offer a rapid cycle of instruction, measurement of student performance based on the instruction, and re-tailoring of instruction to address the needs identified.

RTI seeks to prevent academic failure through early intervention, frequent progress measurement, and increasingly intensive research-based instructional interventions for children who continue to have difficulty. This active cycle of instruction and assessment calls for frequent assessment that can be delivered and acted upon quickly. We welcome the opportunity to help you meet your assessment challenges.

What is Academic ROI?

This is a measure of the amount of money made (Return) for a given amount of money invested (Investment). 

Academic ROI is a translation of this concept into the education world as a way to measure the amount of academic outcome achieved for a given amount of investment made. 

The Academic Return on Investment (Academic ROI) can be measured as the amount of learning or other educational outcome achieved per dollar spent on a school product or service; in the true vernacular, "Bang for the Buck."

For example, let's assume that a school is considering purchase of a particular supplemental product and we want to compute the Academic ROI.  Assuming that a sound research study was available that listed the amount of gains achieved on the state test we can compare the cost per student to the gains achieved to determine the ROI. If the study reported that students using the product in a one year study gained 10 points and the cost per student of this product is $20, we can calculate the Academic ROI as 10/20, or .5.  This means that for each dollar spent, we can expect to see, on average a 1/2 point  increase in student scores.


If we use a an indicator of outcomes that is consistent across studies, such as the field-standard Effect Size, we can use this method to compare several competitive products.  The Academic ROI for each can be determined and we can consider this information along with other factors to make more informed purchasing decisions.   

Academic ROI is likely to take on a growing role in school decision making as limited resources continue to be a problem and as school administrators are faced with an ever increasing set of purchasing options from which to choose. Educational publishers, technology providers and service providers would be wise to provide sound effectiveness research to support buying decisions.

Where is SEG Measurement Based?

Our headquarters are in New Hope, PA, in Bucks County, along the scenic Delaware River bordering New Jersey.  New Hope is located on the east coast of the US, about halfway between New York City and Philadelphia, making it a local trip to New York City.

Research
Why Is a Research Study Necessary?
School buyers are demanding proven solutions; they want to see evidence that your product actually increases student achievement. Moreover, many government funding programs are demanding scientifically-based research to support funded products and services. A research study is an effective solution to enhance product sales and meet federal funding program requirements, boosting your sales and marketing efforts by proving that your product can increase student achievement.
If you are launching a new educational product, a research study can give insight on what works and what doesn't, so that you may quickly revise your product prior to further development and public release, as well as stating proven effectiveness right from the start.
 
 
While there is not full agreement on what constitutes scientifically based research, it is widely recognized that the research must be based on experimental and/or quasi-experimental  research that includes both a treatment group (users of the product) and a control group (those not using the product) with a comparison of academic growth over time (pre and post test).  The research should be conducted by an organization other than the provider of the product to provide greater objectivity.  Ideally, multiple studies should be available to support efficacy and the results should ideally be made available in a peer reviewed publication or at a peer-review based conference.
What Is Included In the Research Study Design?

When working with SEG Measurement on a research initiative, we will help you with the research design.  The first thing we need to do is determine the specific research question(s) that we want the research to answer.  Working from the research goals and questions, we can develop a research design that will answer the research question(s).  It is important to consider time, financial, and other logistical constraints when planning the research design.  Every project is unique and as such each research design and plan will be unique.  We will work with you through the planning process and will develop a research design that will meet your needs.  The research design will document the research questions and goals, the data sampling plan, data collection process, plans for data analysis, and plans for reporting and communication.      

Quasi-Experimental VS. Experimental Study Design, What is the Difference?
With an experimental research study, the participants in both the treatment (product users) and control (product non-users) groups are randomly assigned. Quasi-experimental research designs do not randomly assign participants to treatment or control groups for comparison. Quasi-experimental designs are often used in educational research since students and teachers are not typically able to be randomly assigned to districts, schools, or classrooms for research purposes.
What Is Included In a White Paper? How Does It Differ From a Research Study?
Basic White Paper Structure and Contents
·       Up to 20 pages in length (including all components, cover and references)
·       Introduction (overview of product)
·       Description of the product
·       Description of the theoretical and conceptual basis for the product (the thinking behind why the product works)
·       Discussion of how the product addresses potential customer needs and solves customer problems
·       Review and analyze relevant literature organized into 5-7 major categories supporting the theoretical basis for the product (e.g., differentiated instruction, language/literacy, social-emotional development, assessment)
·       References
The White Paper is usually the first step in a process to build a scientific base for the product, to be followed by a full research study.  
When Is a Study Considered to be Outdated?
A research study is considered to be outdated when it is over three years old due to market/economic and consumer behavior variations, demographic changes, and alterations to the product.
What is Statistical Power and Why is it Important?

Statistical Power is the probability that a statistical test will detect differences when they truly exist.  Think of Statistical Power as having the statistical "muscle" to be able to detect differences between the groups you are studying, or making sure you do not "miss" finding differences.     

In many efficacy studies we are looking at the differences between a treatment group (e.g., using a product) and a control group (e.g., not using a product).  If our treatment is truly different than the control we want to make sure that our statistical test will detect that difference.  This is where Statistical Power comes in.  We want to make sure that we have enough Statistical Power to ensure that we find the differences if they are there.

Statistical Power is determined by several factors, most importantly the size of the statistical significance selected, the size of the effect (amount of difference) we are expecting and the sample size.  Since we typically use significance levels of  .05 or .01 and we do not know the size of the effect in advance, we are often left with having to make decisions about sample size when planning a study to achieve sufficient Statistical Power. In short, we need to make sure that we have enough study participants to statistically detect the statistical differences observed. 

Estimating Statistical Power and determining the appropriate sample size can get a bit tricky, and the mathematical details are beyond the scope of this article.  But, SEG Measurement can work with you to navigate this and other study design decisions at the outset of your study

Can You Help Me Create and Administer a Survey?

Yes, we provide survey development, creation, administration, and reporting services.  Whether you would like to use an online survey tool or send a survey in the mail, we can help you make your survey a success.

What is ANCOVA?
ANCOVA stands for analysis of Covariance", a general linear model with a continuous outcome variable (quantitative, scaled) and two or more predictor variables where at least one is continuous (quantitative, scaled), and at least one is categorical (nominal, non-scaled).  The Analysis of Covariance is often used when comparing groups in a quasi-experimental research study.  ANCOVA allows for the comparison of groups while controlling for one or more factors (covariate).  For example, we can compare post-test scores between the treatment and control group while controlling for any differences in pre-test scores. 
Technical
What Is a White Paper?
Originally, the term white paper referred to an official government report, but has taken on broader meaning as the approach has been adopted by non-governmental organizations.  A white paper is typically used to present and argue a specific position or to propose a solution to a problem. They have become popular marketing tools for organizations, particularly with the growth of the Internet.  Organizations use white papers to sell their products or solutions that address customer needs.  The hope is that the presentation of their solution will positively influence the decision-making of potential customers.
 
The audience for a white paper for most organizations is potential buyers of their products that seek solutions to their problems or needs. For a white paper to be effective it must clearly address the potential customers need(s) and offer a solution to their problems.