New! Making sense of Ed tech product effectiveness research


A Practical, Affordable Research Strategy for Your Education Technology Product

What types of research evidence matters?

The Next Step:  Judging Research Relevance

Judging Research Quality

A Practical, Affordable Research Strategy for Your Education Technology Product

“We can’t afford that!”
“It’s too little, too late for us”

When ed tech product developers think of research, these are often the objections we hear.  In the real world, developers of education technology face a research dilemma:  they need research, to improve their product, to show educators how it works as designed when used as designed, and to build a convincing value proposition.  But some nasty realities which often are show stoppers:  First, almost no product can support a business case for a large-scale, seven-figure study of effectiveness.  Consequently, in most circumstances a federally-funded research grant is the only option for such a study.  Second, even a grant-based strategy for a large-scale effectiveness study isn’t something you can put in your business plan:  only a small percentage of research grant proposals are funded, and even then the study’s results will be published years after your product has moved on to its next release, so the study will be outdated before it is published.  To add insult to injury, some journals won’t even publish studies of commercial products, especially if the name of the product is included.

But there’s more to the story than the large-scale experiment:  that kind of study may not even be the most informative for you or educators who want to learn more about your product.

Educators and product developers need a research strategy that’s better, faster – and cheaper.  They need a strategy that answers questions including:

  • How does this product implement the current state of the art in learning science?
  • What’s working, and what’s not working, about this product’s current design, and what should we do about it?
  • How (and by whom, in what context) should this product be used for best effect?
  • What are the (many) benefits of using this product (not just better test scores)?
  • What are the cost-benefit and cost-effectiveness tradeoffs involved in implementing this product effectively in schools like ours, with curricula, teachers and students like these?

These are examples of the questions educators really need answers to, and the answers must be grounded in well-designed, credible research – regardless of who pays for it.  Ed Tech product developers need to develop timely, research-based answers to these questions, and communicate answers in ways educators can readily understand, believe and use.  Educators also need to learn how to interpret and use this kind of research – and collaborate in developing it.

In sum, the good news is that research methods are available to answer these questions in ways that are practical, affordable, and timely.  A sound research strategy should:

  • complement modern product development methodologies.
  • add value to products in ways that make business sense
  • be consistent with your marketing strategy. 
  • use different kinds of studies to answer different types of questions which teachers, administrators, and product developers have. 
  • grows to scale as the product goes to scale. 

Making Sense of Research on Education
Technology Products

Before they adopt education technology products, educators want trustworthy information on how well the products are likely to work for them.  Producers of those products would like to provide that information, in a form that the educators can trust and can use.  Research on the technologies is currency of that information exchange.  And like a currency, research comes in big denominations (which you’re likely to pay a lot of attention to, but you don’t see very often), and small denominations (which are much more common, but attract less attention).  In education technology, the full spectrum of research types and sizes is useful – but you have to do the research thoughtfully, and use it well.  Furthermore, it takes years – often more than a decade – for the full research story to be completed.  That’s a real problem for education technology:  educators can’t wait for solutions to today’s pressing needs, and technology products are likely to be obsolete long before it’s possible to develop the strongest forms of evidence of effectiveness.


What types of research evidence matters?

To solve the research problem for ed tech, the US Ed Office of Educational Technology recent commissioned Dr. Barbara Means of SRI International’s Center on Technology and Learning to convene a panel of leading technology researchers and make recommendations on how to build the research base for an education technology product.  The recommendations are in this report.  Particularly interesting is the report’s recommendation for a progression of research evidence types, which builds from the initial design of a product, through its use and refinement, and on to support its scalability.  This approach incorporates these types of evidence:

  • Learning science research used to guide design of the product
  • Co-design by collaborative teams of educators, developers, researchers
  • Data mined from early trials of prototypes
  • Rapid A/B experiments done within the learning system to improve specific design decisions
  • Collaborative design-based implementation research performed jointly by practitioners and researchers
  • Integral technology-supported assessments of learning outcomes using evidence-centered designs
  • Small-scale experiments using learning outcome measures external to the product
  • Collaborative research using linked data from learning systems, education records, and social service agencies.
  • User ratings
  • Larger scale, multisite experiments on effectiveness
  • Research consortia that combine and analyze data from multiple studies

For educators and for product developers, all of these types of research evidence are useful.  They tell you different things about the product and how to use it well, and about its effectiveness.  But here’s the key point for educators:

  • New products usually have only the early forms of research evidence: typically, #1-6 and #9.  But these are the types of research that tell you the most about how a product should be used effectively.
  • Mature products, typically with years of use, may be capable of doing #7 in a small number of studies. These tell you some things about effectiveness, but usually not about how to use a product effectively.
  • The strongest forms of evidence on effectiveness, #8, 10, and 11, come later – if at all.  They are rarely available except when studies are combined across many successive versions of a large-scale widely used product, and often after a decade or more of use.


The Next Step:  Judging Research Relevance

If you could wave a magic wand, you would like every kind of research study done for every product you want, and done in your schools so you know it applies to you.  That won’t happen!  So, you have to be an informed consumer of research, by looking critically at research – whether it comes from the product’s producer or a third party; whether it’s published by the producer or by a journal.  Here are four questions you can ask of any research study you examine:

Does the study answer a question that’s
important to me?

Typical examples of study questions include:

  • What is the proficiency level at the end of use?
  • What is the before-and-after gain in learning?
  • What are the benefits to administrators, teachers, students and/or parents of using this product?
  • How is this product used effectively?
  • How to teachers and students like using this product?
  • How does this product save time or money?

How meaningful to me are the measures used for each benefit claimed?

Things to think about:

  • A state test pass rate or score may or may not be sensitive enough to measure what the product is designed to teach or facilitate.
  • Another state’s, district’s or teacher’s tests may only approximate the content of the tests you use, or those built into the product.  The closer the approximation, the more meaningful the study is to you. 
  • Before-and-after gains are meaningful only if both measurements are done with the same test, or tests that are designed to be compared.
  • Data on learning which the product itself generates are usually most sensitive indicators of product effectiveness.
  • Measures other than conventional tests are needed for benefits other than cognitive learning.

How well, and how long, was the product used, in how many different settings?

Things to think about:

  • Does the study tell you if the product was used as designed? (no product will work if you don’t use it properly)
  • Do the results hold up after long-term proper use?
  • Do the results hold up when many teachers in many schools use the product properly?
  • How did the product perform in contexts like mine, with students and teachers like mine?

If the study claims achievement gains, then ask, “compared to what?”

  • If there is no comparison group, you can’t tell if the product improves on whatever you’re doing now (“business as usual”)
  • The comparison is meaningful only if the study shows both groups were similar at the start of the study, or if statistical adjustments are made to compensate for those differences.


Judging Research Quality

Of course, it’s only worthwhile to pay attention to research which is of high quality.  Unfortunately, judging the quality of research is quite involved and requires a number of technical skills.  Also unfortunately, there is no fully satisfactory “consumer reports” rating agency for reviewing all of the 11 kinds of research listed in the first blog.  For example, the What Works Clearinghouse only accepts #10 and #11.  Academic journals sometimes accept #7 – but only a few publish research on specific named technology products.  You and your staff certainly don’t have the time (or maybe the expertise) to review the quality of all the research you need. 

What you need to know goes beyond what’s typically in research design textbooks.  You also need to know about:

  • The independence of the researcher from the design team.  This is usually not done (and often not desirable) for studies of types #1-#5, but it is important for types #6-#11.
  • Who had final editorial control over the research report.  Especially for #6-#11, the right answer is: the independent researcher.
  • What the producer’s policy is on what studies are released/published.  Especially for #6-#11, you need to know if all studies are released, regardless of the conclusion, as long as they were technically sound.

These factors are much more important than who paid for the study.  The reality is that only the producer of a product typically has funding for any kind of research – whether the funding is from a development grant or from corporate revenue.

There are many other technical design issues to consider in judging study quality, and many issues in research ethics.  Some of the most important guidelines for these issues have been published by:

  • American Educational Research Association
  • American Evaluation Association
  • Software and Information Industry Association
  • What Works Clearinghouse

Ask the product producer what their policies are on conduct and release/publication of research, and ask them how well the study conforms to the applicable professional and technical guidelines for studies from these sources.


Rob Foshay, PhD, CPT