Every district leader hears the same question: how do we know our edtech is helping students?
An analysis like that deserves scrutiny. Here is how it works.
The Short Version
Insights compares how much students used an edtech tool with how much their scores grew between two assessments. It accounts for where each student started, their school, and student characteristics. It reports a result only when the data and the model pass a set of checks.
Results are associations, not causal. They help you ask better questions.
What Insights Measures
Insights then estimates how score gains differ at different levels of usage. That relationship may not be a straight line. A student's hundredth hour may not relate to scores the way the tenth does. The model lets the relationship vary at different levels of usage, so it can show where additional usage starts to add less.
The Checks Every Analysis Must Pass
Insights checks the data before it analyzes a tool.
If a tool misses a requirement, Insights shows which one.
Insights then tests the model against students it has not seen. If the model is not accurate enough, Insights reports no result.
Passing these checks means the analysis can run. It does not guarantee a clear result.
How to Read a Result
A six-point gain can be large on one assessment and small on another. Insights reports each result as an effect size, which puts results on one common scale.
To calculate it, Insights compares two estimates for a typical student: the score at a given level of usage and the score with no usage. It divides the difference by how much scores vary within the student's grade.
The comparison to no usage matters. Students grow over time with or without a given tool, so comparing against no usage separates the gain associated with usage from ordinary growth.
Insights labels each effect small, medium, or large, using benchmarks adapted from education research (Kraft, 2020). Those benchmarks were built for studies designed to show cause. Insights results are associations, so treat the labels as a way to prioritize where to look, not as proof of what a tool caused. When the data cannot rule out no gain at all, Insights does not assign an effect size label.
Each report also charts estimated gains at different levels of usage. For example, more usage might be associated with larger gains, but only up to a point. Where the evidence supports it, the report shows the weekly usage level where the estimated gain peaks. Treat it as a guide, not a required amount.
Results by Student Group
Insights lets a group's result differ from the district's to the extent the evidence supports it. When a group is small, or its students all used the tool about the same amount, Insights pulls the group's result toward the district result. A group result that matches the district can mean two things: the group follows the same pattern, or there is not enough evidence to show a difference.
Group size sets what is shown.
Treat a group result as a prompt to look closer at access and implementation. A single striking result is not a conclusion until it holds up over time or across schools.
What the Results Can and Cannot Tell You
Insights describes how score gains varied with usage among your own students during the period you analyzed.
Results are well suited to:
- Seeing how gains changed as usage increased across the district
- Seeing where additional usage starts to add less
- Comparing student groups, with caution, especially small ones
- Raising questions worth a closer look
Results should not be used to:
- Predict how an individual student will do
- Conclude that a tool caused gains, or a lack of gains
- Set a required amount of usage
- Evaluate teachers or students
A small effect or a missing label is not a verdict on a tool. For example, a program adopted in the fall may have no label in the spring because too few students used it enough to produce an estimate. Look first at access, rollout, and how the tool is being used.
For Research and Assessment Teams
The details your team will want to check:
- Usage measures. Active hours, total sessions, sessions with at least five active minutes, and days with a session.
- Model. ElasticNet regression, which applies a penalty that shrinks coefficients and can drop predictors that add no predictive value. Usage hours enter as a cubic spline with three interior knots, at the 25th, 50th, and 75th percentiles of usage.
- Model validation. Five-fold cross-validation, with accuracy checked on students held out from model fitting.
- Groups. Group models estimate how much a group's gain per hour of usage differs from the district's, and where along the usage range that gain builds, with shrinkage toward the district estimate.
- Effect size. The estimated score at a usage level, minus the estimated score at zero usage, divided by the standard deviation of post-test scores for the student's grade.
Built to Start Better Conversations
Insights does not hand your district a verdict. It gives your team local evidence of how usage relates to growth, and a clear place to start the next renewal, budget, or board conversation.
Reference: Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253.
For a plain-language introduction, or to request a demo or quote, visit classlink.com/insights.