The Real Reason Some AI Products Get Better and Others Just Get Older

Imagine this. Two edtech companies obtain licenses for the same AI model simultaneously. Within a year, one moves ahead of the other, and it’s clear that the outcome has little to do with the model itself.

So, what made the difference? For the last few years, a capable AI model felt like the finish line. Ship something powered by a strong model, and the differentiation would take care of itself. However, with models and open-source alternatives converging so closely in performance, the model no longer offers a competitive edge. The differentiating factor is something less visible but harder to copy.

This discussion formed the crux of my conversation with Dakshay Mehta on a recent episode of Tech in EdTech.

In this blog, we unpack what that harder-to-copy advantage is, why it has become more valuable than the model sitting underneath most edtech products, and what Dakshay had to say about building it deliberately rather than stumbling into it. Along the way, we will also get into where this approach can go wrong if a company leans on it without the right judgment guiding it.

Why AI Models No Longer Differentiate Products

Stanford’s 2026 AI Index found that the performance gap between the best frontier AI models and the best open-source alternatives shrank from about 8 percent in 2025 down to roughly 1.7 percent in 2026. Around the same time, newer open-source models started shipping at a fraction of the cost of comparable frontier systems. The model your product is built on no longer stands as a convincing differentiator, because competitors likely have access to something just as good.

A recent enterprise AI report from a16z found that 81 percent of enterprises now run three or more model families at once, up from 68 percent a year earlier. Companies are no longer committed to a single model.

The Advantage Your Competitors Cannot Copy

So what is the biggest differentiator? Data is the biggest moat in any industry right now. While products can be built on a good model, what sets one apart is what a company does with the conversations users are already having with it.

Dakshay described it as recursive learning, where every time a learner asks a question, gets confused, or works through a concept inside a product, that’s new information that didn’t exist before. This is precious data, and it needs to be evaluated to understand what the system got right and what it missed. The results then need to be fed back into the product.

The answer was always in front of us, and the mechanics of it aren’t even that complicated.

Why This Matters More in Education than Anywhere Else

Now that we have identified data as the hero, it’s time to get into the specifics of what education leaders can lean on to get the most out of their data.

A genuine data advantage tends to come from three layers working together:

• Unique interactions competitors can’t access
• Historical data that helps improve decisions over time
• Continuous feedback that makes the system sharper with every use

We’re looking at mapping everything a learner does with the product on a daily basis, such as asking the AI tutor questions, getting stuck on a particular concept, or abandoning a lesson halfway. This is data unique to every edtech company.

Over the years, I’ve watched my fair share of edtech companies lose valuable data because they didn’t have the discipline to use what their learners were telling them. It’s an operations gap, not a technological gap.

Not Every Struggle Needs to Be Fixed

There is, however, an uncomfortable question that recursive learning doesn’t answer on its own. If a product keeps reshaping itself around what learners do, who decides what “better” is? If a learner moves away from a difficult but important concept, that doesn’t mean the concept needs to be made easier. Some concepts are meant to be challenging, and struggle is part of the learning process.

If every drop-off point gets eliminated, it risks removing the challenges that help learners grow. This brings us to the importance of having guardrails and human judgment in place. During our conversation, Dakshay explained how we need to ask the AI why it’s making changes instead of simply accepting its suggestions for better clicks and completion rates. Data gives us a choice, and we need to know when to make changes and when not to.

Turning Insight into Action

If you’re leading product, content, or strategy at an education company today, here’s what I’d take from this conversation:

• Stop asking which model you’re using.
• Start asking what your learners are already telling you through every interaction.
• Build the discipline to capture and act on those insights instead of trying to bolt it on later.
• Treat this as a governance challenge, not just a technology one.

I left my conversation with Dakshay thinking about something much less flashy but far more important. The companies that come out ahead won’t be the ones with the best AI models. They’ll be the ones that learn the most from their learners.

Building Products That Learn Responsibly

The challenge isn’t collecting more data but knowing which signals matter, how to interpret them responsibly, and how to turn them into better learning experiences. It’s a shift we’re seeing across the industry, and one that increasingly shapes the conversations we, at Magic EdTech, are having with education organizations looking to build AI products that improve with every learner interaction.

If you’re exploring how to build AI products that learn from learner interactions without losing sight of learning outcomes, let’s start a conversation.