Beyond Vanity Metrics: Why Data, Not Just AI, Will Shape the Future of EdTech

Back in New York, I sat down for another conversation to dig deeper into the role of data in education technology. While AI dominates the headlines, data, and how it is collected, will determine the success of new EdTech.

Building Features vs. Delivering Outcomes

At conferences like ISTE, it is easy to see the hype. Everyone wants to say they are ‘doing AI.’ But features without impact are fluff.

The real test is not whether a product has AI stitched into it. It is whether it helps students learn better, makes teachers’ lives easier, and drives measurable improvements in outcomes. That means engagement, understanding, and progression, not just test scores. The message is clear: EdTech’s credibility hinges on outcomes, not marketing claims. If solutions cannot demonstrate value for learners and educators, they will not last.

Measuring What Matters

Avoid the dangers of ‘vanity metrics.’ Monthly active users, logins, or enrollment figures might look impressive on a slide deck, but they do not tell us whether learning is actually happening.

Instead, we need to focus on hard-to-measure but essential indicators:

• Learning Progression: Are students mastering concepts they once struggled with?
• Engagement Quality: Are they actively participating, or just clicking through?
• Satisfaction: Are they finding value and enjoyment in the learning process?

These are the metrics that matter. And while they’re harder to capture, they’re also the only ones that give us a true picture of impact.

The Data Gap in Today’s AI Solutions

Imagine a system that learns from a student’s repeated struggles in algebra, remembers that pattern, and adapts content delivery accordingly. That is personalization, and it requires leveraging actual learner data, not just generic AI.

But solving this is hard. Every institution’s data is scattered across dozens of systems. That is why data unification and governance, building a single source of truth before AI can deliver real value, is more important than ever.

Turning Insights into Action

Data by itself is not useful unless it is actionable. A teacher does not need a flood of statistics. They need a clear signal about which students need attention and where. Likewise, administrators do not just need dropout rates after the fact; they need predictive analytics that warn them six weeks ahead of time, which students are at risk, so interventions can happen.

This focus on actionable insights is where the real potential lies. Dashboards should empower
decision-making, not overwhelm with noise.

Of course, more data brings new risks. Bias in datasets can reinforce harmful patterns if not carefully managed. If data shows that a certain demographic is at higher risk of dropping out, that is an insight to act on, not a profile to box students into. The way we interpret and present data will determine whether it empowers equity or perpetuates stereotypes.

The Future Is Hyper-Personalization

Data is king.

AI will only be as powerful as the data behind it. When used responsibly, that data enables
hyper-personalization, delivering the right content in the right format for each learner, whether that is text, audio, or video.

This is not just a technical shift. It is a cultural one. EdTech companies need to move past surface-level innovation and vanity numbers and toward solutions backed by credible, data-driven results.

Recorded in the cafe in the Paramount Building in New York City. Part of the EdTech on the Street: Real Talk in the City That Never Sleeps series.