AI Won’t Fix Your Workflow Problems Until You Fix AI’s: The Hard Truth of AI Adoption

Understanding the Work Before the Wins

How are you preparing for AI to automate most of what you already do today? At Magic EdTech, for all AI-based workflows we’ve developed for our partners and clients, we ensure that two key elements are in place:

1. The entire workflow is built out.

2. The expectation is that some level of scalability will be achieved over time.

For instance, when building an AI workflow to translate an entire repository of textbooks, there is always an initial period where foundational understanding is developed. We understand the types of textbooks, their domains, their structure, and their layouts. There are numerous parameters and attributes that need to be accounted for before AI can be effectively implemented.

In the case of translations, this includes determining:

• Different styling methodologies are required.
• The appropriate grade band for the translations.
• Specific language nuances – e.g., Spaniard Spanish vs. Venezuelan Spanish.
• Regional linguistic differences within a language.

Then we go about creating glossaries and terminology databases so that translations maintain consistency.

This process requires significant time, and even once a workflow is implemented, the first iteration will inevitably have errors, discrepancies, and gaps.

Usually, AI workflows follow an iterative process:

1. We identify common errors.

2. Refine prompts and helper functions to mitigate those errors.

3. Test and implement changes in subsequent iterations.

4. Continue refining until the output reaches an acceptable accuracy level.

At a certain point, refinements may yield diminishing returns, meaning that no matter how many modifications are made, the outcome will remain relatively stagnant. This is where we advocate for a human-centric workflow. Subject matter experts fill in the gaps that AI cannot yet address.

This entire preparation is easier said than done. Hence, it becomes imperative to communicate this reality to clients at an early stage.

Reading Through the Pitfalls

AI is dynamic. What works today may not work tomorrow. Without proper oversight, it can introduce risks that impact outcomes in unexpected ways. A misstep in AI implementation would be:

1. Overlooking Real-Time Monitoring

This applies to organizations that have a lot of data within their infrastructure that isn’t structured or hasn’t been monitored as aggressively as it probably should be. An ideal first step would be to build continuous monitoring tools that analyze model outputs in real time using domain-specific fairness metrics. These systems go beyond standard audits to detect deviations or biases as they happen, allowing for immediate intervention before they impact users.

2. Ignoring Model Drift

This is particularly relevant for companies that are training and fine-tuning their models. Many of these foundational and LLM models are trained on billions and sometimes even trillions of different data parameters. When new information is introduced, a process known as data drifting occurs. Data drift occurs when new, changing data influences the model beyond its initial training set, potentially causing unexpected or inaccurate results. Hence, implementing systems that continuously track, not just data drift, but also shifts in model behavior over time, is crucial. Real-time monitoring alerts organizations to subtle changes that indicate growing biases or errors over time.

Avoiding these two pitfalls is especially relevant for technical personnel, as they have a greater impact on how AI will be utilized and the kind of outcomes produced. Ultimately, these considerations have a large overarching effect on what trickles down to publishers and how they leverage AI.

An Approach to Adopting AI Solutions In EdTech

So where does this leave us? What can edtech companies, publishers, or institutions do to adopt AI-driven solutions for their education products? Here’s an approach for successful adoption:

Define a Clear Objective

Pinpoint the specific challenges that need to be addressed.

Conduct a Data Audit

There’s a saying that “good data is the foundation of any successful AI.” Large models like ChatGPT, Google Cloud’s AI, and Anthropic’s Sonnet are as powerful as they are because of the quality of data they are trained on. Thus, you need to make sure that the data is high quality and current. It should comply with the necessary regulations.

Engage with Stakeholders

One of the major challenges we faced when first integrating AI into our workflows was working with subject matter experts (SMEs) who were accustomed to a legacy approach to content creation. Hence, setting clear expectations becomes really important. Bring together leadership, educators, and technical teams to ensure alignment on the vision and expected outcomes. By doing so, we were able to shift SMEs into an editorial role and free them from the tyranny of a blank page.

Similarly, even non-technical roles such as sales and marketing personnel need to understand the technicalities of AI to convey its benefits effectively to a wider audience.

Launch Small Pilot Projects

AI can be integrated in multiple ways, from small optimizations to major workflow transformations. Testing AI solutions in controlled environments allows organizations to identify potential challenges and refine their approach before full-scale implementation.

By focusing on these foundational elements, businesses can avoid common pitfalls and unleash AI’s full potential for workflow efficiency and innovation.