Case Study
“I’m sorry Dave, I’m afraid I can’t do that.” (without your help…)
Evaluating & Refining AI-Generated Models for Student-Focused Content
Key Result Highlights
- 100+ questions posed, and feedback analyzed and reported upon, relative to the chatbot tutor(s)*
- 120+ practice questions (including feedback) assessed*
- 400+ summaries - in both bulleted and paragraph form - reviewed*
- *All feedback submitted by SMEs used to “fine-tune” the AI models.
The Client
The client is a leading global educational technology and publishing company serving students in K-20.
The Challenge
The client was testing the use of AI-generated content to aid students assigned an array of their best-selling higher education titles. The AI models were used to generate “title-specific” summaries, practice questions (with feedback), and provide students with tutor-like advice, guidance, and study tips through a chatbot feature. The challenge was to provide subject matter experts in key domains who could provide feedback on the accuracy, clarity, engagement, appropriateness, tone, helpfulness, and ethical considerations (e.g., IP) of the AI.
Critical Success Factors
- Train and fine-tune the models for the AI-generated content – i.e., summaries (both bulleted and in paragraph form), practice questions (with feedback), and chatbot tutor suggestions to students – to ensure accuracy and achieve overall quality criteria.
- The chatbot, regardless of the specificity or broadness of the question posed to it by a user, responds with helpful, title-specific, discipline-accurate feedback and/or guidance to the student, in an appropriate, approachable tone.
Our Approach
- Magic assigned subject matter experts familiar with and experienced in using the relevant titles to evaluate the summaries and practice questions (+ feedback) – to the AI-generated content offered comprehensive synopses and useful practice/clarifying feedback, respectively. SME responses aided in training and “fine-tuning” the AI models in use.
- Magic’s SMEs (several on each title) posed a battery of varying questions to the chatbot. They then reviewed and provided feedback on the accuracy and appropriateness of the responses. AI-chatbot responses to the SMEs were evaluated on accuracy (including the evasion of “hallucinations”), clarity, helpfulness, tone, and ethical considerations.
- Magic’s subject matter experts were selected both because of their experience and their combined open-mindedness and healthy skepticism relative to AI.
Get in Touch
Looking to achieve the same results for your organization? Speak with our Team!