1,100 simulated learners in one week: how we test our AI's instructions

Some of the software we build at Neovation teaches through AI. A learner types a command, such as "/learn how to read a balance sheet," and the AI starts a session using Socratic/conversational learning techniques to identify what they know, where their knowledge gaps are, and how to close them.

A human instructor adjusts to each learner without thinking about it. Whereas an AI adjusts only as far as its instructions cover the situations it will run into. So writing those instructions can be very challenging, and testing them is even harder.

So we built a harness that simulates the learners for us. One simulated learner is a complete beginner. Another answers every question in a single word.

Another is just angry, and so on. Each goes through the learn command experience from start to finish, on a variety of topics.

Automated checks review every reply and classify the quality of the AI's response. Then the AI rewrites the instructions, and we run another batch of tests.

Some of these testing sessions run for hours, through hundreds of conversations. In one week this month we recorded more than 1,100 of them, across /learn and four other learning commands, each built for a different kind of practice.

Four of those rounds tested all five commands together, and through a series of improvement loops, quality rose from 91% to nearly 99%.

This is AI instruction hardening through recursive self-improvement. It's the key to developing repeatable, scalable, and reliable AI learning experiences (also, it's how the AI labs are building future AI models).

You can borrow the habit yourself. Before your team relies on instructions you've written for an AI tool, especially one that helps your people learn (opens in a new tab), run some simulation loops to harden them.

Dan Belhassen
Founder & CEO, Neovation

Try this

Paste this whole prompt into any AI assistant. It tests a sample set of instructions; to test your own, replace everything after "Instructions to test".

One more tip. When a long conversation with an AI starts going in circles, ask it to "step back and look at this with fresh eyes. What are we assuming that might be wrong?" It often finds a different way in, instead of repeating the same attempt.

From Neovation

A learning expert inside your team. Knowledge Manager gives you a fractional learning and development expert and instructional designer from our own team. Start with the question you have about your training, and keep them on as the work grows. Explore Knowledge Manager (opens in a new tab)

One question. Have you ever tested an AI prompt with more than one kind of person? What broke? Hit reply and tell me.