---
title: "Measuring accuracy on held-out data"
course: "AI101 · AI, explained"
unit: "Building with rules and search"
lesson: 14 of 36
source: https://open.maestrocollege.edu/ai-explained/measuring-accuracy-on-held-out-data
publisher: Maestro College
---

# Measuring accuracy on held-out data

Lesson 14 of 36 in **AI, explained** (AI101), unit "Building with rules and search". An open lesson from Maestro College.

## What the learner will be able to do
- Understand why a system must be measured on held-out examples that were never used to build or tune it, and why adding rules to fix each held-out error inflates the measured accuracy instead of improving the system.
- Create an evaluation function that computes accuracy for any prediction function and labeled dataset, and compare a system's accuracy on build data, on held-out data and against always guessing the most common category.
- Analyze a system's errors by listing each wrong prediction and sorting it into a cause such as a missing keyword, unexpected wording, an abbreviation or a rule conflict.

## Teaching notes
STARTING STATE: a priority-ordered route(); no RESULTS.md or ERRORS.md yet. IN THIS LESSON: the learner writes evaluate.py with accuracy(predict, tickets), writes down the test accuracy they expect, then prints train and test accuracy for route and the test accuracy of always guessing the most common training category. Then misses(predict, tickets), returning id, channel, text, expected and predicted; they sort at least eight TRAINING misses by cause, with counts. Rules may improve from training misses only. END STATE: RESULTS.md (router train and test, most-common test) and ERRORS.md 'Training misses', committed as 'Measure router on held-out tickets'.

## Course teaching approach
- Start from a real-world problem, then introduce the concept as the solution.
- Keep it true beginner level, hands-on, a little gamified. The learner should feel they are winning.
- Balance explanation, demonstration and practice. No copy-paste exercises: give small story-based tasks.
- End with one fun closed question about the topic.

## How to use this document
You are acting as a tutor for a learner on Maestro Open. Teach from this document, one small step at a time. Ask the learner to try things, wait for their answer, then give feedback. Keep it beginner-friendly. If the lesson involves code or spreadsheets and you cannot run them, ask the learner to try it on their own computer and paste the result back. Do not award grades, credit or certificates.

## When the lesson is done
The lesson is complete only when the learner has shown they can do every outcome listed above. Check each one with a small task or question; if an answer is wrong, explain and let them try again before moving on.
When every outcome is covered:
1. Say clearly: "Lesson 14 complete: Measuring accuracy on held-out data."
2. Give a two-line recap of what they learned.
3. Name the next lesson, "Chaining rules into an expert system" (lesson 15 of 36), and give the learner its link: https://open.maestrocollege.edu/ai-explained/chaining-rules-into-an-expert-system.md. Tell them to paste that page into this chat, or fetch it yourself if you can open links, then start it the same way. Do not start teaching the next lesson's topic until the learner has it.

Previous lesson: Ordering rules and checking coverage
Next lesson: Chaining rules into an expert system