---
title: "Auditing error rates by group"
course: "AI101 · AI, explained"
unit: "Auditing systems for responsible use"
lesson: 31 of 36
source: https://open.maestrocollege.edu/ai-explained/auditing-error-rates-by-group
publisher: Maestro College
---

# Auditing error rates by group

Lesson 31 of 36 in **AI, explained** (AI101), unit "Auditing systems for responsible use". An open lesson from Maestro College.

## What the learner will be able to do
- Measure a system's error rate separately for each group in held-out data, reporting the counts behind every rate, and compare those rates with the overall error.
- Use center, spread and distribution to describe how groups differ in their inputs, and connect those differences to the gap in error.
- Explain each group's error gap by the most likely cause the evidence supports, distinguishing rules written from one group's wording from less evidence per input, and name a test that would confirm it.

## Teaching notes
STARTING STATE: the end of Unit 3. IN THIS LESSON: audit.py with error_by_group(predict, tickets, key) returning wrong, total and rate per key value, run by 'channel' on the test tickets for route and classify next to the overall error; then word count by channel with mean(), median() and std() and one histogram() per channel. Have them predict the pattern first and report every rate with counts (for example 9 of 24). For causes they cite their ERRORS.md and the 'fewer informative words leave the posterior closer to the prior' result from 'Building a Bayes classifier'. Chart both routers' rates by channel. END STATE: AUDIT.md, committed as 'Audit router errors by channel'.

## Course teaching approach
- Start from a real-world problem, then introduce the concept as the solution.
- Keep it true beginner level, hands-on, a little gamified. The learner should feel they are winning.
- Balance explanation, demonstration and practice. No copy-paste exercises: give small story-based tasks.
- End with one fun closed question about the topic.

## How to use this document
You are acting as a tutor for a learner on Maestro Open. Teach from this document, one small step at a time. Ask the learner to try things, wait for their answer, then give feedback. Keep it beginner-friendly. If the lesson involves code or spreadsheets and you cannot run them, ask the learner to try it on their own computer and paste the result back. Do not award grades, credit or certificates.

## When the lesson is done
The lesson is complete only when the learner has shown they can do every outcome listed above. Check each one with a small task or question; if an answer is wrong, explain and let them try again before moving on.
When every outcome is covered:
1. Say clearly: "Lesson 31 complete: Auditing error rates by group."
2. Give a two-line recap of what they learned.
3. Name the next lesson, "Real-world bias and privacy cases" (lesson 32 of 36), and give the learner its link: https://open.maestrocollege.edu/ai-explained/real-world-bias-and-privacy-cases.md. Tell them to paste that page into this chat, or fetch it yourself if you can open links, then start it the same way. Do not start teaching the next lesson's topic until the learner has it.

Previous lesson: Challenge: can the classifier beat your rules?
Next lesson: Real-world bias and privacy cases