---
title: "Experiments and A/B tests"
course: "DATA 101 · Question the numbers"
unit: "Where data comes from"
lesson: 11 of 16
source: https://open.maestrocollege.edu/question-the-numbers/experiments-and-a-b-tests
publisher: Maestro College
---

# Experiments and A/B tests

Lesson 11 of 16 in **Question the numbers** (DATA 101), unit "Where data comes from". An open lesson from Maestro College.

## What the learner will be able to do
- Explain why randomly assigning people to groups is the strongest way to test whether something causes a change.
- Describe a simple A/B test: two versions, random assignment, one main measure and a decision rule set in advance.
- Explain why small tests can show differences that are just chance, and why checking results many times or across many measures can mislead.
- Judge whether an example experiment supports the claim made about it.

## Teaching notes
Use an invented online shop testing two checkout button labels. Avoid formulas; use the idea of flipping coins to show how chance creates apparent differences in small groups.

## Course teaching approach
- Start from a real-world problem, then introduce the concept as the solution.
- Keep it true beginner level, hands-on, a little gamified. The learner should feel they are winning.
- Balance explanation, demonstration and practice. No copy-paste exercises: give small story-based tasks.
- End with one fun closed question about the topic.

## How to use this document
You are acting as a tutor for a learner on Maestro Open. Teach from this document, one small step at a time. Ask the learner to try things, wait for their answer, then give feedback. Keep it beginner-friendly. If the lesson involves code or spreadsheets and you cannot run them, ask the learner to try it on their own computer and paste the result back. Do not award grades, credit or certificates.

## When the lesson is done
The lesson is complete only when the learner has shown they can do every outcome listed above. Check each one with a small task or question; if an answer is wrong, explain and let them try again before moving on.
When every outcome is covered:
1. Say clearly: "Lesson 11 complete: Experiments and A/B tests."
2. Give a two-line recap of what they learned.
3. Name the next lesson, "Challenge: does the evidence hold up?" (lesson 12 of 16), and give the learner its link: https://open.maestrocollege.edu/question-the-numbers/challenge-does-the-evidence-hold-up.md. Tell them to paste that page into this chat, or fetch it yourself if you can open links, then start it the same way. Do not start teaching the next lesson's topic until the learner has it.

Previous lesson: Correlation and causation
Next lesson: Challenge: does the evidence hold up?