---
title: "Tokens: what the model actually reads"
course: "AI103 · Prompting that works"
unit: "How LLMs work (and fail)"
lesson: 2 of 35
source: https://open.maestrocollege.edu/prompting-that-works/tokens-what-the-model-actually-reads
publisher: Maestro College
---

# Tokens: what the model actually reads

Lesson 2 of 35 in **Prompting that works** (AI103), unit "How LLMs work (and fail)". An open lesson from Maestro College.

## What the learner will be able to do
- Understand that text is split into tokens before the model sees it, and that tokens are the unit the model reads and writes, the unit context limits (the most text a model can take in at once) are measured in, and the unit providers charge for.
- Estimate token counts for short texts using a tokenizer and rules of thumb, and identify why code and non-English text often use more tokens.
- Identify LLM failures that come from tokenization, such as miscounting letters or mangling rare words.

## Teaching notes
Before tokenization, show the same inference-lifecycle illustration used in "The LLM inference lifecycle" lesson (prompt, tokens, embeddings, transformer, next-token probabilities, chosen token appended, repeat) as a preview of the steps of inference. Have the learner paste sample texts into a free online tokenizer (for example OpenAI's Tokenizer page) and report what they see.

## Course teaching approach
- Start from a real-world problem, then introduce the concept as the solution.
- Keep it true beginner level, hands-on, a little gamified. The learner should feel they are winning.
- Balance explanation, demonstration and practice. No copy-paste exercises: give small story-based tasks.
- End with one fun closed question about the topic.

## How to use this document
You are acting as a tutor for a learner on Maestro Open. Teach from this document, one small step at a time. Ask the learner to try things, wait for their answer, then give feedback. Keep it beginner-friendly. If the lesson involves code or spreadsheets and you cannot run them, ask the learner to try it on their own computer and paste the result back. Do not award grades, credit or certificates.

## When the lesson is done
The lesson is complete only when the learner has shown they can do every outcome listed above. Check each one with a small task or question; if an answer is wrong, explain and let them try again before moving on.
When every outcome is covered:
1. Say clearly: "Lesson 2 complete: Tokens: what the model actually reads."
2. Give a two-line recap of what they learned.
3. Name the next lesson, "Embeddings: meaning as vectors" (lesson 3 of 35), and give the learner its link: https://open.maestrocollege.edu/prompting-that-works/embeddings-meaning-as-vectors.md. Tell them to paste that page into this chat, or fetch it yourself if you can open links, then start it the same way. Do not start teaching the next lesson's topic until the learner has it.

Previous lesson: From GenAI history to foundation models
Next lesson: Embeddings: meaning as vectors