--- title: "AI103 · Prompting that works" publisher: Maestro College (Maestro Open) lessons: 35 source: https://open.maestrocollege.edu/prompting-that-works full: https://open.maestrocollege.edu/prompting-that-works/llms-full.txt --- # Prompting that works (AI103) How large language models work and where they fail, and how to write prompts and build simple LLM systems that give reliable results. ## Course objectives - Explain how large language models process language, generate outputs, and exhibit limitations such as hallucination, bias, and context sensitivity. - Design and refine prompts that provide effective instructions, context, examples, constraints, and structured outputs for reliable model behavior. - Develop Python-based LLM applications that use APIs, structured outputs, tool calling, and cost-aware request handling. - Evaluate and select LLM models and system architectures based on task requirements, capabilities, privacy, performance, and cost considerations. - Design supervised agent workflows and use evaluation, tracing, and monitoring practices to assess and improve LLM systems. ## Course teaching approach - Start from a real-world problem, then introduce the concept as the solution. - Keep it true beginner level, hands-on, a little gamified. The learner should feel they are winning. - Balance explanation, demonstration and practice. No copy-paste exercises: give small story-based tasks. - End with one fun closed question about the topic. ## Lessons ### Unit 1: How LLMs work (and fail) #### 1. From GenAI history to foundation models - Place major GenAI milestones—including early statistical language models, the transformer architecture, large pretrained GPT-style models, and chat assistants—on a timeline leading to today’s foundation models, and explain that training minimizes loss (prediction error) to improve predictions, and distinguish training (learning the model's parameters once, in advance, from data) from inference (using the trained model, with those parameters fixed, to respond to each new prompt). - Understand next-token prediction: an LLM generates text by repeatedly predicting how likely each possible next piece of text is, given everything before it, and picking one, so every output is a likely continuation, not a looked-up answer. - Identify what makes a foundation model: one large model pretrained (trained in advance) on broad data and then adapted to many tasks. #### 2. Tokens: what the model actually reads - Understand that text is split into tokens before the model sees it, and that tokens are the unit the model reads and writes, the unit context limits (the most text a model can take in at once) are measured in, and the unit providers charge for. - Estimate token counts for short texts using a tokenizer and rules of thumb, and identify why code and non-English text often use more tokens. - Identify LLM failures that come from tokenization, such as miscounting letters or mangling rare words. Teaching notes: Before tokenization, show the same inference-lifecycle illustration used in "The LLM inference lifecycle" lesson (prompt, tokens, embeddings, transformer, next-token probabilities, chosen token appended, repeat) as a preview of the steps of inference. Have the learner paste sample texts into a free online tokenizer (for example OpenAI's Tokenizer page) and report what they see. #### 3. Embeddings: meaning as vectors - Understand that a vector is a list of numbers that can represent a piece of text, and that similarity is a simple way to compare how closely two vectors represent related meanings. - Predict which of several texts would land closest together as embeddings, and identify how this lets a system find related text by meaning rather than exact words. #### 4. The LLM inference lifecycle - Trace the lifecycle of an LLM from pretraining on large text data to a trained model with fixed parameters, then through inference on each request: the prompt becomes tokens, tokens become embeddings, the transformer produces next-token probabilities, one token is chosen and appended, and the loop repeats until the response ends. - Distinguish what happens once during training from what happens on every request during inference, and identify which steps use the tokens and embeddings learned about so far. Teaching notes: Create an illustration of the LLM inference lifecycle: pretraining on text data producing a trained model with fixed parameters, then for each request a prompt flowing through tokens, embeddings, and transformer layers to next-token probabilities, with the chosen token appended to the text in a repeating loop. Reuse this same illustration in the tokens lesson. #### 5. How attention works inside transformers - Identify a transformer as a model architecture and describe, at a basic level, how attention lets it weigh and relate tokens in context when building each token's meaning. - Trace in plain language how a prompt flows through an LLM: text to tokens, tokens to embeddings, attention layers, and finally a probability distribution over the next token. - Identify why the placement of key instructions and facts in a long prompt affects how strongly the model attends to them. Teaching notes: Create an illustration that shows tokens flowing through embeddings and attention layers into a bar chart of next-token probabilities. #### 6. How base models become assistants - Understand that pretraining applies the loss-minimizing training you already know to next-token prediction on huge text corpora, giving a model its patterns and frozen knowledge, while post-training (instruction tuning, human feedback) shapes how it behaves as an assistant. - Distinguish a base model from an instruction-tuned assistant by how each responds to the same input. - Identify what a knowledge cutoff means and why a model's weights do not learn from your conversations, even though products may add memory features and providers may use chat data to train future models. Teaching notes: Role-play a base model and then an instruction-tuned assistant answering the same input, then step out of role to compare. #### 7. What the model sees in its context window - Understand that on each call the model sees only the tokens in its context window (system instructions, conversation history, retrieved documents, tool results) and keeps nothing between calls unless code sends it again. - Understand inference as a generation loop in which each new token is added to the context before the next is predicted, so longer outputs take more time and tokens. - Identify what happens when a conversation grows past the context window, and how code handles it by trimming or summarizing older turns. #### 8. Sampling, temperature, and non-determinism - Predict how the same prompt's output changes at low and high temperature, and understand that the model samples the next token from its probability distribution, with temperature and top-p controlling how random the choice is. - Identify non-determinism in practice, and understand why a low temperature makes outputs more repeatable but not more correct. Teaching notes: Use math-visual to draw bar charts of the same next-token probability distribution at low and at high temperature. #### 9. Why models hallucinate and show bias - Understand that because an LLM optimizes for plausible continuations, fluent output is not verified output: each output is evidence to check, not a fact to trust. - Identify hallucinations in sample outputs and trace each to a likely cause, such as missing knowledge, a knowledge cutoff, or pressure to give a plausible answer. - Identify bias in sample outputs and trace it to patterns in pretraining data or choices made in post-training. #### 10. Context sensitivity and reasoning failures - Identify context sensitivity: how small changes in wording, order, or placement of information change a model's output. - Identify inconsistent reasoning, such as contradictions or wrong steps behind a confident answer, and understand why it emerges from next-token prediction. - Classify a flawed LLM output by failure mode (hallucination, bias, non-determinism, context sensitivity, or inconsistent reasoning) and name the model characteristic that most likely caused it. Teaching notes: Role-play a generic LLM answering slightly reworded versions of the same prompt, then step out of role to show how the wording shifted your prediction. ### Unit 2: Prompt and context engineering #### 11. Anatomy of a strong prompt - Understand that a strong prompt is a designed context that deliberately supplies the instruction, relevant context, examples, constraints, and output format the model needs, so the desired output becomes the most likely continuation. - Identify which of these components are missing from a weak prompt and predict how the model will fill the gaps. - Predict how a model will read a prompt before running it, and recognize the predict-design-inspect-refine loop as the method the rest of the unit practices. #### 12. Clear instructions and roles - Write instructions that state the task, audience, and success criteria in specific, direct language instead of vague requests. - Distinguish a system prompt from user messages, and use a role in the system prompt to set expertise and tone. - Identify an ambiguous instruction by predicting two different reasonable ways the model could read it. #### 13. Supplying relevant context - Identify the background facts, data, or documents a model needs for a task that it cannot know from pretraining. - Organize supplied context with clear delimiters (such as headings or XML-style tags) and place the instruction where the model will weigh it. - Distinguish relevant context from noise that distracts the model or wastes tokens in a long prompt. #### 14. Few-shot examples - Distinguish zero-shot prompting (no examples) from few-shot prompting (worked input-output examples included in the prompt), and identify when examples help more than extra instructions. - Create few-shot examples that are varied, correct, and formatted exactly like the desired output. - Identify how examples can mislead a model, such as copying surface details or skewing toward one label. #### 15. Constraints and output formats - Write explicit constraints on length, scope, tone, and what the model should do when information is missing. - Specify an output format (such as a list, table, or labeled sections) precisely enough that the output can be used without editing. - Identify when a positive instruction (do X) works better than a prohibition (don't do Y). #### 16. Prompting for JSON outputs - Understand that when code consumes an LLM's output, the output must follow an agreed structure, such as a JSON schema, that code validates before acting. - Write a prompt that asks for JSON matching a defined schema with field names, types, and allowed values. - Identify common ways model-generated JSON breaks (extra text, missing fields, wrong types, invented values) and adjust the prompt to prevent them. #### 17. Breaking tasks into steps - Explain step-by-step reasoning prompting as asking a model to work through intermediate steps, identify when it can help with complex multi-step tasks, and recognize that the reasoning and final answer can still contain errors. - Break a complex task into smaller steps, either inside one prompt or as a chain of prompts where one output feeds the next. #### 18. Iterative prompt refinement - Create a small set of test inputs, including edge cases, to check whether a prompt works beyond a single run. - Refine a prompt one change at a time, comparing outputs across the test inputs to see which change helped. - Judge whether a prompt is reliable by evidence (consistent results across test inputs and repeated runs), not by one good output. Teaching notes: Alongside your role-play, have the learner run their prompt versions and repeated runs in a free no-code playground such as Google AI Studio, so consistency is checked on a real model. #### 19. Diagnosing failing prompts and prompt injection - Trace a bad output to the missing, ambiguous, or misleading part of the context that caused it, naming the failure mode it shows. - Fix the prompt by changing only the component responsible for the failure. - Identify prompt injection, where instructions hidden in supplied content override the intended task, and add basic defenses such as delimiters and system-prompt rules, knowing that these reduce the risk but do not eliminate it. #### 20. Challenge: engineer a reliable prompt - Rewrite a weak prompt into one with clear instructions, relevant context, at least one example, explicit constraints, and a defined output structure. - Refine the prompt over at least two iterations until it produces the required output consistently across three test inputs. Teaching notes: Create a challenge-based lesson grounded in everything covered in this unit so far. Run four to eight interactive, dynamic rounds (varied formats, rolling challenges, mini tasks). Do not correct during the challenge; at the end, give an encouraging summary of strengths and areas to refine. ### Unit 3: Building and choosing LLM systems #### 21. First API call in Python - Set up a Python project in VS Code with a virtual environment, an LLM provider's SDK, and an API key kept in a .env file that is never shared or committed. - Write a script that sends a system message and a user message to a free-tier LLM API and prints the response text. - Understand that the model keeps nothing between calls, so with a stateless chat API your code must resend the conversation history on every call to hold a conversation. #### 22. Token usage and API pricing - Read the usage data an API response returns, including input and output tokens per call. - Calculate the cost of a call and of a projected workload from per-million-token prices, accounting for history that is billed again on every call. - Identify ways to cut token cost (shorter context, smaller models, capped output length) and the free-tier rate limits that cap usage. Teaching notes: Supply a short script the learner runs in their own VS Code to print the usage data of one API call, or a sample API response with usage data if they cannot run it. Seed the spreadsheet with labeled input cells (calls per day, tokens per call, history growth per turn, per-million-token input and output prices), and have the learner build the projected-workload cost model there. #### 23. Validating JSON from an API - Request JSON output that matches a defined schema through an API's structured output or JSON mode (a provider feature that makes the response follow valid JSON or a given schema), setting temperature and maximum output tokens in code. - Parse and validate the returned JSON in Python, and handle invalid output or API errors such as rate limits with a retry or a safe fallback. #### 24. Tool and function calling - Understand the tool-calling flow: code describes tools to the model, the model returns a tool-call request with arguments, and code runs the function and sends the result back for the final answer. - Write a script that defines at least one Python function as a tool, executes the model's tool-call request, and returns the result to the model. - Identify why code must validate tool-call arguments and decide whether a requested call should run. #### 25. Challenge: build a working LLM script - Build a Python script that calls a free-tier LLM API with a designed prompt, returns validated structured output, and defines at least one tool. - Estimate the token cost of the script's calls from their usage data. Teaching notes: Create a challenge-based lesson grounded in everything covered in this unit so far. Run four to eight interactive, dynamic rounds (varied formats, rolling challenges, mini tasks). Do not correct during the challenge; at the end, give an encouraging summary of strengths and areas to refine. #### 26. Modalities and reasoning modes - Identify the input and output modalities (kinds of data such as text, image, audio, or video) that models support and match them to tasks, including multimodal pipelines that chain models across modalities (for example speech-to-text, then an LLM, then text-to-speech). - Understand reasoning modes, where a model generates extra reasoning tokens before answering, and run the same multi-step question with reasoning on and off to judge from reasoning-token counts and latency whether the accuracy gain is worth the added cost. Teaching notes: Supply a script that calls a free-tier model with a reasoning setting (for example a Gemini model with a thinking budget), and have the learner paste back the answers, usage, and timing. #### 27. Choosing commercial or open-weight models - Distinguish closed (proprietary) commercial models from open-weight models, knowing that either can be used through a hosted API but only open weights can be downloaded and self-hosted, and identify what each means for privacy (where your data is sent and stored) and why open-weight is not automatically free, private, or fully open source. - Identify what a public benchmark or leaderboard measures and why a high rank does not prove a model fits your task. - Understand that no model is best for every task, and for a described use case compare at least two closed commercial models and one open-weight model on capability, modality, reasoning mode, latency, privacy, deployment, model size (parameter count), and token cost, justifying a recommendation in 3-5 sentences. Teaching notes: Have the learner check current specs, prices, and leaderboard ranks on provider pages or a comparison site, since names and prices change often. #### 28. Retrieval-augmented generation (RAG) - Understand RAG: code retrieves relevant documents, usually by embedding similarity, and places them in the context so the model answers from supplied facts instead of frozen knowledge, without retraining. - Trace the steps of a RAG pipeline: chunking documents, embedding them, storing them in a vector database (a store that finds the embeddings closest to a query), retrieving the top matches, and building the prompt. - Identify how RAG fails (wrong or missing chunks, retrieving too few or too many chunks, the model ignoring sources) and how asking for cited sources helps check answers. Teaching notes: Create an illustration that shows the RAG data flow from documents through chunking, embedding, and retrieval into the prompt. Also role-play an LLM answering from a supplied set of chunks where one is wrong or irrelevant, then step out of role to show how it misled you. #### 29. Choosing a GenAI system architecture - Identify compound AI systems and context systems as combinations of parts you already know (several model calls, retrieval, tools, and the history trimming and summarizing you have already seen) that assemble what reaches the context on each call. - Understand that an architecture (plain prompt, RAG, multimodal pipeline, tool workflow, or compound system) is chosen by what information or capability the model is missing and how code will supply it, preferring the simplest option that works. - Sketch a labeled data-flow diagram of the right architecture for a product requirement, including the kind of model it needs (for example commercial or open-weight), and justify why it fits better than a simpler option. #### 30. From tool call to agent loop - Understand that an agent is an LLM called repeatedly inside a loop that code controls: the model picks the next tool call, code runs it and adds the result to the context, and the loop repeats until the model gives a final answer or a limit is reached; the growing message list is the agent's state. - Extend a single tool-calling script into a loop that handles several tool calls in sequence, with a maximum step count as a safety stop. - Add a human-approval check that pauses before a tool with real side effects runs, and identify what could go wrong without it. ### Unit 4: Agents, evals, and observability #### 31. Inside an agent harness - Understand that a harness is the code around the agent loop that supplies state and memory, sets which tools and permissions the agent has, and defines when it stops or hands control to a human, and distinguish short-term state (the running message list in the context) from long-term memory (information stored and retrieved across runs). - Distinguish a single agent from an orchestrator that hands subtasks to worker agents, and identify when splitting a task across agents is worth the extra cost and failure points. - Match the main agent risks (runaway loops, unsafe or irreversible actions, prompt injection through tool results or retrieved content) to controls you already know (step limits, human approval, argument validation) plus scoped permissions. #### 32. Building evals beyond benchmarks Teaching notes: Supply a Python script that loops over the learner's eval cases and records latency and usage for each case. #### 33. Tracing runs and monitoring production - Understand tracing as recording each step of an LLM system run (prompts, retrieved context, tool calls, outputs, tokens, latency) so failures can be located. - Read a trace of a failed multi-step run and identify the step where it went wrong. - Define what to log and monitor in production, including user feedback, latency, error rates, and token cost per request, and when an alert should fire. Teaching notes: Create an illustration of a trace as a timeline of nested steps with tokens and latency shown on each step. #### 34. Challenge: design an agent with oversight - Decide whether a given task needs an agent or a simpler fixed workflow, and for an agent task sketch its control loop, labeling its tools, memory, state, orchestration, permissions, and the points where a human approves or stops actions. - Identify at least two risks the design prevents, including prompt injection through tool results or retrieved content, and name the control that blocks each. Teaching notes: Create a challenge-based lesson grounded in everything covered in this unit so far. Run four to eight interactive, dynamic rounds (varied formats, rolling challenges, mini tasks). Do not correct during the challenge; at the end, give an encouraging summary of strengths and areas to refine. #### 35. Challenge: evaluate and improve an LLM system - Diagnose the main failure by mode, make one change (the prompt, the model, or the number of documents retrieved), rerun the eval, and decide whether to keep the change by weighing quality against latency and token cost. Teaching notes: Create a challenge-based lesson grounded in everything covered in this unit so far. Run four to eight interactive, dynamic rounds (varied formats, rolling challenges, mini tasks). Do not correct during the challenge; at the end, give an encouraging summary of strengths and areas to refine. ## How to use this document You are acting as a tutor for a learner on Maestro Open. Teach from this document, one small step at a time. Ask the learner to try things, wait for their answer, then give feedback. Keep it beginner-friendly. If the lesson involves code or spreadsheets and you cannot run them, ask the learner to try it on their own computer and paste the result back. Do not award grades, credit or certificates. Start with lesson 1 unless the learner asks for a specific lesson. Keep track of which lesson you are on. ## When a lesson is done Teach one lesson at a time. A lesson is complete only when the learner has shown they can do every outcome listed for it; check each with a small task or question. When it is complete: 1. Say clearly: "Lesson N complete: ." (use the lesson's number and title). 2. Give a two-line recap of what they learned. 3. Name the next lesson, say "Lesson N+1 of 35: ", and ask if they're ready to start it. Start it only when they say yes. After the last lesson, congratulate them on finishing the course.