Study Guide
When AI Gets It Wrong
Why a confident, fluent AI answer can be wrong or invented, what the context window and the knowledge cut-off do to answers, how to check a claim at its source, and when checking is not optional.
Lesson 8.3 AI Foundations
For: Learners aged 14 and over (Key Stage 4, sixth form) and adult beginners
Stage: Think
An answer that sounds expert
Amira is writing a history homework about the first public railways. She asks a chat assistant for three short quotations to use and one book to cite. It replies at once, in confident, well-organised sentences, with quotation marks, a named author and a publication year. She also asks it for last night’s football score, and it gives one.
The answer is fictional, made up for this lesson, but it is typical of what assistants produce.
Think it through
Everything in the reply sounds expert. How could Amira tell which parts are true?
- Hint 1: Does sounding confident tell you anything about being correct?
- Hint 2: Where would each quotation and the book have to exist if they are real?
Show what happens
She cannot tell from the reply itself. Confidence and good grammar say nothing about accuracy. The only way to know is to find each quotation and the book in an original source. The rest of this lesson explains why, and how.
Stage: Discover
Why fluent text can be false
Lesson 2 showed that a language model writes by predicting likely next tokens. Think about what that means for a quotation or a reference.
Think it through
Show the answer
Answer: B. Text shaped like a reference, whether or not that book exists.
A hallucination is a confident, fluent answer that is wrong or made up, with nothing in it that signals a problem. It is not an occasional fault: it follows from how the text is produced. Newer models make fewer, but no current model is free of them.
The riskiest parts of any answer are the ones that look most authoritative:
- Statistics that sound precise (“63% of passengers…”, an invented example) but were never published.
- References to books, articles or reports that do not exist.
- Quotations given to real people who never said them.
- Case studies describing events that never happened.
Stage: Experiment
What the model can see at once
A model does not remember a conversation the way you do. Each time it replies, it is given a block of text to read: the instructions, anything pasted, and the conversation so far. That block has a maximum size, the context window. Anything outside it is not partly remembered; for the model, it is not there at all. (Some applications have a separate “memory” feature that saves notes and pastes them back into the window; the model still sees only what is in the window.)
A made-up window that holds only the four most recent parts of a conversation. Real windows are far larger, but behave in the same way at their limit.
You
Step 1 of 4: The first instruction
“Always answer in British English and in under 100 words.” The window holds: this instruction.
You and the assistant
Step 2 of 4: Three more messages
You ask a question, the assistant answers, and you ask a follow-up. The window holds: the instruction, then those three messages. It is now full.
You
Step 3 of 4: A long paste
You paste a long article to summarise. To make room, the oldest part, your first instruction, falls outside the window.
The assistant
Step 4 of 4: The reply
The summary is 300 words long and uses American spelling. Nothing warned you: the instruction was simply no longer in what the model could read.
The window size here is invented to make the effect visible. Applications handle a full window differently: some drop the oldest text, some summarise it, and some stop and ask you to start a new chat.
Check your understanding
Show the answer
Answer: C. Repeat the rule, keep pasted text short, and start a fresh chat for a new task.
Long chats and long pastes can push early instructions out, and even inside a large window an instruction far back is easier to overlook. Repeat key instructions in your latest message, and break big tasks into parts.
Stage: Predict
A model frozen in time
A model’s training text runs up to a certain date, its knowledge cut-off. The model itself knows nothing after that date: not last night’s score, not this year’s exam dates, not a law that changed last month.
Check your understanding
Show the answer
Answer: A. What are this year’s school holiday dates?
Before relying on an answer, ask: is this time-sensitive? If it is, check a current, official source.
Some applications add a live web search and paste the results in for the model to read. That can bring an answer up to date, but the model itself is unchanged, and the search results can be wrong or misread. Check whether search was used before treating an answer as current.
Stage: Apply
Checking a claim at its source
To check a claim, go to its primary source: the original place it comes from, such as the official document, the original study or the organisation that published it. A summary, a social-media post or another AI answer is not a primary source.
Apply it
Show the answer
Answer: C. Find it in his published letters or speeches, or a biography that cites the original.
Ask two questions of every important claim: where did this come from? and can I confirm it there? If not, treat it as unverified, however convincing it looks.
The assistant’s reference gives an author, a title, a year and a publisher. The reference, the author and what Amira finds are all fictional.
- Amira searches a library catalogue, such as her local library’s or the British Library’s, for the exact title. Nothing matches. Reason: a published book is normally listed there.
- She searches for the author. The name belongs to a historian who does write about railways, but no list of his books includes this title. Reason: a real name makes a reference look genuine without making it so.
- She drops the reference and cites a book she found in the catalogue and has opened herself.
Result: the reference was invented, even though the author’s name was genuine. Change one fact: if the catalogue had listed the book, the reference would be real, but Amira would still need to find each quotation inside it before using it. Checking happens claim by claim, at the source.
Stage: Explain
Making errors less likely
Some habits make errors less likely, though none removes them:
- Give it the real material. Paste the actual text you want summarised or answered from, instead of asking the model to recall facts. Errors are then more often misreadings of text you have in front of you, which are easier to spot, though a model can still add details the text does not contain.
- Ask it to say when it is unsure. This brings some doubts into the open, but a model can still be confidently wrong.
- Check against primary sources, never against another AI.
- Keep a person responsible for anything that matters.
Some systems build the first habit in. Retrieval-augmented generation (RAG) first finds relevant passages in a chosen set of sources, such as a school’s policies, then has the model write its answer from them. It helps answers stay current and grounded, but it reduces errors rather than removing them: it can fetch the wrong passage, or the model can misread the right one.
“The answer came with sources, so it must be right.” A linked source shows where an answer says it came from. Open the source and check that it really says what the answer claims.
Check your understanding
Show the answer
Answer: B. Errors are less likely, but it can fetch the wrong passage or misread the right one.
RAG gives the model the real text to work from, so answers are more often current and grounded. A person should still open the cited policy before acting on anything that matters.
Stage: Check
When checking is not optional
Check your understanding
Show the answer
Answer: A. A letter to parents about a medical condition and what to do in an emergency.
Checking is essential when the stakes are high: health and medical matters, safety, money, legal matters, decisions about people, and anything published for others to rely on.
Go deeper Why newer models still hallucinate
Improvements in training reduce how often models invent things, and tools such as search and retrieval reduce it further. But as long as an answer is produced by predicting likely text, a likely-sounding error remains possible. That is why the checking habit matters more than any particular tool.
Stage: Consolidate
Bring it together
Think it through
Without looking back, write three pieces of advice for Amira before she hands in her homework.
Show a model answer
Find each quotation and the book in an original source, and drop any you cannot find. Do not trust the football score: it is after the model’s cut-off unless a search was used, so check a current source. And treat confident wording as style, not proof.
Apply it
Show the answer
Answer: B. Find the figure in an official or published source, or take it out.
A statistic is one of the highest-risk parts of an AI answer. Unless it can be found at its primary source, it is unverified and should not be relied on. The essay and its figure are fictional.
- A hallucination is a fluent, confident answer that is wrong or invented; it follows from next-token prediction.
- Statistics, references, quotations and case studies are the highest-risk parts of an answer.
- The context window limits what the model can see at once; early instructions can fall out without warning.
- The knowledge cut-off means a model alone cannot know recent events; ask whether a question is time-sensitive.
- Check important claims at the primary source, never with another AI. RAG and supplied sources reduce errors but do not remove them.
Sources and scope
- What this lesson is. The third of four introductory AI Foundations lessons, written for this site; a draft awaiting the site owner’s editorial review, and an introduction, not preparation for any certificate. Its curriculum links are provisional: the wording of each linked specification point has been checked against the published specification, but no link has yet been verified by the site owner. The links show where the lesson may help with a course; they do not imply that any examination board or awarding body has reviewed or endorsed it.
- Planning reference. Its scope was planned against areas 2.3, 2.4, 3.3, 5.5 and 6.1 to 6.5 of the UK AI Council’s AI Foundation technical syllabus (version 1.0), using the site’s own mapping. Nothing here is endorsed by, affiliated with or approved by the UK AI Council, and no text or example is taken from its course materials.
- Original material. Amira’s homework, the check of her book, the window walkthrough (with an invented window size), the statistics, the questions and their scenarios were written for this lesson and are fictional. The library catalogues named in the worked example are real; the searches in it are not.
- Needs checking before publication: how today’s chat applications behave when the context window is full, from their own documentation, so that the note under the walkthrough stays accurate.
Computer systems designed to do tasks that would normally need human reasoning, such as recognising, predicting, deciding or producing content.
Everything a language model can take into account at once: the prompt, any pasted text and the conversation so far. What falls outside it is not seen at all.
A confident, fluent answer from an AI system that is wrong or made up, with nothing in it to show that it is wrong.
The date up to which a model’s training data runs. The model itself knows nothing about later events.
A model that predicts which piece of text is likely to come next, and so can produce text one piece at a time.
The stage in which a model’s parameters are adjusted, using large amounts of example data, until its predictions fit the data well.
How a language model writes: it gives every possible next token a probability, chooses one, adds it, and repeats.
The original place a fact comes from, such as the official document, the original study or the organisation that published it.
A way of answering in which a system first finds relevant passages from chosen sources, then has the model write an answer based on them.
A piece of text a language model reads and writes: often a whole common word, sometimes part of a word or a punctuation mark.
Your progress on this lesson
Saved in this browser only. It is not sent anywhere, does not follow you to another device, and is lost if you clear this site’s data.
This browser is not allowing the site to save progress, so it will be forgotten when you leave the page.