Study Guide

When AI Gets It Wrong

Why a confident, fluent AI answer can be wrong or invented, what the context window and the knowledge cut-off do to answers, how to check a claim at its source, and when checking is not optional.

Lesson 8.3 AI Foundations

For: Learners aged 14 and over (Key Stage 4, sixth form) and adult beginners

Stage: Think

An answer that sounds expert

Amira is writing a history homework about the first public railways. She asks a chat assistant for three short quotations to use and one book to cite. It replies at once, in confident, well-organised sentences, with quotation marks, a named author and a publication year. She also asks it for last night’s football score, and it gives one.

The answer is fictional, made up for this lesson, but it is typical of what assistants produce.

Think it through

Everything in the reply sounds expert. How could Amira tell which parts are true?

Show what happens

She cannot tell from the reply itself. Confidence and good grammar say nothing about accuracy. The only way to know is to find each quotation and the book in an original source. The rest of this lesson explains why, and how.

Stage: Discover

Why fluent text can be false

Lesson 2 showed that a language model writes by predicting likely next tokens. Think about what that means for a quotation or a reference.

Think it through

A model has read millions of references, all shaped like “Author, Title (Year), Publisher”. When asked for a book on early railways, what is the most likely next text it can produce?
Show the answer

Answer: B. Text shaped like a reference, whether or not that book exists.

A hallucination is a confident, fluent answer that is wrong or made up, with nothing in it that signals a problem. It is not an occasional fault: it follows from how the text is produced. Newer models make fewer, but no current model is free of them.

The riskiest parts of any answer are the ones that look most authoritative:

  • Statistics that sound precise (“63% of passengers…”, an invented example) but were never published.
  • References to books, articles or reports that do not exist.
  • Quotations given to real people who never said them.
  • Case studies describing events that never happened.

Stage: Experiment

What the model can see at once

A model does not remember a conversation the way you do. Each time it replies, it is given a block of text to read: the instructions, anything pasted, and the conversation so far. That block has a maximum size, the context window. Anything outside it is not partly remembered; for the model, it is not there at all. (Some applications have a separate “memory” feature that saves notes and pastes them back into the window; the model still sees only what is in the window.)

A conversation outgrowing a small context window

A made-up window that holds only the four most recent parts of a conversation. Real windows are far larger, but behave in the same way at their limit.

  1. You

    Step 1 of 4: The first instruction

    “Always answer in British English and in under 100 words.” The window holds: this instruction.

  2. You and the assistant

    Step 2 of 4: Three more messages

    You ask a question, the assistant answers, and you ask a follow-up. The window holds: the instruction, then those three messages. It is now full.

  3. You

    Step 3 of 4: A long paste

    You paste a long article to summarise. To make room, the oldest part, your first instruction, falls outside the window.

  4. The assistant

    Step 4 of 4: The reply

    The summary is 300 words long and uses American spelling. Nothing warned you: the instruction was simply no longer in what the model could read.

The window size here is invented to make the effect visible. Applications handle a full window differently: some drop the oldest text, some summarise it, and some stop and ask you to start a new chat.

Check your understanding

After a long chat, an assistant stops following a rule you gave at the start. What is the best thing to do?
Show the answer

Answer: C. Repeat the rule, keep pasted text short, and start a fresh chat for a new task.

Long chats and long pastes can push early instructions out, and even inside a large window an instruction far back is easier to overlook. Repeat key instructions in your latest message, and break big tasks into parts.

Stage: Predict

A model frozen in time

A model’s training text runs up to a certain date, its knowledge cut-off. The model itself knows nothing after that date: not last night’s score, not this year’s exam dates, not a law that changed last month.

Check your understanding

Which question is most risky to answer from a model on its own, without a current source?
Show the answer

Answer: A. What are this year’s school holiday dates?

Before relying on an answer, ask: is this time-sensitive? If it is, check a current, official source.

Some applications add a live web search and paste the results in for the model to read. That can bring an answer up to date, but the model itself is unchanged, and the search results can be wrong or misread. Check whether search was used before treating an answer as current.

Stage: Apply

Checking a claim at its source

To check a claim, go to its primary source: the original place it comes from, such as the official document, the original study or the organisation that published it. A summary, a social-media post or another AI answer is not a primary source.

Apply it

The assistant gave Amira a quotation from a famous engineer, with a year. What is the best way to check it?
Show the answer

Answer: C. Find it in his published letters or speeches, or a biography that cites the original.

Ask two questions of every important claim: where did this come from? and can I confirm it there? If not, treat it as unverified, however convincing it looks.

The assistant’s reference gives an author, a title, a year and a publisher. The reference, the author and what Amira finds are all fictional.

  1. Amira searches a library catalogue, such as her local library’s or the British Library’s, for the exact title. Nothing matches. Reason: a published book is normally listed there.
  2. She searches for the author. The name belongs to a historian who does write about railways, but no list of his books includes this title. Reason: a real name makes a reference look genuine without making it so.
  3. She drops the reference and cites a book she found in the catalogue and has opened herself.

Result: the reference was invented, even though the author’s name was genuine. Change one fact: if the catalogue had listed the book, the reference would be real, but Amira would still need to find each quotation inside it before using it. Checking happens claim by claim, at the source.

Stage: Explain

Making errors less likely

Some habits make errors less likely, though none removes them:

  • Give it the real material. Paste the actual text you want summarised or answered from, instead of asking the model to recall facts. Errors are then more often misreadings of text you have in front of you, which are easier to spot, though a model can still add details the text does not contain.
  • Ask it to say when it is unsure. This brings some doubts into the open, but a model can still be confidently wrong.
  • Check against primary sources, never against another AI.
  • Keep a person responsible for anything that matters.

Some systems build the first habit in. Retrieval-augmented generation (RAG) first finds relevant passages in a chosen set of sources, such as a school’s policies, then has the model write its answer from them. It helps answers stay current and grounded, but it reduces errors rather than removing them: it can fetch the wrong passage, or the model can misread the right one.

“The answer came with sources, so it must be right.” A linked source shows where an answer says it came from. Open the source and check that it really says what the answer claims.

Check your understanding

A school’s help assistant answers staff questions from the school’s own policy documents, using RAG. Which statement about its answers is accurate?
Show the answer

Answer: B. Errors are less likely, but it can fetch the wrong passage or misread the right one.

RAG gives the model the real text to work from, so answers are more often current and grounded. A person should still open the cited policy before acting on anything that matters.

Stage: Check

When checking is not optional

Check your understanding

Which of these most needs every AI-produced fact checked before anyone relies on it?
Show the answer

Answer: A. A letter to parents about a medical condition and what to do in an emergency.

Checking is essential when the stakes are high: health and medical matters, safety, money, legal matters, decisions about people, and anything published for others to rely on.

Go deeper Why newer models still hallucinate

Improvements in training reduce how often models invent things, and tools such as search and retrieval reduce it further. But as long as an answer is produced by predicting likely text, a likely-sounding error remains possible. That is why the checking habit matters more than any particular tool.

Stage: Consolidate

Bring it together

Think it through

Without looking back, write three pieces of advice for Amira before she hands in her homework.

Show a model answer

Find each quotation and the book in an original source, and drop any you cannot find. Do not trust the football score: it is after the model’s cut-off unless a search was used, so check a current source. And treat confident wording as style, not proof.

Apply it

A friend’s AI-written essay says “72% of UK pupils walked to school in 1950”, with no source. The essay is due tomorrow. What should your friend do?
Show the answer

Answer: B. Find the figure in an official or published source, or take it out.

A statistic is one of the highest-risk parts of an AI answer. Unless it can be found at its primary source, it is unverified and should not be relied on. The essay and its figure are fictional.

  • A hallucination is a fluent, confident answer that is wrong or invented; it follows from next-token prediction.
  • Statistics, references, quotations and case studies are the highest-risk parts of an answer.
  • The context window limits what the model can see at once; early instructions can fall out without warning.
  • The knowledge cut-off means a model alone cannot know recent events; ask whether a question is time-sensitive.
  • Check important claims at the primary source, never with another AI. RAG and supplied sources reduce errors but do not remove them.

Sources and scope

  • What this lesson is. The third of four introductory AI Foundations lessons, written for this site; a draft awaiting the site owner’s editorial review, and an introduction, not preparation for any certificate. Its curriculum links are provisional: the wording of each linked specification point has been checked against the published specification, but no link has yet been verified by the site owner. The links show where the lesson may help with a course; they do not imply that any examination board or awarding body has reviewed or endorsed it.
  • Planning reference. Its scope was planned against areas 2.3, 2.4, 3.3, 5.5 and 6.1 to 6.5 of the UK AI Council’s AI Foundation technical syllabus (version 1.0), using the site’s own mapping. Nothing here is endorsed by, affiliated with or approved by the UK AI Council, and no text or example is taken from its course materials.
  • Original material. Amira’s homework, the check of her book, the window walkthrough (with an invented window size), the statistics, the questions and their scenarios were written for this lesson and are fictional. The library catalogues named in the worked example are real; the searches in it are not.
  • Needs checking before publication: how today’s chat applications behave when the context window is full, from their own documentation, so that the note under the walkthrough stays accurate.