5 min
Why it makes things up
See why a chatbot can sound confident and still be wrong, and how to check it.
By the end of this lesson you can
- Explain why a chatbot can sound confident and still be wrong
- Pick out which answers need checking before you use them
From earlier lessons · 1 of 2
Where does a chatbot's knowledge live?
Pick one answer.
From earlier lessons · 2 of 2
Why might it not know about last week's news?
Pick one answer.
Your friend asks a chatbot for three sources for an essay. It gives three: real-sounding authors, real-sounding journals, page numbers and everything. The teacher looks them up. Two don't exist. The chatbot didn't lie on purpose. So what happened?
Take a guess
Why would a chatbot invent a source that doesn't exist?
No pressure. Guess now, and the lesson will show you if you were right.

it gave my friend two fake sources. so it lied??
not on purpose. it's doing the one thing it learned: write the likely next token
a fake journal isn't likely though
the shape is. author, title, journal, year. when it doesn't have the real one, it fills the shape with likely-sounding text
and it doesn't sound any less sure?
same tone either way. that's called a hallucination: a confident answer that's made up
so the confident tone tells me nothing?
very little. it sounds just as sure when it's guessing. the tone isn't the signal. checking is

So check these before you rely on them: names, numbers, dates, quotes, citations, laws, and medical or money advice. Ask for sources, then open them. If a link is dead or a book doesn't show up in a library search, treat the claim as unverified.
Predict, then reveal
Plot twist time. Everyone assumes that if a chatbot gives you a link, the claim is backed up. Is it?

A chatbot gives you a statistic in a very confident tone. What does the confidence tell you?
Pick one answer.
Which of these most needs checking before you use it?
Pick one answer.
FlashcardsIndex cardsFlashcards ⚡
Card 1 of 4Worth keeping
- A hallucination is fluent, likely-looking text with no fact behind it.
- A confident tone tells you very little about accuracy. It sounds just as sure when it's guessing.
- Check names, numbers, dates, quotes, citations, laws and medical or money advice before relying on them.
- A link is a lead, not proof. Open it and find the claim yourself.
One more mystery. Mention your dog's name early in a chat, and twenty messages later it still knows it. Start a new chat, and it's gone. If chatting doesn't teach the model anything, how does it remember the dog at all?