QlearnAI

5 min

Why it makes things up

See why a chatbot can sound confident and still be wrong, and how to check it.

By the end of this lesson you can

  • Explain why a chatbot can sound confident and still be wrong
  • Pick out which answers need checking before you use them

From earlier lessons · 1 of 2

Where does a chatbot's knowledge live?

Pick one answer.

From earlier lessons · 2 of 2

Why might it not know about last week's news?

Pick one answer.

Your friend asks a chatbot for three sources for an essay. It gives three: real-sounding authors, real-sounding journals, page numbers and everything. The teacher looks them up. Two don't exist. The chatbot didn't lie on purpose. So what happened?

Take a guess

Why would a chatbot invent a source that doesn't exist?

No pressure. Guess now, and the lesson will show you if you were right.

Chatbot asked for three sources: author, journal, year, page numbers, all the right shapes. Two of them don't exist Picture: A cartoon robot proudly holding up three framed certificates; one looks official, the other two are clearly drawn in crayon, and the robot l
Zoe

it gave my friend two fake sources. so it lied??

Dev

not on purpose. it's doing the one thing it learned: write the likely next token

Zoe

a fake journal isn't likely though

Dev

the shape is. author, title, journal, year. when it doesn't have the real one, it fills the shape with likely-sounding text

Zoe

and it doesn't sound any less sure?

Dev

same tone either way. that's called a hallucination: a confident answer that's made up

Zoe

so the confident tone tells me nothing?

Dev

very little. it sounds just as sure when it's guessing. the tone isn't the signal. checking is

Chatbot when it knows the answer vs chatbot when it doesn't: same energy Picture: Two side-by-side panels of the same cartoon robot giving a confident thumbs up with a big grin, identical in both panels, one panel with a s

So check these before you rely on them: names, numbers, dates, quotes, citations, laws, and medical or money advice. Ask for sources, then open them. If a link is dead or a book doesn't show up in a library search, treat the claim as unverified.

Predict, then reveal

Plot twist time. Everyone assumes that if a chatbot gives you a link, the claim is backed up. Is it?

Teacher: where did you get this quote? Me, who trusted the chatbot: from a book that, it turns out, has never been written Picture: A cartoon student at a desk sweating nervously while a stern cartoon owl wearing glasses holds up an empty book with blank pages.

A chatbot gives you a statistic in a very confident tone. What does the confidence tell you?

Pick one answer.

Which of these most needs checking before you use it?

Pick one answer.

FlashcardsIndex cardsFlashcards ⚡

Card 1 of 4

Worth keeping

  1. A hallucination is fluent, likely-looking text with no fact behind it.
  2. A confident tone tells you very little about accuracy. It sounds just as sure when it's guessing.
  3. Check names, numbers, dates, quotes, citations, laws and medical or money advice before relying on them.
  4. A link is a lead, not proof. Open it and find the claim yourself.

One more mystery. Mention your dog's name early in a chat, and twenty messages later it still knows it. Start a new chat, and it's gone. If chatting doesn't teach the model anything, how does it remember the dog at all?