6 min
Where it learned everything
Learn how training works, what parameters are, and why there's no answer database.
By the end of this lesson you can
- Explain where a chatbot's knowledge comes from
- Say why it may not know about recent events
From earlier lessons
How does a chatbot write its reply?
Pick one answer.
Ask a chatbot the capital of France and it says Paris. Ask who won a game last night and it might say it can't know, or worse, make something up. Same chatbot. So where did it learn Paris, and why didn't it learn last night?
Take a guess
Where is "Paris is the capital of France" stored inside the model?
No pressure. Guess now, and the lesson will show you if you were right.

ok where did it learn paris. is there a fact file in there
no fact file. training starts with a huge pile of text: web pages, books, articles, code
then it plays one game over and over. read some text, guess the next token, check the real one, adjust
adjust what
its parameters. a giant set of numbers. big models have billions
so paris is one of the numbers?
spread across all of them. after enough rounds, "the capital of france is" makes paris the likely next token
no table, just numbers tuned until paris comes out. wild

Raw next-token guessing isn't a helpful assistant yet. So there's a second stage, fine-tuning. The model is trained more on example conversations, and on people's ratings of which answers are better. That's where it learns to answer questions, follow instructions and decline some requests.
Training stops at some date, the training cutoff. The model never saw anything after it.
Predict, then reveal
Plot twist time. Everyone assumes that if a chatbot mentions this morning's news, the model must have learned it. Did it?

What changes during training?
Pick one answer.
A chatbot with no web search gets asked about yesterday's news. What's most likely?
Pick one answer.
FlashcardsIndex cardsFlashcards ⚡
Card 1 of 4Worth keeping
- Training tunes a huge set of numbers. Knowledge is spread across them, not stored as a fact table.
- Fine-tuning on example chats and ratings is what makes it act like an assistant.
- After its training cutoff it knows nothing new, unless the app adds a search tool.
Here's the catch. The model learned to produce text that sounds right. Usually that's also true. But when it isn't sure, it doesn't go quiet. It keeps predicting, just as smoothly.