In a nutshell Property #2 of four: what a model actually knows — and why it can be confidently wrong. Its knowledge is "baked in" during training, goes stale at a cut-off date, and doesn't cover everything. And since it builds a plausible continuation (see next-token prediction), on the unfamiliar it doesn't go quiet — it smoothly makes things up. That's a hallucination.
Where the model's knowledge lives
During training the model tunes billions of weights to predict text well. Facts and patterns from what it read are dissolved into those weights — this is parametric knowledge. Three consequences:
- Knowledge is "smeared", not filed on shelves. There's no cell "capital of France = Paris"; there's a tuning that makes "Paris" the likeliest continuation of "The capital of France is". Common, stable facts it knows firmly; rare ones, fuzzily.
- There's a knowledge cut-off. The model knows the world roughly up to when its training data was gathered. What happened later it doesn't know "off the top of its head".
- No live access to the world. By itself it doesn't browse, open your files, or know today's date — unless given that through tools (search, documents) or right in the prompt.
Analogy. An erudite locked in a room with no internet and no clock. They remember enormously much — but only up to the day they were shut in, and only what they'd read. Yesterday's news or your private document, they cannot know.
Why a model can be "confidently wrong"
That's a hallucination — a falsehood in the same confident tone as the truth. Typical victims:
- rare, narrow facts (a little-known person, a local event);
- anything after the cut-off date;
- precise details — numbers, dates, quotes, names, links, article titles: especially easy to "fill in";
- invented links and non-existent sources (they sound real).
Why: to a token predictor, a "plausible link" and a "real link" look equally good — both continue the text smoothly. It can't tell them apart without an external check.
How to work with it
Hallucinations aren't a bug that gets fixed — they follow from the model's nature. So you work around them:
- Put the facts in the prompt. Not "what does this law say?", but paste the law and ask about it. The model is great with what's in front of it (see working memory & context). This is the RAG principle — "add the source, then ask".
- Give it tools. Search, database/document access, a calculator — then facts come from a real source, not from memory.
- Ask it to cite and admit uncertainty. "Answer only from the given text", "if unsure, say so", "state where the fact comes from".
- Double-check the critical yourself. Numbers, dates, quotes, legal/medical facts, links — always against the primary source. A confident tone is no guarantee.
- Mind the cut-off. Recent events — only via search/data, not "off the top of its head".
Takeaways
- The model is strong where a topic is plentiful online; weak on the rare, the recent and on precise details.
- Fluent ≠ true. Confidence is a style, not a check (see next-token prediction).
- Want reliability — bring the facts yourself or give it a tool, don't lean on the model's memory.
Source
Anthropic Academy course AI Capabilities and Limitations, section "Knowledge".