Why AI chat assistants sometimes make things up

If you use chat assistants, you have probably seen it: a clear, confident answer that turns out to contain a made-up fact, a quote nobody said or a source that does not exist. It feels like lying. It is something different, and once you see how these systems work, the pattern becomes much easier to predict.

What a language model actually does

Chat assistants are built on large language models. During training, such a model reads enormous amounts of text and learns one skill extremely well: predicting which piece of text is likely to come next.

When you ask a question, the assistant writes its answer one small piece at a time. At each step it picks a continuation that fits your question and everything written so far. Repeat that thousands of times and you get a fluent paragraph.

1. Your question is turned into numbers that the model can work with.
2. The model scores many possible next pieces of text, based on patterns from training.
3. One piece is chosen, often with a little randomness so answers do not sound identical.
4. It is added to the answer and the whole process repeats for the next piece.
5. Optional extras: some products search the web or your files first and hand the results to the model as extra context.

Notice what is missing from that description. There is no step where the model looks up a fact in a verified database and copies it. Search and document tools help, but the core engine is a predictor of plausible text.

Why plausible is not the same as true

Most of the time, the most plausible continuation is also correct. Ask for the capital of Canada and text about Ottawa is overwhelmingly likely, so the model writes Ottawa.

Problems start where the training text is thin or mixed. Ask about a small local business, an obscure court case or a niche research paper, and the model may have seen little or nothing about it. It still has to produce text, so it builds something that looks like a typical answer: a name that sounds right, a date in a sensible range, a reference with a realistic title. The result reads smoothly and is simply wrong.

People call this a hallucination. The word is a metaphor. The model is not seeing things. It is doing exactly what it was trained to do, in a situation where that is not enough.

The situations where it happens most

• Specific details: exact quotes, page numbers, statistics, phone numbers and web addresses.
• Rare topics: small places, little-known people, and events after the model's training data was collected.
• Reference lists: books, articles or legal cases, where the model can produce realistic titles that do not exist.
• Loaded questions: ask "why did X happen?" and the model may explain an event that never took place instead of questioning the premise.
• Long chains of reasoning: one small error early on can carry through every later step.

Why it sounds so confident

Human writers usually show doubt with words like "I think" or "probably". A language model learned mostly from text written with confidence: encyclopedias, news articles, manuals. So its default tone is confident too, whether or not the content is reliable.

Developers work on this. Many assistants are now trained to say when they are unsure, to decline questions they cannot answer and to show the sources they used. These steps reduce the problem. None of them removes it completely, because the underlying prediction mechanism stays the same.

A useful rule of thumb: the tone of an answer tells you nothing about its accuracy. A hesitant answer can be right and a polished one can be invented.

How to check an answer

You do not need to stop using chat assistants. You need to match your checking to the stakes.

1. Low stakes, light checking. For brainstorming or rewording an email, a mistake costs little and you will notice it anyway.
2. Facts you will repeat. Ask the assistant for its source, then open that source yourself. If it does not exist, or says something different, drop the claim.
3. Health, legal, tax or money questions. Use the assistant to learn the vocabulary, then confirm with an official source, such as a Government of Canada page, or with a qualified professional.
4. Numbers, names and quotes. Check these every time. They are the most common places for invented details.

It also helps to phrase questions openly. "Is there research on X?" gives the model room to say no. "Summarize the research on X" pushes it to produce something whether or not that research exists.

The takeaway

A chat assistant is a remarkable writer and an unreliable witness. It predicts what a good answer would look like, which is usually close to the truth and sometimes far from it. Once you know that, you can use it for what it does well, such as drafting, explaining and summarizing, and check it carefully wherever accuracy matters.

This article is general educational content. It is not financial, legal or technical advice. Check important decisions with an official source or a qualified professional.