Ask a chatbot a question it cannot answer and it will often answer anyway, fluently and with apparent certainty. The information may be invented: a quotation nobody said, a study nobody wrote, a legal case that was never decided. Researchers call these outputs hallucinations, and they are among the most important limitations of today’s AI systems. Understanding why they happen is the first step toward using these tools safely.
What a hallucination looks like
Consider a hypothetical student who asks an AI assistant who first proposed gravitational lensing, the bending of light by gravity. A plausible wrong answer would credit Isaac Newton and his Principia. In fact the modern prediction comes from Einstein’s general relativity, and the first observational confirmation came in 1919. The fabricated answer is wrong but sounds authoritative, which is what makes it dangerous.
A second illustrative case: ask a model to list the symptoms of an invented disease, and it may produce a serious-sounding list of fatigue, memory loss and joint pain. The question itself supplied the premise, and the model completed the pattern. The same thing happens with citations. Asked for a reference to a study by a named researcher on a given topic, a model may generate an author, year, journal and page numbers for a paper that does not exist.
Why models do this
Large language models do not look facts up in a database and they do not possess understanding in the human sense. They are trained on enormous amounts of text to predict which words are likely to come next. When a question has a well-documented answer that appears often in the training data, prediction and truth usually line up. When it does not, the model still produces the most statistically plausible continuation, and a plausible continuation of “give me a citation” is something shaped like a citation.
This is also why the tone rarely changes. A human who does not know an answer hesitates or says so. A language model, unless specifically trained and prompted to express uncertainty, generates text with the same fluency whether the content is accurate or invented. Terms such as “clearly” and “certainly” are simply common in the kind of explanatory text the model learned from. The confidence is a feature of the style, not a measure of reliability.
Real consequences
The best-known case is Mata v. Avianca, decided in the U.S. District Court for the Southern District of New York in 2023. Lawyers for the plaintiff filed a brief citing several judicial decisions that did not exist. They had used ChatGPT for research, and when opposing counsel and the court could not locate the cases, the lawyers initially produced further fabricated excerpts that the chatbot had generated. Judge P. Kevin Castel fined the attorneys and their firm $5,000 and required them to notify the judges falsely identified as authors of the fake opinions. The court’s order made clear that relying on a tool is not a defense for filing false material.
Similar problems have surfaced in journalism, academic publishing and medicine, where fabricated references or incorrect details have slipped into published work. In each case the failure was the same: a confident output accepted without verification.
Reducing the risk
The problem is being worked on from the technical side. Developers now train models to decline or qualify answers more often, connect them to search engines or document stores so that responses can be grounded in retrievable sources, and build reasoning steps that let a model check its own output before replying. Error rates have fallen, but no current system is free of hallucinations, and model developers say so in their own documentation.
The rest of the solution falls to users. Practical habits help:
- Treat any specific claim, number, quotation or citation from a chatbot as unverified until you have checked it against a primary source.
- Ask the model for its sources and then open them. If a link or reference cannot be found, assume it was invented.
- Be especially skeptical when a question contains a false premise, because models tend to accept the premise and elaborate on it.
- Use tools that ground their answers in retrieved documents for research tasks, and prefer those that show which passages support each claim.
- In professional settings such as law, medicine and finance, keep a human review step that would catch a fabricated reference before it reaches a client, a patient or a court.
Hallucinations are not a bug that will be patched out in the next release. They follow from how language models are built. Treated as powerful drafting and research aids whose output must be checked, these systems are useful. Treated as oracles, they will eventually lead a user into the same position as the lawyers in Mata v. Avianca: defending a confident statement that was never true.
