How to Prevent LLM Hallucinations in Production

A federal appeals court in the United States sanctioned a pair of attorneys in 2026 after finding more than two dozen fabricated citations in their filing. In another case that year, a lawyer submitted three separate briefs over five months containing fifteen invented case citations and eight invented quotations. The running public database of court decisions involving AI-hallucinated citations now holds well over a thousand entries and grows daily.

What makes these cases instructive is not that the models made things up. It is the shape of what they made up. In many of them the legal reasoning was sound and the case names were real; the docket numbers were fiction. Nobody catches a fake reference number sitting next to a genuine ruling.

This piece covers three things: why models invent answers instead of admitting they do not know, what this has already cost real companies, and which layers production systems use to contain it. We mapped the wider set of AI infrastructure decisions in our non-technical leader's guide; this one examines a single technical reality that touches all of them.

What exactly is an LLM hallucination?

It is a model presenting information that does not exist as though it were fact, in confident language. The technical description in the literature is "plausible falsehood". The crucial part: the model is not aware it is fabricating, because it never separates truth from invention and picks one. It produces both through the same mechanism.

That is the part companies find hardest to absorb. When a calculator returns a wrong number it is broken and you repair it. When a language model returns a wrong answer it is not broken, it is working as designed. Which makes "fix the hallucinations" a request with no technical meaning. The request that does have meaning is different: when a hallucination happens, does your system catch it?

Why does a model invent instead of saying it does not know?

Because guessing is rewarded. Research published by OpenAI scientists with Georgia Tech gives an unusually clean answer: models behave like students facing a hard exam question. Leaving it blank scores zero; guessing has some chance of scoring. Under those conditions guessing is always the profitable move.

Beyond the analogy there is a concrete finding. The same work shows that nine out of the ten major evaluation benchmarks in common use apply binary grading, and none of them award credit for "I don't know". A wrong answer and an honest admission of uncertainty score identically: zero. Models have been optimised against these measures for years.

Compare a model that correctly signals uncertainty with one that never does and always guesses. Under the scoring systems in use, the second one wins.

The second reason concerns the structure of the data. The same paper works through this: the proportion of facts appearing exactly once in the training data sets a floor under the hallucination rate. A person's birthday, a docket number, a reference code, anything with no pattern to infer from, is either memorised or invented. The fabricated case numbers in those court filings fall squarely into this category.

Would a smarter model fix it?

No, and sometimes it goes the other way. The numbers in OpenAI's own system card are uncomfortably clear: on a factual questions benchmark, the o1 model hallucinated at 16% and o3-mini at 14.8%, while the stronger reasoning models scored 33% for o3 and 48% for o4-mini.

The reason is straightforward. As reasoning capability rises, the model makes more claims. More correct claims, and more invented ones. That table alone explains why "let's upgrade the model version and the problem goes away" does not work.

What does work sits somewhere else entirely. Some newer models were trained to stay quiet when they do not know, and the effect was measured: of two models with nearly identical accuracy, the one taught to abstain hallucinated at 26% and the one that was not at 75%. The difference comes from behavioural design rather than intelligence. That should change how you evaluate vendors: instead of asking which model is smarter, ask which one shuts up when it is unsure.

The most dangerous hallucination is not the obviously wrong one

It is the partly correct one. The most insidious failure mode we see in the field is output where ninety percent is flawless and ten percent is invented. A human reads the correct part, treats the rest as verified by association, and the checking reflex switches off.

Academic work on the Whisper speech-to-text model measured a version of this: it inserts sentences that were never spoken in roughly 1% of transcriptions, particularly during silences, sometimes inventing medications that do not exist. One percent looks like a small number. It stops looking small next to the fact that a medical transcription tool built on the same technology has processed over seven million patient conversations.

A more troubling result appeared in a peer-reviewed medical journal. Researchers planted a single fake lab value or finding into each of 300 physician-written clinical vignettes. Models repeated that fabricated detail, and elaborated on it, in up to 83% of cases. In other words the model builds on a user's false premise rather than correcting it. Adding a cautionary prompt roughly halved the rate. It did not eliminate it.

What has this cost real companies?

Three ways: damages, reputation, and cancelled subscriptions. The precedent-setting case came out of Canada. A chatbot on Air Canada's website described a bereavement fare policy that did not exist. In tribunal, the company argued the chatbot should be treated as a separate legal entity responsible for its own statements. That argument was rejected and damages were awarded for negligent misrepresentation.

The sum was small, the precedent large: whatever the chatbot on your site says is your commitment. A second case shows where the boundary sits. A car dealership's bot was manipulated into agreeing to sell a $76,000 vehicle for one dollar; that was not held binding, because the bot had no authority to set prices. The line is clear enough. When the bot speaks within its authority, it binds the company.

Direct revenue loss has its own example. A software company's support bot invented a subscription rule ("your plan is limited to one device") to explain why users were being logged out. The real cause was a race condition in session handling. Once the invented rule spread on social media, developers working across multiple machines cancelled their subscriptions and the founder issued a public apology.

And the case that removes anyone's right to say it could not happen here: a consulting report delivered to an Australian government department, priced at 440,000 Australian dollars, was found to contain fabricated court citations and non-existent academic sources. The firm that produced it is one of the largest audit practices in the world, and it refunded part of the fee.

Hallucination engineering: the layers

There is no single fix; production systems stack layers that back each other up. What follows are the ones with either measured effect or sound logic, ordered roughly from highest to lowest return.

• Never let the model produce a verifiable value. Reference numbers, product codes, prices, stock levels, account numbers, statute sections: anything queryable should come from a system. Have the model write the query, not the answer. Most of the fabricated-citation cases would have been prevented by this rule alone.
• Enforce structured output. Ask for schema-bound output (JSON) instead of free text, so that anything produced can be validated programmatically before it goes anywhere.
• Give the model permission to say it does not know. Stop penalising abstention in your own internal evaluations and define an explicit confidence threshold in the prompt. The largest measured improvement comes from this layer.
• Ground answers with RAG, then verify that the cited source actually supports the claim (see the next section).
• Flag suspicious output with a second pass. Methods that generate multiple answers and measure disagreement at the meaning level (semantic entropy), or a second model acting as judge, both work. Get the order right for cost: cheap rule-based checks first, and only the small flagged subset goes to the expensive judge.
• Require human approval for anything irreversible. Payments, contracts, price commitments, legal text, anything published externally. A careful prompt halves the risk; the last line of defence is a person.
• Set temperature to zero for factual tasks. Creative generation and factual generation should run at different settings inside the same system.

Also draw an authority boundary around the bot. On binding subjects such as pricing, discounts, refunds and warranties, do not let it generate an answer at all; serve fixed text or system data. That closes hallucination and prompt injection with the same move.

Does RAG solve it?

It reduces it. It does not end it. The methodology behind the leading independent hallucination leaderboard tests the easiest possible version of RAG: the model is handed a document and asked only to summarise it, so the correct information is already in front of it. Even then, as of mid-2026 the best models produce source-unfaithful output somewhere between 1.8% and 3.3% of the time.

The second and less discussed problem: citing a source is not the same as being faithful to it. In an independent test of one search engine's AI summaries, more than half of the answers that were factually correct were not fully supported by the sources shown beneath them. The user sees a link and trusts it; the link does not verify the claim.

We covered how RAG is built and where it stalls in a separate guide. The lesson here is narrower: RAG is a hallucination reducer, not a hallucination solution. Do not ship it without a verification layer on top.

Is it worse outside English?

There is not enough evidence to claim a categorically higher rate; different measurement approaches disagree, and normalised comparisons do not track neatly with how much of a language is online. But for a company operating outside English, the real risk sits in two other places.

The first is content. Local regulations, court rulings, procedures and institution names appear rarely in training data, which puts them in exactly the single-occurrence category described above. Their fabrication risk is structurally high. Building a system that generates local legal or regulatory content without a verification layer means standing on the weakest ground in the riskiest area.

The second is tooling. Almost every automated hallucination-detection tool was built for English. Language-specific detection models are only now beginning to appear in academic work, and their accuracy remains modest. If you operate in another language, expect to build much of the verification layer yourself.

Who is liable?

Case law is still forming, but the direction is visible. The Air Canada decision binds nobody outside its jurisdiction, yet the underlying reasoning (a company is responsible for representations made on its own channels) has close equivalents in consumer protection and misrepresentation law almost everywhere.

The regulatory picture is moving too. In the European Union, the bulk of the obligations covering high-risk systems become applicable during 2026, which reaches any software company serving EU customers regardless of where it is based. Several other jurisdictions have draft AI legislation in progress. Treat this as general information rather than legal advice.

One caution about this article itself. While researching it, we could not trace a significant share of the striking statistics in circulation ("hallucinations cost the world X billion dollars", "RAG reduces hallucinations by Y percent") to any primary source, so we left them out. Filling an article about hallucination with unverified numbers would be committing the error it describes. Treat that as a general reflex for any AI statistic you read: click the source.

Frequently asked questions

Can hallucination be eliminated completely?

No. Both theoretical work and measurement say it cannot reach zero. It can, however, be pushed down to a negligible level through system design. The right question is not "is it zero" but "what happens when it is wrong".

Would fine-tuning on our own data fix it?

No. Fine-tuning corrects tone and terminology; it does not stop factual invention, and trying to teach a model facts it does not know can increase hallucination. We worked through the details in our decision tree for choosing a method.

Do we have to check every output by hand?

Not all of them. Verifiable fields should be checked automatically by the system, suspicious outputs should be flagged automatically, and human approval should trigger only on irreversible actions. Reading every output manually is neither feasible nor necessary.

Is the company liable if our chatbot gives wrong information?

On the existing precedent, yes, when the bot spoke within its authority. The practical mitigation is to prevent the bot from generating answers on binding subjects at all.

We are a small business. Do we need all these layers?

Not all of them. Start with two: pull verifiable data from your systems rather than the model, and keep human approval on anything leaving the building. Most incidents are closed by those two alone.

So what should you actually do?

• Design on the assumption that the system will be wrong and the error will be caught, not that it will never be wrong. Your acceptance criterion is not zero errors; it is no uncaught errors.
• List every verifiable field in your output and fill those fields from your own systems rather than the model.
• Write down the irreversible actions (payments, commitments, legal text, external publication) and require human approval on all of them.
• When evaluating models, test the ability to abstain. Measure what a model declines to answer, not only what it gets right.
• If you work with sparse, local or specialist information, plan the verification layer from the start rather than treating it as a feature to add later.

None of the incidents above were technical bad luck; they were predictable outcomes. A fabricated reference number could have been checked against a database in milliseconds, and was not. When you put a language model into your business, the question worth asking is not how good the model is. It is which part of its output can be verified by machine, and who is going to build that check.