4. Checking the outcome

The trickiest error a language model makes is the answer that reads fluently and yet makes no sense at all. No warning light comes on, and the text does not sound uncertain. Anyone who sends out AI output under their own name is responsible for every figure, every source and every sentence in it. This chapter is about the checking that goes with that.

Why it happens

A language model continually predicts what is likely to follow on from what is already there. It has no list of facts to look something up in and see whether it is correct. Ask it for the author of a book the model does not know, and "I don't know" is a statistically unusual answer to such a question. A plausible-sounding name fits the pattern better. The result is a fluent sentence with an invented core. In the field this is called a hallucination, and the term covers it reasonably well: the model sees something that is not there.

The same mechanism lies behind the weak spots in numerical work. Predicting is not the same as calculating. Small sums usually go well, because those combinations appear thousands of times in the training data. But with a discount on a discount, VAT on an amount that already includes a margin, working days between two dates or "give me exactly 150 words", the answer becomes a plausible guess in numerical form. It is often close and just not right.

The signals

Five things in an answer deserve extra suspicion.

• Suspicious precision. Exact case numbers, page references or percentages to two decimal places appear without you having supplied a source yourself.
• Familiar name, unfamiliar work. The model links an existing author or organisation to a report you cannot find anywhere.
• Slick certainty on a niche question. The more specialised your question, the thinner the material the model was trained on. A fluent answer about an obscure Dutch regulation is suspect.
• Links you cannot open yourself. Invented web addresses are a classic, certainly when the model is not searching the web live.
• Answers that confirm exactly what you suggested. If your question contains an assumption, the model will happily build on it, even if that assumption is incorrect.

Proofreading is not checking

When proofreading, you look out for what sounds odd, and that is exactly where the problem lies: invented passages do not sound odd. They often read more pleasantly than the truth, because the model chooses the most common phrasing. A fact check therefore works differently. You do not read by feel but by category, and every flagged claim gets an outcome: it is correct, it is incorrect, or it cannot be traced. That third outcome is not a halfway version of the first: a claim you cannot verify is one you remove or rewrite.

• Numbers. Percentages, amounts, years, quantities and market figures. The model fills in plausible values here effortlessly.
• Proper names. People, organisations, job titles, product names. Things go wrong particularly with Dutch names and small organisations.
• References. Legal provisions, collective agreement clauses, standards, case law, page numbers and web addresses. The more precise the reference, the sooner you should check it.
• Quotations. Every sentence in quotation marks that is attributed to someone. An invented quote is both factually wrong and damaging to reputation.
• Causal claims. "Research shows that X leads to Y." The phrasing suggests evidence that is not always there.

Where to check

The source you check against must not come from the same stable as the claim. Asking the model whether its answer is correct mainly produces agreement, and a second AI assistant trained on similar material makes the same error with the same confidence. What is useful is the place the fact comes from: wetten.overheid.nl for legislation, CBS for figures, the annual report of the organisation in question, the original article behind a quotation. You use a search engine to reach that place, not to read a summary of it.

This still applies even when the model searches the web itself. You are then shown links, and those links are real. What the summary of that link says is still generated. Opening the source remains your job.

How much checking is enough

Verifying everything often takes more time than the answer saved you, which defeats the purpose. The measure is the potential damage: the more visible the output and the harder it is to undo, the more thoroughly you check. A list of angles for an internal brainstorm needs a glance. A letter to a client, a grant application, a quotation or a text on your website needs all five categories, line by line.

Summary

• A language model predicts what is likely to follow; it checks nothing, so it invents without noticing.
• Numerical work is weak for the same reason: the answer is often close and just not right.
• Signals: suspicious precision, a familiar name with unfamiliar work, smooth certainty on a niche question, links that do not open, and answers that confirm your own assumption.
• Ask neutral questions, because an assumption in your question comes back as confirmation.
• Proofreading does not work: invented passages do not sound odd. Read by category.
• Always mark figures, proper names, references, quotations and causal claims.
• Check at the original source, never with the model or with a second AI assistant.
• The more visible and irreversible the output, the more thorough the check.