What are AI hallucinations – and why are they a compliance risk?

What is an AI hallucination?

An AI hallucination (also "confabulation") is a convincingly worded but factually incorrect or entirely invented output of an AI model. The term has no legal definition; it describes a technical behavior of large language models. Because these models generate answers based on probability and do not carry out their own fact-checking, they can invent sources, figures, or quotations that never existed – and do so in the same self-assured tone as with correct information.

Why hallucinations arise

Based on its training data, a language model predicts the most likely next word in each case. It does not "understand" content in the human sense and does not have verified factual knowledge. If reliable information is missing or a question is very specific, the model fills the gap with a plausible-sounding invention. Hallucinations are therefore not a rare slip but a system-inherent property. How often they occur depends heavily on the model and field of application: a Stanford University study found error rates of 69 to 88 percent, depending on the model, for specific legal questions (as of 2024).

Why AI hallucinations are a compliance risk

For companies, hallucinations are more than a quality problem. Anyone who adopts an incorrect AI output without checking it can give customers incorrect information, spread false legal or technical statements, or make decisions on an invented basis. A vivid example: if a clerk adopts a court decision or deadline invented by the AI into a customer letter without checking it, it is not the AI system that is liable but the company. Hallucinations thus touch on questions of due care and liability. The EU AI Act addresses this, among other things, with the obligation of human oversight for high-risk systems (Art. 14 EU AI Act) and requirements for accuracy and robustness (Art. 15). The European Commission expressly names hallucinations as one of the risks about which employees must be informed as part of AI literacy under Art. 4.

What companies can do about it

Hallucinations cannot be completely prevented, but they can be effectively contained. Proven measures include the four-eyes principle for important outputs, requiring sources, the use of methods such as retrieval-augmented generation (which have the model look things up in verified documents), and clear internal rules on what AI may be used for. The most important protective measure, however, remains the trained human: only those who know that AI can be convincingly wrong check its results critically. Conveying precisely this awareness is a core objective of every AI literacy training – the AI glossary provides an overview of the basic terms. The Artificial Intelligence topic hub brings together further articles on the topic.

Can AI hallucinations be avoided?
They cannot be avoided completely because they are inherent in the way language models work. However, they can be significantly reduced – for example through a requirement to cite sources, methods such as retrieval-augmented generation, and above all through critical human review of important outputs.

Why are hallucinations a legal risk?
Anyone who adopts incorrect AI outputs without checking them can spread incorrect information or make decisions on an invented basis. This touches on duties of care and liability. The EU AI Act therefore requires human oversight for high-risk systems (Art. 14) and requires employees to be informed about risks such as hallucinations (Art. 4).

• German Bundestag, Research Services, „Bias und Halluzinationen von KI-Systemen“ (WD 5-082/25), bundestag.de. As of 19 Nov 2025.
• Stanford University (RegLab/HAI), "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models". As of 2024.
• Regulation (EU) 2024/1689 (EU AI Act) – Art. 4, 14, 15; EUR-Lex. As of July 2026.
• European Commission, "AI literacy – questions and answers". As of 19 Nov 2025.