The Problem Is Structural
The discussion about artificial intelligence in the legal profession usually revolves around one nightmare image: the fictitious court decision in a pleading. The image is justified, but it is too narrow. Anyone who regards hallucinations as a temporary teething problem that will be resolved by the next model generation or the right tool is underestimating the issue. Hallucinations are not a bug that gets programmed away. They are a structural property of the underlying technology.
Three Courts, the Same Lesson
Three German courts addressed AI hallucinations in quick succession in 2025. The LG Frankfurt am Main (25 September 2025, 2-13 S 56/24) found that counsel had submitted fabricated BGH citations — non-existent source, date and case number. The AG Köln (2 July 2025, 312 F 130/25) objected to a brief in which all citations from a certain page onwards were invented. The OLG Celle (29 April 2025, 5 U 1/25) had previously addressed the same phenomenon. Three courts within a few months is not an outlier — it is a pattern.
Internationally the picture is starker. Attorney Damien Charlotin maintains a public database of documented cases; it currently lists over 1,600 cases worldwide, more than 1,400 from the United States alone.
Why No Tool Eliminates the Problem
A general language model without database access hallucinates on legal questions in between 58 and 88 percent of queries. The standard fix is RAG (Retrieval-Augmented Generation): a search step retrieves real documents and presents them to the model before it generates text. That reduces the error rate considerably. But a Stanford study of the leading commercial legal RAG systems found error rates of 17 percent for the best-performing product (Lexis+ AI) and 33 percent for Westlaw AI. One in three answers, on average.
The reason lies deeper than product quality. A language model generates text by predicting the most plausible next word. It has no internal concept of "true" or "false". Theoretical work (Kalai and Vempala, 2024) has shown that calibrated language models must hallucinate at a certain rate — regardless of architecture, training data or size. The hallucination is inherent to the mechanism, not a fixable defect.
RAG therefore shifts rather than solves the problem. It largely replaces the invented citation with a subtler class of error: the system retrieves the correct decision and still misrepresents its content. The source exists, is checkable, the case number is right — only the proposition attributed to it does not hold. The most dangerous variant of this error is misattribution: a real decision, correctly retrieved, but with a statement attached to it that is not in the judgment. The decision number disarms verification reflexes.
The Real Defect: Research That Stops Too Early
All this concerns errors of generation. The more serious error is one of completeness: the system stops searching too early. A hallucinated decision can be exposed by checking. A counter-decision that was never found leaves no trace.
An agent starting from a thesis formulates its queries to confirm that thesis. What it does not look for, it does not find. It is the machine equivalent of confirmation bias — without the human corrective of doubt. A found answer is the stop signal. That is exactly where the machine stops and where the careful lawyer would only be beginning: at the question, what speaks against this?
Professional Responsibility
The legal framework is clear. BRAK and DAV have confirmed that AI-generated work products cannot be trusted the way one trusts an experienced senior colleague. The DAV requires all citations to be checked in full, even in the absence of any specific hallucination suspicion. Under the general duty of care in § 43 BRAO, the lawyer is responsible for the quality of all work products regardless of the tool used to produce them.
The only point at which the chain closes reliably is the legal judgment of the human who signs the document. This responsibility cannot be delegated — and will not be for the foreseeable future.