Practical Tips & Safety12 min read

AI Hallucinations

When AI makes things up with a straight face — and a lawyer learns the hard way
scope:Core Conceptsdifficulty:Beginner

The Lawyer Who Trusted ChatGPT

In June 2023, a New York lawyer named Steven Schwartz filed a legal brief in federal court. The brief cited six court cases as precedents to support his argument. The citations looked perfect — case names, dates, court decisions, page numbers. Very professional.

There was just one problem: none of the cases existed.

Not a single one. Schwartz had used ChatGPT to research case law, and the AI had confidently generated six completely fabricated court cases, complete with fake judges, fake rulings, and fake legal reasoning. When the opposing lawyers couldn't find the cases, they informed the judge. The judge was, to put it mildly, not amused.

Schwartz was sanctioned (fined) and his case became international news — not for the legal issue, but as a cautionary tale about AI hallucinations.

This wasn't a bug. It wasn't a glitch. It was a fundamental feature of how large language models work. And understanding it is one of the most important things you can learn about AI.

What Exactly is a Hallucination?

An AI hallucination is when an AI model generates information that is false, fabricated, or nonsensical — but presents it with complete confidence, as if it were true.

The key characteristics:

  • Confident delivery. The AI doesn't say "I'm not sure" or "I made this up." It states the false information with the same tone and certainty as true information.
  • Plausible sounding. The hallucinated content follows the right patterns — proper grammar, correct format, realistic-sounding names and dates. It looks right.
  • Unflagged. The AI gives no warning that this particular piece of information might be unreliable. You get no asterisk, no disclaimer, no "I'm guessing here."

Think of it like a student who doesn't know the answer to an exam question but writes a beautifully structured, articulate, completely wrong response. The handwriting is neat, the grammar is perfect, and the argument flows logically — but the facts are fiction.

Note: Why the name "hallucination"? The term borrows from psychology, where hallucinations are perceptions that feel real but aren't. AI hallucinations work similarly — the model "perceives" patterns in its training data that lead it to generate content that feels factual but doesn't correspond to reality. Some researchers prefer the term "confabulation" (making up stories without intent to deceive), but "hallucination" has stuck.

Why Do Hallucinations Happen?

To understand hallucinations, you need to understand what LLMs actually do. An LLM like ChatGPT is, at its core, a next-token prediction machine. Given a sequence of words, it predicts the most likely next word. Then the next. Then the next.

Here's the critical insight: predicting the most likely next word is not the same as knowing what's true.

Let's break down the specific reasons:

1. Pattern Matching, Not Knowledge

An LLM doesn't have a database of facts that it looks up. It has learned statistical patterns about which words tend to follow other words. When you ask "What is the capital of France?" it doesn't look up the answer — it recognizes that in its training data, the word "Paris" almost always follows that question pattern.

This works great for well-known facts. But when the patterns are ambiguous, rare, or the question is about something the model hasn't seen much training data for, those statistical patterns can lead to plausible-sounding but wrong answers.

2. No Concept of "I Don't Know"

LLMs are trained to always generate a response. They're completion machines — you give them a prompt, they complete it. They're not naturally trained to say "I don't know" or "I'm not sure about this." The model will always produce the most statistically likely continuation, even when the honest answer would be uncertainty.

3. Training Data Isn't Truth-Labeled

LLMs are trained on massive amounts of internet text — which includes correct information, incorrect information, myths, fiction, jokes, and lies. The model has no reliable way to distinguish between "information that appeared in a textbook" and "information that appeared in a Reddit comment."

4. The Blending Problem

LLMs can mix and match real details into false combinations. It might know that there's a "Journal of Cognitive Science" and a researcher named "Sarah Mitchell" — but combine them into a citation that never existed. Each individual piece might be real; the combination is fabricated.

Demonstrating Why Hallucinations Happen

import random


def next_token_prediction(prompt, knowledge_base):
    """Simplified model: predicts next tokens based on patterns.
    Shows how hallucinations emerge from pattern matching."""
    
    # The model has learned these PATTERNS (not facts):
    patterns = {
        "capital of France": "Paris",          # Correct - seen 1M+ times
        "capital of Australia": "Sydney",       # WRONG - common misconception!
        "wrote Romeo and Juliet": "Shakespeare", # Correct
        "invented the telephone": "Alexander Graham Bell",  # Debatable!
    }
    
    for pattern, completion in patterns.items():
        if pattern in prompt.lower():
            return completion
    
    # When no strong pattern exists, the model STILL generates something
    # This is where hallucinations come from!
    fake_but_plausible = [
        "According to a 2021 study by Dr. James Chen...",
        "The landmark case of Thompson v. Miller (2019)...",
        "Research published in Nature (Vol. 47, p. 312)...",
    ]
    return random.choice(fake_but_plausible)  # Confident, but made up!


# Well-known fact → correct
print("Q: What is the capital of France?")
print(f"A: {next_token_prediction('capital of France', {})}")
print("(Correct! Strong pattern in training data)\n")


# Common misconception → hallucination
print("Q: What is the capital of Australia?")
print(f"A: {next_token_prediction('capital of Australia', {})}")
print("(WRONG! It's Canberra. Sydney appears more often, so the model picks it)\n")


# Obscure question → fabricated answer
print("Q: What did the court rule in AI copyright cases?")
print(f"A: {next_token_prediction('AI copyright ruling details', {})}")
print("(HALLUCINATION! No strong pattern, so it generates plausible-sounding fiction)")
Output
Q: What is the capital of France?
A: Paris
(Correct! Strong pattern in training data)

Q: What is the capital of Australia?
A: Sydney
(WRONG! It's Canberra. Sydney appears more often, so the model picks it)

Q: What did the court rule in AI copyright cases?
A: According to a 2021 study by Dr. James Chen...
(HALLUCINATION! No strong pattern, so it generates plausible-sounding fiction)

Famous Hallucination Incidents

The lawyer case isn't an isolated incident. Hallucinations have caused real problems across industries:

  • The Lawyer Brief (2023) — Steven Schwartz cited 6 fabricated court cases generated by ChatGPT. He was sanctioned by the court.
  • Google Bard's Launch (2023) — During Google's live demo of Bard, the AI incorrectly stated that the James Webb Space Telescope took the first pictures of exoplanets. It didn't. Google's stock dropped $100 billion in a single day.
  • Medical Misinformation — Studies have found AI chatbots generating false medical advice, including incorrect drug dosages and fabricated drug interactions. Potentially life-threatening.
  • Fake Academic Papers — AI has been caught generating entire fake research papers with fabricated data, fake authors, and made-up experimental results that passed initial peer review.
  • News Articles — AI-generated news articles have included fabricated quotes attributed to real people, invented events, and wrong dates for real events.

The Hallucination Spectrum

Not all hallucinations are created equal. They range from minor to catastrophic:

  • Minor inaccuracies — Getting a date slightly wrong ("founded in 1997" when it was 1998). Annoying but low-risk.
  • Fabricated details — Inventing a specific statistic ("studies show 73% of users prefer..." when no such study exists). Misleading.
  • Fabricated sources — Generating fake citations, fake research papers, or fake court cases. Dangerous in professional contexts.
  • Contradicting reality — Stating something completely opposite to established fact. Dangerous if the reader trusts the AI.
  • Fabricated expertise — Providing detailed medical, legal, or financial advice based on fabricated knowledge. Potentially catastrophic.

How to Spot Hallucinations

Developing a "hallucination radar" is one of the most valuable skills in the AI age. Here's how:

Red Flags to Watch For

  • Very specific numbers or statistics — If the AI quotes a precise percentage, dollar amount, or measurement, verify it. Hallucinated stats often sound suspiciously specific ("73.2% of respondents" rather than "about three-quarters").
  • Specific citations — Author names, journal titles, case names, dates. These are among the most commonly hallucinated content types.
  • Uncommon or niche topics — The less popular a topic is, the less training data exists, and the higher the hallucination risk.
  • Biographical details — AI frequently invents or mixes up details about real people, especially those who aren't very famous.
  • Recent events — LLMs have training cutoff dates. Anything after that date is either unknown to the model or based on limited data, increasing hallucination risk.

Verification Strategies

  • Cross-reference. Never rely on a single AI source. Check the AI's claims against reliable sources — official websites, academic databases, established encyclopedias.
  • Ask for sources, then verify them. If the AI cites a study or case, search for it independently. If you can't find it, it might not exist.
  • Use grounded AI tools. Tools like NotebookLM are "grounded" in specific sources, dramatically reducing hallucination risk.
  • Apply the "too good" test. If the AI's answer perfectly supports your argument with a specific stat or quote, be extra skeptical. Hallucinations often tell you exactly what you want to hear.
  • Check the basics. If the AI mentions a person, verify they exist. If it mentions an organization, check it's real. If it quotes a law, look it up.
Note: The Confidence Trap: The most dangerous thing about hallucinations is that AI doesn't hedge. A human expert will say "I think" or "if I remember correctly." An AI will state fabricated information with the same absolute confidence as verified facts. This is why you should treat AI output like a Wikipedia article — a great starting point for research, but never the final word.

Why Hallucinations Are So Hard to Fix

If hallucinations are such a big problem, why don't AI companies just fix them? Because it's incredibly difficult. Here's why:

  • It's baked into the architecture. LLMs are next-token predictors. They're designed to produce fluent, plausible text — not verified truth. You'd need to fundamentally change how these models work.
  • Truth is hard to define. What counts as "true"? Scientific consensus changes. Historical interpretations evolve. Even "facts" can be contested. Building a system that verifies truth is arguably harder than building the language model itself.
  • Scale of knowledge. Verifying every claim would require checking against the entire body of human knowledge in real time. That's computationally impractical.
  • The long tail. AI companies can reduce hallucinations on common topics, but there's a long tail of obscure, niche, and emerging topics where hallucination remains common.

Current approaches to reducing hallucinations include:

  • RLHF (Reinforcement Learning from Human Feedback) — Training models to be more cautious and say "I don't know"
  • RAG (Retrieval-Augmented Generation) — Having the model look up information before answering, rather than relying on memory alone
  • Grounding — Connecting the model to verified knowledge bases (like NotebookLM does with your documents)
  • Chain-of-thought reasoning — Making the model show its reasoning step by step, which can reduce errors
  • Confidence calibration — Training models to express uncertainty when they're less sure
Challenge

Quick check

Why do LLMs hallucinate?

Continue reading