Extrinsic Hallucinations in Large Language Models: Causes and Detection

Large language models (LLMs) have made remarkable strides in generating human-like text, but they still struggle with a critical issue: hallucination. This phenomenon occurs when models produce content that is fabricated, inconsistent, or nonsensical. While the term 'hallucination' has been used broadly to describe various model mistakes, recent research focuses on a specific type: extrinsic hallucination.

What are extrinsic hallucinations?

Extrinsic hallucinations happen when an LLM generates output that isn't grounded in either the provided context or its pre-training dataset (which serves as a proxy for world knowledge). This is distinct from in-context hallucinations, where the model's output conflicts with information given in the prompt or context.

To avoid extrinsic hallucinations, LLMs need to accomplish two key tasks:

1. Produce factual information that aligns with real-world knowledge
2. Acknowledge when they don't know something, rather than fabricating an answer

The roots of hallucination

Hallucinations can arise from issues at both the pre-training and fine-tuning stages of model development.

Pre-training data challenges

The massive datasets used to train LLMs, often scraped from the public internet, inevitably contain outdated, missing, or incorrect information. Models may incorrectly memorize this flawed data, leading to mistakes in their outputs.

Fine-tuning complications

While fine-tuning is commonly used to improve specific model capabilities, it can inadvertently introduce new knowledge. Recent research by Gekhman et al. (2024) investigated whether fine-tuning on new information encourages hallucinations. Their findings reveal a complex dynamic:

• LLMs learn new knowledge more slowly than information consistent with their pre-existing knowledge.
• Once new knowledge is learned, it can increase the model's tendency to hallucinate.
• The best performance occurs when the model learns most of the 'known' training examples but only a few 'unknown' ones.
• Interestingly, examples that are 'maybe known' to the model contribute more to overall performance than those that are 'highly known'.

These results highlight the risks associated with using supervised fine-tuning to update an LLM's knowledge base.

Detecting and measuring hallucinations

As hallucinations pose a significant challenge for LLM reliability, researchers have developed various methods to detect and quantify them.

Retrieval-augmented evaluation

Lee et al. (2022) introduced the FactualityPrompt benchmark, which uses Wikipedia as a knowledge base for grounding factual claims. This approach employs two key metrics:

1. Hallucination Named Entity (NE) errors: Measures the fraction of named entities in the model output that don't appear in the ground truth document.
2. Entailment ratios: Calculates the fraction of generated sentences that are relevant to paired Wikipedia sentences, as determined by an entailment model.

Lower NE errors and higher entailment ratios indicate better factuality. Importantly, these metrics correlate well with human annotations, and larger models tend to perform better on this benchmark.

FActScore: Atomic fact validation

Min et al. (2023) proposed FActScore, which breaks down long-form text into atomic facts and validates each against a knowledge base. This method calculates the precision of sentences supported by the knowledge source for each model generation. Key findings include:

• Rarer entities and facts mentioned later in the generation have higher error rates.
• Using retrieval to ground model generations significantly reduces hallucinations.
• The best factuality estimator may vary depending on the specific model being evaluated.

SAFE: Search-augmented factuality evaluation

Wei et al. (2024) introduced SAFE (Search-Augmented Factuality Evaluator), which uses a language model as an agent to iteratively query Google Search and reason about whether the results support a given fact. This approach outperforms human annotators in factuality checking, achieving a 72% agreement rate and winning 76% of disagreements, while being 20 times more cost-effective.

SAFE employs an F1 @ K metric, balancing both precision (factuality) and recall (completeness) in long-form responses.

The path forward

As LLMs continue to evolve, addressing hallucinations remains a critical challenge. The research highlighted here demonstrates the complexity of the problem and the innovative approaches being developed to detect and mitigate unfaithful outputs. By understanding the causes of hallucinations and improving our ability to measure them, we can work towards more reliable and trustworthy AI language models.

However, it's important to note that completely eliminating hallucinations may be an unrealistic goal. Instead, the focus should be on minimizing their occurrence and developing robust methods for identifying when they do happen. This will allow users to make informed decisions about when and how to rely on LLM-generated content.