Abstract

We study hallucination rates across 12,000 entities stratified by frequency, type, and recency. Hallucination follows a power law $H(f) \propto f^{-0.43}$, but is modulated by entity type (persons 2.3x more than locations) and temporal recency (4.1x increase for post-cutoff entities). We introduce the Entity Hallucination Profile (EHP) for targeted mitigation.

1. Introduction

Hallucination—the generation of factually incorrect content—is a central challenge for deploying LLMs in high-stakes applications [1]. A widely held intuition is that hallucination correlates inversely with entity frequency in the training data: models know more about common entities and hallucinate more about rare ones [2].

While this intuition is directionally correct, it is overly simplistic. Not all rare entities hallucinate equally, and the nature of hallucination varies systematically across the frequency spectrum. Understanding these patterns is crucial for developing targeted mitigation strategies.

2. Entity Stratification

2.1 Frequency Bins

We use Wikipedia page views (2023 annual) as a proxy for training data frequency:

2.2 Entity Types

Each frequency bin is balanced across four types:

• Person (P): Named individuals
• Location (L): Geographic entities
• Organization (O): Companies, institutions
• Event (E): Historical events, conferences

2.3 Temporal Recency

Entities are additionally tagged as:

• Stable (S): Core facts unchanged in 5+ years
• Dynamic (D): Key facts changed in last 2 years
• Post-cutoff (C): Key facts changed after the model's knowledge cutoff

3. Experimental Setup

For each entity, we generate 5 factual questions (verified by human annotators). Models answer under greedy decoding with no retrieval augmentation.

Hallucination is judged by three annotators per question ($\kappa = 0.81$) as: Correct, Hallucinated (confident false claim), Abstained (model declines to answer), or Hedged (uncertain but incorrect).

4. Results

4.1 Frequency-Hallucination Power Law

Across all models and entity types:

Fitting $H(f) = af^{-b}$: $b = 0.43 \pm 0.04$, $R^2 = 0.987$.

4.2 Entity Type Modulation

Hallucination rate by type, controlling for frequency (F4 bin):

Locations hallucinate least because their core facts (coordinates, country, population) are more constrained and more frequently repeated in training data.

4.3 Temporal Recency Effect

Post-cutoff entities exhibit near-total hallucination because models generate outdated information with high confidence.

4.4 Hallucination Typology

For common entities, models are confident even when wrong (they "know" the entity but get specific facts wrong). For rare entities, models confuse similar entities (they mix up the entity with a more common neighbor).

4.5 Model Comparison

All models follow similar power-law exponents ($b \approx 0.43$), suggesting this reflects a fundamental property of autoregressive pretraining rather than architecture-specific behavior.

5. The Entity Hallucination Profile (EHP)

We define EHP as a three-dimensional risk map: $\text{EHP}(f, t, r) \rightarrow [0, 1]$, where $f$ is frequency, $t$ is type, and $r$ is recency. Given a new entity, its predicted hallucination rate is:

$$\hat{H} = a \cdot f^{-b} \cdot \gamma_t \cdot \delta_r$$

where $\gamma_t \in {1.0, 1.61, 1.92, 2.31}$ for ${$L, E, O, P$}$ and $\delta_r \in {1.0, 1.81, 4.09}$ for ${$S, D, C$}$.

Validation on a held-out set of 2,000 entities: predicted vs. actual hallucination rate correlation $r = 0.91$, MAE = 4.2 percentage points.

6. Discussion

6.1 Targeted Mitigation

EHP enables targeted retrieval augmentation: instead of applying RAG uniformly (which adds latency and cost), systems can selectively augment queries about entities in high-risk regions of the EHP space (rare + person + post-cutoff).

6.2 Limitations

1. Proxy frequency: Wikipedia page views are an imperfect proxy for training data frequency.
2. English-only: Entity frequency distributions differ across languages.
3. Binary hallucination: We don't grade hallucination severity.
4. Static analysis: Models may reduce hallucination through retrieval or self-verification at inference time.
5. No causal mechanism: We characterize the relationship statistically but don't explain why persons hallucinate more than locations at the representation level.

7. Conclusion

Hallucination rates follow a power law with entity frequency ($b = 0.43$), are modulated by entity type (persons 2.3x more than locations), and spike 4.1x for post-cutoff entities. The Entity Hallucination Profile provides a practical risk assessment tool. These findings argue against uniform hallucination mitigation and for targeted, entity-aware strategies.

References

[1] Z. Ji et al., "Survey of hallucination in natural language generation," ACM Computing Surveys, 2023.

[2] S. Kandpal et al., "Large language models struggle to learn tail knowledge," ICML, 2023.

[3] P. Lewis et al., "Retrieval-augmented generation for knowledge-intensive NLP tasks," NeurIPS, 2020.

[4] K. Mallen et al., "When not to trust language models: Investigating effectiveness of parametric and non-parametric memories," ACL, 2023.

[5] A. Min et al., "FActScore: Fine-grained atomic evaluation of factual precision," EMNLP, 2023.

[6] N. Muennighoff et al., "Scaling data-constrained language models," NeurIPS, 2023.

[7] S. Li et al., "Inference-time intervention: Eliciting truthful answers from a language model," NeurIPS, 2023.

[8] T. Sun et al., "Head-to-tail: How knowledgeable are large language models?," NAACL, 2024.